It always pays to stop and evaluate the situation. Lets look at the evidence given:

AT90C2051
Any port pin 2.3v as a high
Measured with a multimeter.

Firstly - what was it measured with? What sort of multimeter? It could be anywhere between an AVO8 or a Fluke 189. If we assume it is a digital multimeter, then the load presented to the port pin is small.

Secondly - most of us realise that most 8051's don't drive their port pins high - they are pulled up by an internal resistance. This could explain the low '1' voltage. I tend to use pullups when driving other logic and when used as inputs. If one is used to using other micros like PICs ,AVRs etc - these can actively drive the port pin as an output so the pin can swing from rail to rail almost (depending on the load). So remember - most 8051's can't source current, but they can sink it! 2051's on P1 can sink over 20mA per pin if my memory serves me right.

Apart from the low '1' voltage, does it work otherwise?

Jan - I'm the one with the crystal ball and the magic 8 ball! There's only room for one mystic here!