You have not stated which 8052 derivative you are using and at what frequency you are running it. However I do have a couple of observations.

1. The SMOD bit in PCON does not affect baud rates when using Timer2.


2. Your value for RCAP2 does not pass my sanity check. For the sake of discussion, let's just assume you are running at 11.0592MHz. To generate 9600 baud, I would expect the value for RCAP2 to be 0FFDCH.

For using Timer2 as the BRG, the formula is:

    baud = OSC / (32 * (65536 - RCAP2))