There could be many reasons why you project is not working. Firstly - how do you know if the hardware is correct? What steps have you taken to ensure your software is correct? If you have problems with both you will be chasing your tail for some time. As for the software, have you done a search for drivers for the 16550? The linux drivers may be a start. Then you may want to test your code on a PC with a 16550 installed (very likely in most pc's). I assume you're writing your code in 'c' - written carefully you'll be able to use most the code on either the pc or the 8051 (or other cpu for that matter) - just call a routine to read and write the 16550 (only these two routines will differ between the 8051 and the PC).

Try to stay away from interrupts first up - get the init, send and receive routines working and then introduce interrupts.

As for the hardware - does the crystal oscillate and can you get a 16X baud rate clock out of the device (I can't recall which pin this is).

Write some code to dump the modem status register to the terminal - see if you can detect changes on the rts and other modem controls. Write code to toggle the modem controls CTS etc. Try to collect as much information you can by doing very simple operations - this should determine whether you have s/w or h/w prob or both.