Make yourself an adaptor lead that will allow you to "Tee" off and use Hypoterminal to actually see what they're doing!
If your PC has two COM ports (or you have 2 PCs), you can use two Hypoterminal sessions to watch both Rx and Tx at the same time.


How do you know that?
A modem is likely to require a different lead to a PC.


Have you tried it in the simulator?
Set it up so that the simulated 8051 UART drives the PC's COM port, and connect the modem to the COM port.

If this fails, it shows that your AT Command/response handling is incorrect - and it will be a lot easier to see what actually is happening!

If this works, it shows that your AT Command/response handling is correct, but you need to look at your hardware interface to the modem; eg,
* Baud rate (exact baud rate - see Erik's crystal comment);
* data/start/stop/parity bit settings;
* control lines;
* cables, RS232 transceivers;
* inter-character timing;
etc, etc, etc,...