1. check your modem with PC and a terminal emulator program to see, if your modem is OK. Type ATZ, modem should respond OK.

2. check if your 8051 board runs OK. Connect it to the PC via serial cable and check data serial transmit and receive (observe AT commands on the PC)
Take care to the RS232 parameters.

3. check your 8051 board to modem connection (take care to TX, RX, RTS, CTS, DTR, DCD, DSR and probably RI signals). TX line of DTE (like 8051 board) is output, but of DCE (like modem) input. So TX of 8051 should be connected to TX of modem etc. All of them should be defined properly. Note that modem is a DCE equipment. If you're not using all signals make unused signals loopback or set properly in modem setup and store settings in modem NV memory.