Hi Prakash

If your modem does not get any dial tone then it gives result code as NO DIALTONE
Still you can avoid this by giving ATX3

But if you have problem in giving AT command itself (e.g AT OK)the reason behind this may be of some setting problem of modem registers. If the reqired baud rate is 9600 the initialization is as follows

LCR = 0x80; /*Enabling the DLATCH 0*/
THR = 0x0C; /*Setting the baud rate 9600*/
DLM = 0x00; /*MSB value for baud rate
setting */
LCR = 0x03; /*Disabling the DLATCH*/
/*Setting in 8 bit mode*/
FCR = 0x01; /*enable FIFO mode*/
MCR = 0x0f; /*initialize modem control
register*/
DLM = 0x01; /* Enabling receiver data available interrupt bit */

provided these modem parallel registers are memory mapped to the 8051 processors. You can access these registers by the particular offsets given in the data sheet of modem

regards
Prakash Aital