Whilst you maybe busy doing other things, you're not spending the time to solve your problem! As for the handshake control lines I think you need only worry about DSR (or is it DTR) as the modems can be programmed to hang up the line if this signal is dropped. You can use CD (carrier detect) to monitor if the modem is connected.

Connect your PC to the modem and use hyperterminal (or whatever terminal program you like) to talk to the modem directly (direct to com port in hyperterminal speak) and manipulate the modem by typing in the commands manually. You can measure the state of the modem control lines to see what the PC/modem is doing. You can get an rough idea of the delays by seeing the response from the modem. Now translate this into your code on the 8051 - when the 8051 is conencted to the modem, have the PC look at the communications happening between them. From this you will be able to observe what the 8051/modem is doing.

No magic answers sorry! I've got to do the same for a GSM modem I'm playing with. My code will implement timeouts and retries to cope with the modem being disconnected, failing etc.