You will need to initialize the UART regsters in the 8051. There are numerous posts and sample code posts here on how that can be done.

When you set out to interface the protocol of the phone modem you will need to understand that protocol before you can be successful in making the link. If you have no documentation that tells how this works then I can suggest only one thing...

Maybe it is possible to connect the phone modem to a PC comm port. Then operate the phone modem from some software on the PC that you know functions correctly and the way you want to emulate with the 8051. Then using a comm line snooper or data analyzer capture all the bytes sent between the PC and the phone modem. (Capture both directions so you can observe the handshakes). Then study the captured data and from that learn how the phone modem protocol works.

There are ways to make a comm port snooper using TWO comm ports on a 2nd PC. You can do WEB searches and find informaton for doing this (i.e. how to build the special cable required and find the capture software).

Good Luck
Mike Karas