I'm sure this command is send by uart since the hyperterminal can read it.

No, the command is in pure text, hyper terminal is your first tool. usually the serial port of the gsm modem is ttl level, you need MAX232 to convert the voltage levels to be able to interface it with a pc's serial port, but for your 8051 you don't need the max232 because the serial uart of the 8051 is ttl level.

I'm thinking the GSM modem doesn't seem to recognize this command. Anybody can help me?

the best thing to do is start with hyper terminal to get familiar with the gsm modem commands as I mentioned above.
<big><big>Read the GSM modem manual</big></big> to see the supported commands but here are the most common ones:
1. initialise the modem by sending the command AT+CNMI = 2,2,0,0,0<CR> (can't remember what this command does).
2. issue the command to set service center number AT+CSCA = "+xxxxxxxxxxxx"<CR>
3. to send SMS issue the command AT+CMGS="distination"
after you get ">" in hyper terminal type your message terminated by "ALT Z" ascii 26 hex i think.
4. to be able to display incomming SMS messages you need to issue the command AT+CLIP =1.
After you are happy with hyper terminal start programming with the 8051.
good luck
Mahmood