I have the same problem.
I have cheked my circuit with hyperterminal
and ot worked properly.
my circuit send some text data and hyperterminal
shows them===>my circuit is wright!
but i can connect it with matlab
the settings in matlab and 89c52 are the same:

baudrate=9600
parity=none
stop bit=1

the program in matlab which must be right (but isnt) is:

s=serial('com1')
fopen(s)
fscanf(s)
fclose(s)
delete(s)
clear s

the error is: TimeOut error but i dont know why ?