1.Can you mention what SmartModem type did you use ? And all AT commands issued to the modem in command mode state.

2.You did this by selecting a value different from zero on S0 register


4. It's interesting how you manage syncronization between the BUSY printer flag and XOFF send to the modem.

5. You have to be aware that modem it self doesn't know anything about the information that carries on (not even start or stop bits at low level). It's 8051 duty to rebuild the bytes from the stream. Thus is your software choise if or not to send the bytes to the printer. It's not printer's fault if prints junks.

6. If you want a reliable communication you have to provide a protocol. It's mandatory and the only way you can avoid undesirable bytes. Like you did.

7. This is due to the noise in PSTN line. However, at least like a test for low level procedure try to set SM2 bit in SCON register. Thus RI will not be activated if a valid stop bit was not received. Obvious, if you work in serial mode 1 . However, if you still received them , just build a buffer on external RAM and upload them there. You can parse in silence the incoming undesirable bytes and see their nature. Just for fun.
But if you use frames, as you already did, you can discharged them.