When transferring data over a modem, implementing handshake is a must. Hardware handshake (RTS/CTS) is quite easy - for RTS, implement an input buffer and two "watermarks" - "almost full" and "empty enough" and activate/deactivate RTS accordingly. CTS is even simpler, simply check CTS status befor sending a byte; if active, don't send it (the modem's buffer is full); and implement a mechanism (e.g. using timer or simple polling if the Tx is not interrupt driven) to recheck the CTS status and restart sending when CTS gets deactivated. With a 16450-and-similar hardware UART, it provides you an interrupt on control line changes, too; hence the servicing routine is even simpler.
At that moment, the baudrate is unlimited - the transfer speed is "throttled" by the handshake.
Packetizing/checksumming/confirming protocol such as Xmodem is not absolutely necessary; modern modems employ the same (plus a simple compression), it is called "autoreliable mode" (V.42). This is completely transparent from the point of DTE (PC/mcu).

Jan Waclawek