Kevan:
If you are free to change the crystal frequency why don't you try a frequency that gives you an exact baud rate. Just try it to see if it helps. If so then you know that 4% error is too much. I can share that I have had many problems with high speed serial comms when characters were back to back and the baud rate was off by only 1.5% !!

UARTs often have less capability to re-sync when characters are back to back. Sometimes spacing can be attained by making the PC end send with two stop bits.

Michael Karas