You need to isolate whether the problem is transmit or receive.
1) is the origin a PC ? some PC programs will change the case of characters.
2) do you have a parity check?
3) is the problem the same in either direction?
4) have you tried to transmit a constant i.e. not a received byte. ?
5) The 'masking' does not sound as an UART (soft or hard) problem.

Have fun, do not try to eat the whole cake in one bite - or is that byte :-)

Erik