say 10 bits (start - data - stop) gives yoy 240 bytes/sec at 2400 bits/sec
1/240 ~ 4mS.
One thing mentioned by others that I feel they did not stress enough is keep interrupt routines short and sweet or you WILL get in trouble.
As far as what you are doing, the cookbook method is to have your serial interrupt store received bytes in a circular buffer and have the display routine peel them off in the main loop.
If the timing gets critical, you will gain by using a timer rather than loops for your delays in the display routine, the time keeps running during the serial interrupt, the loop does not.
Have fun,
Erik