Great! This extra detail about what you are doing is very helpful. Now we know exactly what you are trying to do. (Next time, please try to give more detail in the beginning so we can get at the real problem faster.)

Here is what I would do:

1. Configure an unused I/O pin for output and and connect your oscilloscope to it.
2. Set the output pin to '1' at the very beginning of the timer interrupt routine, and then set it back to '0' at the very end of the timer interrupt routine.
3. Run the system and look at the output on the oscilloscope. The duty cycle of the waveform will show you what percentage of the total time is being spent in the timer interrupt routine. Be sure to run this test under the worst case conditions. For example, if the timer interrupt routine runs once per motor step, then do this test with the motor running at its maximum speed.
4. Remove the test code added in Step 2.
5. I assume that you are receiving the incoming RS232 message in an interrupt routine also. Set the output pin to '1' at the very beginning of the RS232 interrupt routine, and then set it back to '0' at the very end of the RS232 interrupt routine. (If you are not using an interrupt routine to receive the RS232 characters, you probably should be!)
6. Use the oscilloscope to measure the duty cycle of the waveform. This will tell you what percentage of the total time is required to process the incoming RS232 message. Again, you should run this test under the worst case conditions. In this case, that means the sender of the RS232 message should send the characters as fast as it can.
7. Remove the test code added in Step 5.

Now, if the worst case timer interrupt time + the worst case RS232 interrupt time add up to less than 100%, it should work in theory. In practice, as you have said, you need some margin to give your non-interrupt code time to run, or in case you need to add code to the interrupt routines, or ???

If you find that you are using too much time, you can do several things:

1. Try to optimize your code. In particular, if you have coded your interrupt routines in C, then you might be able to make the faster by rewriting them in assembly language.
2. Reduce the baud rate of the RS232 messages.
3. Don't do any more processing in the RS232 interrupt handler than absolutely necessary. Your RS232 messages probably come in bursts of characters with lots of time between bursts. If this is the case, you might be able to use the interrupt routine to quickly put the characters in the buffer, then decode and process the message later as time permits in non-interrupt code.
4. If your processor has a PCA, consider using the PCA hardware to drive the stepper motor instead of doing it "by hand" in an interrupt routine.
5. Get a faster processor

One other hint: In systems like this that I have worked on, it was very important to step the motor at exactly the right time, but it was okay to delay the reception of the RS232 characters a little bit, as long as none of them were missed. Therefore, you might need to give the timer interrupt priority over then RS232 interrupt to make sure the motor step pulses occur at exactly the right time.

-- Russ