RTOS in my opinion is only semi useful with the 8051.

RTOS does the following:
Speeds up Dev time usually
Creates a "standard" shell
Usually creates a bugless shell (if commercial)
Allows programmers to move further away from the details of the hardware
Allows a programmer advanced handling of multiple CRITICAL tasks.

I don't know of many companies that would ever consider using RTOS on an 8051, but I could see that maybe someone would use it if getting the product out the door quick was important to them or lots of tasks were performed. RTOS is however frequently used on faster (busier) processors like Motorola 68k, PowerPC, ARM, MIPS, etc..

RTOS allows a program to constantly look for "tasks" to perform whenever needed. These tasks are usually assigned priority. In more complicated programs, an RTOS will allow you to devise a way of handling all tasks in a manor that won't degrade the system. For example, if one task is performing heavy calculations that take up to a minute, an easy way to prevent this heavy calculation from completely hogging the CPU would to assign it a priority and possibly a maximum time slice.

Anywho, you probably won't ever use an RTOS on 8051 unless you design something that has many things going on that need to be carefully handled to not let any one task to hog the CPU.

HTH,
Mike