port this to 8051.

that's easy to say; unlikely to be trivial in practice:

Don't forget that DOS is based on a 16-bit processor - the Intel x86. Apart from the word size, it's a completely different architecture.

QNX is "free for personal use" and largely open-source, though I don't think there's an 8051 port.
http://qdn.qnx.com/support/h...ssors.html

Triscend has a couple of Reference Designs:
http://www.triscend.com/products/indextechlit.html