"a 8051 understandable format, which is either decimal or hex."

The 8051 understands neither decimal nor hex - it only "sees" binary.

"when I multiply it with other decimal numbers, the carry part is a problem because the 8051 assumed that both numbers are in hex form"

Yes, I think Luca is right - you need to leave this stuff to a high-level language; eg, 'C'
Unfortunately, I think floating-point support is excluded from the free Keil Evaluation. :-(