Hi Jason,
The thing you must ask yourself is how accurate do YOU want it to be? Imagine it in your car. Wouldn't 2 digits suffice for RPM? (X.X *1000)? Would your eyes be able to keep track of changes in the bottom digit if you had five? While driving? It's up to you!
I'm not sure I understand how you are trying to do your maths (divide in hex?). There are some great math routines in the code library on this site, maybe that'll help. I hope I'm telling you an obvious thing when I say that 60/(n*000002) can be rewritten so only one constant remains.
A look up table is a good way of cheating if real-time math is dragging your CPU too much, but only useful if you don't have too many possible in- and output values. Eg. translating a raw 16 bit value to a 4 digit display value (= 2 bytes BCD) would require 128 kB ROM. Using less digits or reducing the amount of input bits will make a lookup table more interesting. But then you loose accuracy of course...
Best wishes to everybody!