Use a 'd' type flip/flop.

phase 'a' into D
phase 'b' into CLK

Q has your direction out

If it is back to front either reverse phase 'a' and 'b' or use Qbar output

otherwise using the micro, one phase causes an interrupt which then reads the state of the other phase. (encoder 3600pulses per rev * 100 rpm)/60 = frequency = 6000hz = 166uS/pulse approx. You should have plenty of time to process the input and keep a count with a 12mhz 8051 even more if it is faster part.

Failing that, HP (Agilent) have some nifty ics that do all the hard work (HCTL1000 methinks??).