Erik,

If you are monitoring both the A channel and the B channel, you should be able to determine whether the encoder signal is jitter or if it is real. The logic should be setup such that a it is a real pulse only when both the A and B channels change state. Also you can determine the direction by looking at which pulse changes state first, the A channel or the B channel.

I agree that it much easier to use a hardware solution (PLD or HP chip)though.

Jacob