If you are monitoring both the A channel and the B channel, you should be able to determine whether the encoder signal is jitter or if it is real. The logic should be setup such that a it is a real pulse only when both the A and B channels change state. Also you can determine the direction by looking at which pulse changes state first, the A channel or the B channel.
Sorry if I confoosed someone.
The solution for the jitter issue is a matter of response speed.
The "one of 8 reversals (actually 2 of 16) error" happens regardless of speed.

Erik