Here is a truth table that handles hysteresis of an encoder so that when it is on the edge between two codes the counter will not bounce. Also, it assumes that if you missed a state that it was going in the same direction that it was from the previous state change and the counter is adjust for the missed state.

Enjoy
<>< Lance.

A-B encoder outputs are translated to letters to make the sequencing easier to comprehend.
A B
0 0 = A Forward is A --> B --> C --> D --> A ...
0 1 = B
1 1 = C
1 0 = D

AB' is the previous state of AB.
LD' is the result of LD (Last Direction) from previous state.
F stands for Forward, B for Backwards direction.
Initial values of AB' and LD' are not so important.

I N P U T S     O U T P U T S
AB  AB' LD'     LD  Action
--  --  --      --  ------
 A   A   F       F         
 A   A   B       B         
 A   B   F       F   INC   
 A   B   B       F         
 A   C   F       F   ADD 2 
 A   C   B       B   SUB 2 
 A   D   F       B         
 A   D   B       B   DEC   

 B   A   F       B         
 B   A   B       B   DEC   
 B   B   F       F         
 B   B   B       B         
 B   C   F       F   INC   
 B   C   B       F         
 B   D   F       F   ADD 2 
 B   D   B       B   SUB 2 

 C   A   F       F   ADD 2 
 C   A   B       B   SUB 2 
 C   B   F       B         
 C   B   B       B   DEC   
 C   C   F       F         
 C   C   B       B         
 C   D   F       F   INC   
 C   D   B       F         

 D   A   F       F   INC   
 D   A   B       F         
 D   B   F       F   ADD 2 
 D   B   B       B   SUB 2 
 D   C   F       B         
 D   C   B       B   DEC   
 D   D   F       F         
 D   D   B       B