Every time I have seen inconsistent counts from an encoder it has been due to incorrect handling of gear slack.

Three issues are often forgotten
1) when the encoder shakes and is positioned just over the clear/opaque boundary, you acn see "counts" with an unbelievable frequency.
2) the "B is direction, A is clock" scheme fails in 2 out of 16 direction changes.
3) up count only does not exist for the same reaon as 1)

Erik