Bartosz,
The "fast part" done by custom electronics (in my case it was on 74LS series gates
Please show

Then take A XOR B' as direction and count the turn. Store A and B as A', B' for use in next count
You did implement quadrature.

I am still not totally convinced re filtering, but your post made me think it over again and it is, indeed, possible that filtering works since - with good mechanics - the noise always occur at or about standstill. The one case that would definitely kill filtering would be a lose gear (they all are to some extent) and inconsistent motion which could generate noise at high rotation speed.

When I did the encoder thing the original design used the faulty a-clk b=dir algorithm and a straight running both to interrupts did not work because of noise at standstill. I did then implement it all in a CPLD and it worked.

Erik