Oleg, this has been discussed many times, and the problem is that the pulses income at rate much higher than the uC clock, so before the '51 can "add/substract" a pulse, several other pulses arrive and get forgotten so the readout value is in fact not "fast vibrations" but "binary noise" and the value of next readout is in fact undtermined. (and while theoretically integral over white noise is zero, practically it means the result floats in random direction)

The solution Erik suggests is a sophisticated piece of custom hardware that can process/count the signals at much higher rate than the '51 can, and higher than what the encoder can generate them (even as the noise) therefore keeping the valid count regardles of rotation speed or noise frequency. It was tested. It worked.

The solution I suggested was a simple piece of hardware that just kills all the high-frequency signals (by passing raising edges immendiately and falling edges with minor delay, so if 0 appears for very short, it won't pass the filter, therefore any high-freq noise gets replaced with 1's.). It wasn't tested. Erik insists it won't work.

The solution in software would be just to implement it as if nothing happened... except you'd need to use a '51 clone of high CPU speed, capable of processing all the noise that gets generated by the encoders just like a valid signal without losing counts.