Meta stability is not in general an issue of a slow rise time that can be cured via a schmidt trigger buffer. The problem comes into play when the input to a flip flop has not met the data input setup time before the clock active edge. Under these conditions the clocked state in the flip flop may flop around with uncertainty or stay on an analog state for a period of time.

Yes, a schmidt trigger buffer in line with a slow rise time signal on its way to a clocked flipflop, can help to cut down on the possibility of metastability in the flip-flop because the signal is forced to a valid level sooner and thus the statistical possibility of missing a setup time in a given clock cycle interval will be less.

The meta stability filter in the Agilent/HP chip uses a series string of flip-flops on the encoder signals that are all clocked at the same time. (I believe it was Intel that originally poineered this concept in early work on UARTs and TIMER chips if my memory serves me correctly. Bill Pohlman told me about this some 20 years ago. He was a designer of the LSI-11 chip set at Intel and later designed the multiplex address/data bus of the 8085 and of course we know the 8051 inherited that).

You may be able to improve reliability in a software quadrature decoder like you have suggested by reading the inputs till stable.....but be aware that multiple reads may still each see a high frequency edge change going on if the encoder is "right on the edge". (Also note that the high frequency stuff that can come out of an encoder that is sitting right on the edge can be caused by more things than just mechanical vibration of the slotted encoder wheel. System electrical noise can get inside an optical encoder sensor circuit and dither the sensor comparison threshold and thus make the edges have dither. Dirt and fine fiber particles inside an decoder can cause problems too. And lastly an optical encoder that is not sealed to stray light can be effected by light intensity modulation casued by the power line frequency.

Michael Karas