1.if shaft encoder is mechanically vibrating?
redced vibration by alingment.
2.is the body of shaft encoder making electrical connection with the motor body?
if yes then dont use the same supply for encoder and microcontroller,otherwise you will have no advantage of opto-isolation.use a separated (isolated)supply for encoder
3.or electrically isolate the encoder by using insulator (such as taflon) coupling between motor and encoder.
4.connect a ceramic capacitor of 200 pico farad directly the pin of microcontroller which is recieving pulses from encoder to ground.
5.always use a 100 nano farad copacitor b/w microcontroller vcc pin and ground.this capacitor should be near to the vcc pin of microcotroller.
keep inform me ,what is hapening?