Some things to look into...

First, some encoders are configured with open collector outputs. (i.e. they only sink current). If you have this kind of encoder then you are depending only on the weak pullups of the microcontroller to make the signals go to the high state,,,,which is a very bad thing when off board cables are used. At least provide a low value pullup.
(better to buffer though).

Second consideration. Many encoders are configured with differential outputs. I.E. channel A has a + and a - output and the same for channel B. If you have this type of encoder you should consider using this capability to buffer the signals into the microcontroller. You will get greatly increased noise margin and common mode isolation. The receiver type part to buffer a differential encoder output to a microcontroller is a chip of the same ilk as an RS422/RS485 receiver. An example part to use would be MAX491 or DS26LS32A. If you use these parts do also consider putting a termination resistor between the + and - inputs at the receiver on your microcontroller board. The terminator causes a higher current to flow in the wires back to the encoder output that is balanced. And yes with a differential output the wires should be 120 -> 150 ohm twisted pairs.

You could try opto couplers but I think they would not provide you much benefit unless the encoder ground is completely separated from the microcontroller ground. And also the bandwidth of the opto coupler may be an issue if you have high motor RPM unless you pick the more expensive higher bandwidth devices.

In summation I can say that I have worked with many an encoder in an industrial setting. I have always specified the encoder type with the differential outputs, connected then to the embedded system via twisted pairs, and used 26LS32A receivers. Always worked like a champ!!

Mike Karas