I doubt if the main problem is noise from the encoder on its own - you're probably getting all kinds of crap everywhere ! What decoupling have you got already ? What is the physical layout ? Is the motor drive snubbed/filtered ?

This kind of topic is cropping up more and more , Erik posted his favourite tips for fixing this problems and I agree with all of them, though you would do well to assign a ranking to the ideas -all good, but some more expensive than others ! A steel case is good ALWAYS.

Opto isolation is useful,breaking ground loops and providing glavanic isolation, but it won't help radiated interference,and if the noise is sufficient to mess up counting, you're in deep trouble....

Steve