How often are you sampling?

Assuming the A/D returns 8 bits and the MIDI controller message is only 7 bits, are you at least LSR (shift right) to throw away
the LSB? You should do that first, then if it's still a problem:

Make sure that vRef and Vcc are as stable as possible and perhaps not the same. As someone mentioned, unless the value doesn't change by "x" amount, don't update the controller value until the user moves it beyond "x" from the last accepted value. This is standard procedure for these devices. I wouldn't really get into the averaging if you don't have to, but it depends on the answer to my very first question.