The spikes seem to be at 1/2 second intervals and an amplitude of about 1 volt.

In these instances, you have to try to narrow down the problem - firstly check the obvious:

power rails - switchers aren't too good for analog stuff - too much noise. Normally one would use an analog regulator (7805 etc) after the switchmode to feed the analog circuitry. Try running off batteries!

put your pot directly on the input to the adc - do we still have the spikes?

software - is it a bus contention? s/w bug? Write some code that just reads the adc and outputs it to the serial port as numbers - keep it simple. You can log these with your PC and analyse the values using Excel (or whatever). Another test may be to apply half of full scale voltage to the adc, write some simple code to read the adc and keep a count of how many times the values exceeds +/- 10% , flash a led etc to try to get an idea of what is really happening. Try different power supplies to see if that makes a difference.

When weird things are happening - try to remove as many unknowns as possible to narrow down the cause. In many instances you find multiple causes all conspiring to make your life hell! You may find it is not the adc causing the problem! It could be a serial comms issue (assuming the data is sent via serial) - an extra digit gets added or subtracted.

It's hard to debug these problems by remote control! The more information you can provide us ,the more we can assist in narrowing down the issue.