It works ok with a 12v supply - obviously you have some sort of 5V regulator. If we assume you are using something like a 7805 regulator - have you put the required capacitors on it?

Then we go to the switchmode psu - is this outputting 12V or 5V?

If you are using the 12V then through a 7805 regulator - do you have the required capacitors attached? If not - this would explain the weird behaviour.

Can you tell us how and what you have used.