A good power supply to start with could be a simple circuit using a 7085 voltage regulator. These convert a higher voltage to +5 volts. They are very easy to build and provide a good stable +5V. You can power the input side from a wall mounted transformer of 9VDC or 12VDC.

A common circuit I use has a +12V input. I then put two capactors from the +12V at the input side of the 7805 to ground. One capacitor is a 22uF/35V electrolytic. The other is a 0.1uF/50V ceramic capacitor.
On the output side of the 7805 I also put two capacitors. I typically put a 10uF/16V electrolytic and a 0.1uF/50V ceramic. It is important that these 4 capacitors be quite close to the regulator. You should also distribute more of the 10uf/16V caps at various points around the board between +5 and GND. For a small board with just a few ICs you can get by with a couple of these capacitors. Next you should place 0.1uF/50V capacitors directly in parallel with each of the electrolytic capacitors (like right next to them). And finally (and VERY important) a 0.1uf/50V ceramic capacitor should be placed by each IC chip and connect between the VCC and GND pins of the IC. It is the best to try to achieve as short of connection as possible between these capacitors and the IC pins. (Note some larger ICs these days have more than one set of VCC & GND pins. It is a good idea to put more than one capacitor for these ICs if that can shorten the total connection length to the IC).

A special note here. If you are building a circuit that operates at higher frequencies it may be a good idea to place 0.01uF or 0.001uF caps at the IC chips either in place of the 0.1uF caps or in addition to them. I usually use them in addition to the 0.1uF caps and use them just at those chips that have high frequency stuff going on such as the microcontroller or an external RAM chip.

On some boards I use poly type capacitors instead of ceramic capacitors because I find that they filter some types of noise better. I usually use the low cost Al type electrolytic capacitors instead of the Tant types. In recent years tantalum capacitors have gotton extremely expensive.

With any design there is a balance of going crazy with totally filling a board up with capacitors or minimizing cost by using fewer capacitors. Also beware of bean counters that look at a bill of material and see 45 capacitors on one line item and tell you that the company needs to save money by using less capacitors. They see quantity as cost all too often and yet one IC on a board may cost 2x what all the capacitors cost.

A last subject about onboard type power supplies built with 7805 regulators is power dissapation. The typical 7805 comes in a T0220 type package and can deliver 1 Amp at 5 volts. Evaluate your design carefully and look at the voltage drop you are placing across the 7805 (i.e. Vin - 5 Volts). If you drop 6 volts across the regulator with 1 Ampere load that translates to 6 watts of power in the regulator. That will certainly require a heat sink on the regulator. I have found that with a 9 volt source supply with a load of up to 300 mA on the regulator that you can often get by without a heat sink on the regulator. Higher input voltages and/or larger load currents will require heat sinks.

Story time........
Many years ago as a young engineer I built a board for a logic board setup (pre microcontroller days) that had a 7805 regulator that was being loaded at nearly 1 Ampere. The input voltage was over 17 volts...I had a heat sink on the regulator and it still got hot. As I was debugging the circuit I noticed this mysterious 1 to 5 Hz oscillation going on in the logic circuits. I was stumped for a number of hours because the oscillation seemed to be everywhere in the circuit. Finally I had insight to put the scope on the +5V line and noticed the oscillation there too. What was happening was the 7805 regulator has an over temperature shutdown built inside. The heat sink on my board was right at the shut down temperature for the regulator. The regulator would heat up then shut down for a moment. As it went off the temp would fall just enough for the regulator to come back on. I found that waving air over the regulator would change the frequency. I'm not sure the modern versions of the 7805 would act this same way becasue I think they now have hystersis built into the thermal shut down circuit.

I hope this helps
Michael Karas