Ah, I think I got the problem, Shazhad. You use the same rectified voltage for 12V regulator and 5V regulator, right? Well, then you will of course have an extreme heat dissipation...

For having working 12V regulator you need about 15V rectified input voltage. If you feed your 5V regulator with the same input voltage you have heat dissipaton of 1W when having supply current of only 100mA.
You should proceed the following: Use an additional transformer for your 5V supply. Choose it so, that you have about 9Volts at input of 5V regulator, at maximum load. Then, 100mA supply current would result in only 0.4W.
Why 9V instead of 8V, which would be enough when reading datasheet? You have to keep in mind some undervoltage of mains. It's good to be prepared for at least 10% undervoltage.

There is some trap when dealing with small transformers: When they are used for currents which are much less than current rating, there can be an extremely overvoltage. E.g. 9V 100mA transformer without connected load will produce a voltage of about 12V...13V effective. So maximum voltage can rise up to 18V! This could also be a reason for your big heat dissipation.

You can also use some low drop voltage regulator, like Mahmood suggested. But extra care for decoupling is needed. National Semiconductors have detailed specifications in their datasheets. For LM2940 they write: 'Cout must be at least 22µF to maintain stability.'

What about cooling?
Air is a very bad medium for transfering heat from 5V regulator. And it's again and again a big surprise, how much heat transfer can be improved when using some metallic part connected to it (isolation not to be forgotten!). This could be one wall of enclosure, ground plane of PCB, or something else. Heat transfer of this metallic part is the better the thicker the material is.

Bye,
Kai