" I'd describe it as too hot to touch for a few seconds at a time ... Is this damaging to the part or is this normal?"

The only way to get a definitive answer to that is to actually measure the temperature and read the Datasheet.
Don't forget that it'll be hotter when shut inside a box, and allow for hot days!
As the others have said, you must also do the Power Dissipation sums.

However, as a rule of thumb(!), I'd say that if it's too hot to touch for a few seconds, then it is too hot! You need to add some heatsinking.