If you can up both voltages (power supply and battery) to 7 or more volts, then two diodes followed by a LDO or a DC/DC converter will do the trick. The power need be a bit higher than the battery.

          |
          | 
power ----|  |---*
          | /    |
          |/     |
                 *---LDO ---- 5V
          |     |
          |     |
battery --|  |---*
          | /    
          |/ 

Erik