It could also be that you have the register for the input pins set low. Your pulldown resistors could be fooling you that the pins look like low inputs when really they are acting like low outputs. If you then used a switch to try to directly connect these "low outputs" to 5 volts the chip would certainly draw MUCH more current. To fix this make sure to program the register bits on these pins to 1's.

You should consider instead to put PULLUP resistors on the inputs and have the switches pull the inputs to GND. The advantage of this is that you will extend battery life when you go to battery power. The pulldowns draw current even when the switches are open becasue the pulldowns are overcoming the weak pullups on board the microcontroller.

Michael Karas