I agree with Rob about the light disturbance caused by your hand. To prove if this is the case simply wrap the IR detector in very opaque black plastic. (Black plastic electrical tape will do). If the issue goes away when you press buttons then you know what was causing the problem.
Mike Karas