You overlook, that the port is no simple input but a mix of three active pull-ups! And what you try to do with the 4k7 pull-down resistors will not work. Keep in mind, that there's a weak active pull-up and a very weak one. If you apply logical high level to port pin, while having written logical high to according SFR register, you force the output to turn on weak active pull-up. After releasing the button two active pull-ups are turned-on, the weak and the very weak one. Several hundreds of µA can flow now, not allowing to make the level across pull-down resistor reach zero volt. If the input voltage would fall to less than about 2V, then the weak pull-up would be turned-off and no more than some dozens of µA could flow due to still activated very weak pull-up, making your pull-down resistors do what's intended to do. But because the weak pull-up is still activated, the input voltage will never fall to less than 2V. So, what you plan with this pull-down resistor is irregular, not compatible with the certain active pull-up configuration of 'C51.

The reason, why you observe the problem only when connecting the multimeter is probably due to the fact, that the multimeter picks-up some hum forcing the potential at input of port to be slightly higher than the voltage needed to keep the weak pull-up activated, while without hum your pull-down resistors are just able to make the potential fall to zero volt.

This behaviour is typical for all standard 'C51. You are just not allowed to work with such high pull-down resistors.
By the way, 'C51 are working with negative logic, strongly recommending the use of pull-up resistors with keys making connections to ground.

Kai