I have not checked, but I seem to recall that the 40xx series is quite slow, if you are running your '51 at a high clock frequency this is probably the reason you need the '32. Have a look of your timing with the datasheets in hand. It seems you are using the or as a negative and and in that case your latch will clock on any address 0000-7fff. If you have any RAM connected in that address range the output of the latch will, of course, respond to ram writes.

Have fun checking the propagation delays

Erik