As a radio amateur I am used to write programs that run on CPU's with quite low clock frequency. This is to avoid EMI.

Having complex indexed loads while in interrupt does not only cost the time. It will also burn some bytes from Your stack - certainly more than those three needed in this case.

Also - and this reason is not too apparent. The interrupt code should be made as simple as possible and having some sort of conversions in there will add to the complexity. Debugging interrupts is not an easy task - not for the beginner at least. Debugging complex interrupts is really hard even for the experienced one.

I also metioned the flexibility. In case You want to create special effects like blinking or a cursor it is a LOT easier to do when the segments can be switched on/off individually outside the interrupt.