Mikkel,

I'm not too sure why you have the hc148 priority encoder in the circuit, nevertheless, is the interrupt setup up for edge trigger? The bit in this instance is TCON.0 (IT0) which should be set for edge trigerred interrupts. Also the Alps device could be a mechanical switch (i'm too lazy to search out the data!) in which you need to cope with switch bounce. With these devices, using an external interrupt is probably overkill and my suggestion is to use a timer interrupt to sample the encoder at a fixed rate (say once every 10mS) and have the software decode the quadrature. I dare say the average computer mouse used this technique. This timer interrupt can also be used for other software timing functions.

Raghu's suggestion is probably best if you were using an optical encoder for motor servo control.