Here is what I would do Mikkel.

1) Connect the encoder A & B connections directly to two port pins without the interrupt intention. These encoders typically provide a connection to GND so I would use 4.7 -> 10K pullups to VCC on each pin.

2) Provide for a timer interrupt at 0.5 KHz rate (2 msec).

3) Provide for a debounce filter for the two encoder inputs. This can be done by adding a small task to the timer interrupt that simply reads in the two encoder signals each interrupt and then stores the sample into a stack of two memory locations. If the encoder was on P3.5 & P3.6 ....

ENC_FILT0:
   Ds    1      ; filter stage 1
ENC_FILT1:
   DS    1      ; filter stage 2
ENC_FILT2:
   DS    1      ; filter stage 3
ENC_CURR:
   DS    1      ; current filtered encoder inputs
ENC_PREV:
   DS    1      ; previous filtered encoder inputs
   ...
   ...
   MOV   ENC_FILT2, ENC_FILT1  ; shift the filter
   MOV   ENC_FILT1, ENC_FILT0
   MOV   ENC_FILT0, P3         ; get encoder bits
;
   MOV   ENC_PREV, ENC_CURR    ; place current to prev
;
   MOV   A, ENC_FILT0          ; find stable bits
   XRL   A, ENC_FILT2
   MOV   B, A
   MOV   A, ENC_FILT1
   XRL   A, ENC_FILT2
   ORL   A, B                  ; 1's for unstable bits
   ANL   ENC_CURR, A           ; keep old if changing
   CPL   A                     ; 1's for new stable
   ANL   A, ENC_FILT0          ; get new stable bits
   ORL   ENC_CURR, A           ; produce full current

Note that the above filter actually filters for all 8 bits of P3 so if your other four buttons were on other pins of port 3 this would debounce them too.

4) In the same timer interrupt, using above ideas, derive the logic of the encoder changes by looking at the current and previous state of the bits 5 & 6 of ENC_CURR and ENC_PREV. This may be done somewhat like:

UP_BIT:
   DBIT  1                     ; bit for UP marker
DN_BIT:
   DBIT  1                     ; bit for DN marker

   ...
   ...

   MOV   A, ENC_PREV           ; look for bit 5 0->1
   CPL   A
   ANL   A, ENC_CURR
   JNB   ACC.5, FILT_DONE      ; no P3.5 edge
   MOV   A, ENC_CURR           ; test current bit 6
   JB    ACC.6, FILT_UP
   SETB  DN_BIT                ; show down to main code
   JMP   FILT_DONE
;
FILT_UP:
   SETB  UP_BIT                ; show up to main code
;  
FILT_DONE:
   ...
   ...

5) Take action based upon the UP or DOWN decode to step your volume counter. If the SPI support call that you showed in your initial example is a serial bit banged routine I would avoid calling that in the interrupt routine and instead post the UP and DOWN as bit flags to the mainline program which would have the job of running the SPI interface. As the mainline sees up o down bits set it would take action and then clear these bits.

Michael Karas