if you use a gray code encoder the code is simple

pos // position of machine
TnISR
new_pos = convert_gray_encoder_value_to_binary
if (new_pos < old_pos) new_pos + MAX_GRAY_VALUE +1
pos = pos + (new_pos - old_pos)
old_pos = new_pos

1)The tn interrupt need occur at least often enough to read the encoder twice per revolution.
2) the mechanical linkage and gearing must be so the the max jitter at macine stop and at direction reversal 1/3 enoder revolution.
3) if the above can not be met, other means will be complicated enough to revert to the a/b encoder.
4) of course pos must be reset by a separate zero sensor.
5) never act on position information until the zero sensor has trigged at least once

Erik