If the encoder rotates in reverse direction then after receiving 8 such consicutive pulses I am suppose to generate the error and stop the motor. But will this A=clk and b=input method will work for single directional counting?
1) if you can live with "between 7 and 9" reverse pulses you can do it by simpler means; however, to get this to work, you will need either a low pass filter (which will increase the inaccuracy some) or some hardware outside the uC. I once implemented the quadrature detector in a CPLD, but since you have the agilent chip already there, why not use it.

Erik