What maximum frequency that I can give with the combination of CPLD+89c52 for two channels

A fact that is very often overlooked (I did the first time) is that when the axle stop with the encoder exactly over an opaque to clear transition (assuming it is an optical encoder) the slack there always is will be in the link will cause a high frequency oscillation between forward and reverse.

Before you design the CPLD code make sure you draw out all transitions between fwd and reverse, the usual "B is direction, A is clock" fails in one case. You need to store previous state and compare to obtain true count.

Have fun,

Erik