Well, I think that for f<800kHz the 8052 can handle an encoder just adding some TTL chips or PLD

First congratulations on adding the 'x4' circuit (it is the first time I have seen it done in discteres -bravo). While it does, indeed multiply the output by 4 I dislike the name. The REAL purpose of the 'x4' circuit is to handle the direction reversals that will in some cases give a one count error if the a=clk b=dir method is employed. Many have not implemented the circuit becuse the 'did not need the added resolution'.
A note of caution, I did not verify your circuit.

Second, I still have some doubts that 'handling 600kHz' is enough. When the encoder stop at the border between clear and opague I have seen pusles in the MHz range. Since I am not involved with encoders at this time, I can only recall that I did analyze RC filtering and - at that time - reasoned that it would induce failures.

Third, I think using your circuit with a fast processor such as a Cygnal f12x would be an excellent solution.

Erik