Uus,

If you connected the RC directly to AT89S52's output pin, it won't work properly.
The principle of this "compression" requires a symmetrical push-pull output, whereas the AT89S52's pins are quasibidirectional (if you don't know what does it mean, read the hardware section of "bible"). Simply put, the AT89S' output is not powerful enough to pull the output high.

The simplest remedy is to insert a buffer, e.g. two inverters of 74HC04 type (74LS is not too good either, as it has an asymmetric output, although in this respect better than the AT89S' outputs).

Don't expect spectacular improvements, though. The given compression method is crude and so are the results. Any lossy compression won't reproduce the original sound too cleanly, but the more sophisticated ones (GSM, MP3) employ knowledge of formation of human speech (GSM) and "understanding" of music by human brain (MP3) and use these limitations to determine which parts of the signal are "not so necessary". That these require several orders of magnitude more computing power than this simple compression, is an another issue...

Have you tried to listen to the output from the program? The output of the final circuit will be no better than that (maybe even slightly worse, due to imperfections of that program - doesn't assume correct charging model for the RC, see my previous post).

JW