Hi Kai,
After all the extensive testing you have done any suggestion may a long shot but perhaps worth revisiting.

Are you holding PSEN* low while you press the RESET and then let go of PSEN* a considerable time AFTER you release the RESET?
PSEN* needs to be LOW for some time after the RESET goes HIGH so that the micro forces the program counter to jump to the AT89C51AC2 Boot Loader.
If the boot loader is running, then Flip will be able to communicate with it if the rest of the hardware is ok.
May I suggest a current limited power supply so that you can limit the current to a reasonable level and serious shorts can be detected before serious damage is done.
Are you able to program the chip in some other system and then substitute it into your board. If you can, then program it to continuously send 'U' to the serial port. This binary sequence will allow you to check baud rates and debug hardware, at least in one direction.
I have used this particular micro and other derivitives in a multitude of applications with great success. Once you get Flip working try the Batch ISP in it,(all the commands which you use in Flip can be put into a text file and executed with one click!), it is magic! Although there are forum members who disagree, I have used a simple 2 transistor circuit and a couple of passive components for the serial port and has worked every time up to 115kBaud.

Good luck.
Tony