My experience with Flash-Magic reprogramming an existing program that is using the UART:

Flash-Magic will try to place the chip into Boot mode but since there is data in the PC serial buffer the echo recognition process fails. My solution is to increase the initial hold-off time of PSEN and manually apply power to the chip during this period. So the existing program never starts, and Flash-Magic will perform happily. I use a RS232 Mini-Tester (bi-colour LEDS) in the line so that you actually see the DTR and RTS lines being toggled by Flash-Magic.

I think that both Flash-Magic and Flip will read the chip IDs etc but the programming commands use different subfunction codes in the 03 write function record. Also the security bit read subfunction is different so each programmer rejects the other manufacturer.

Your original post seems to suggest that Boot mode is not left after Flash-Magic failure. Again the mini-tester will show whether you have set the DTR and RTS true/false settings correctly. Please post a description of your hardware connections of DTR and RTS.

David