There are two options - manually resetting your device or letting Flash Magic reset your device using DTR. The latter option requires support in your hardware for connecting DTR to the reset pin.

For manual resets:

When you click on the Start button in Flash Magic it begins to send out a constant stream of 'U's. You can observe this by putting a scope on the transmit pin of your COM port.

Flash Magic then tells you to reset your device. You can do this by cycling the power, etc.

Once the device comes out of reset it looks for a specific time for a 'U' on the UART receive pin. If a 'U' is received then it enters ISP mode. If not then it will execute contents of memory.

To debug:

Put your scope on the COM port transmit pin and look for the stream of 'U's.
Put your scope on the receive pin of the device and look for the stream of 'U's.
Put your scope on the transmit pin of the device and reset the device. Look for a 'U' being transmitted.
Put your scope on the COM port receive pin and reset the device. Look for the 'U' arriving at the PC.

Note that you may need to try a conservative baudrate for the crystal frequency you are using. Start with 9600 or 7200.

Also note that the faster your crystal the shorter the time that the device waits for a 'U' after reset.

Andy