Hi Peter,

ISP doesn't use a fixed frequency, which you could set up directly.
It measures the Baud Rate, when receiving the first serial bit from the PC(FlashMagic).
This enables the micro to adapt to many different baud rates.
If FlashMagic sends with a baudrate of 38400, then the length of a bit is about 26us,
Your uC uses a 12MHz Crystal in 6clk (i guess), that means it can measure with
1us increments.
This gives you an error of 4% maximum, and the UART has only a tolerance of 2.5%.
In your case the error with the 12MHz crystal was too high,
with the 11.0592MHz crystal it worked, but just by accident.
So why don't you just reduce the baudrate
to 9600 for example?