Hows this for an idea? The zener diode will try to keep the voltage from exceeding it's reverse rating. Use a zener that is rated for slightly more than 24V. Then under normal operation the current through the zener would be negligibly above zero. The current through the diode can be used to trip a relay to isolate the supply from the boards. If it's a double throw relay it can crowbar the supply voltage to ground. If this is done at 30V, and the hugh-honking 500V cap holds off the voltage rise for just a bit, anyway it's a thought.

Waaaay too slow, the circuits are burnt before the relay pulls.

The only circuit that is fast enough, to my knowledge, is a crowbar not with a relay. for general crowbaring (argh what a word) have a look at the MC3423.

Erik