I've only ever used a 16MHz crystal, and that works. The USB module needs 48MHz, and the PLL can obviously generate 48 MHz from 16 MHz. I suspect that using 20 MHz might not work.