Here's the rub.

Single-ended systems that use coax cables for interconnect have a real problem, which is the signal return is on the same wire as the shield.

To complete the circuit, you MUST connect the signal ground of your probe to the circuit common.

But to keep shield noise from corrupting your measurement, make sure that you do not connect the shield ground to the sensitive analog reference (also called ground). Rather, what's usually done is the shield connects directly to the chassis, and the ground of the device circuit also connects to the chassis (usually at the power supply). This ensures that noise (AC) on the shield is drained to the chassis, not to the analog circuit reference (ground). Note that it works because the analog circuit ground and the chassis ARE connected in a DC sense.

Many books have been written about this. See, for example, Henry Ott's work.

-a