Very probably, your input is too high ohmically? So, when you are close to the circuit, you look like an antenna, coupling hum into the circuit. When you touch the ground, then this 'antenna' is grounded, no longer being able to couple relevant hum into the circuit.
The reason for this coupling is stray capacitance between your body and the input of circuit. Although very small, it's enough to produce relevant noise, if the input impedance is very high.
Remedy: Decrease input inpedance to absolute minimum. Reduce input line to shortest length, unless you shield the circuit by putting it into a grounded enclosure or by using shielded cable with connecting the shield to signal ground.

Another possible explaination for your problem could be the invoke of oscillation of LM324 if you are close to the circuit and are not grounded. Then you also look like an antenna to the input, but now you couple high frequrency noise into it, or provide a capacitive short circuit from output to input, also causing oscillation. Here only the use of a metallic enclosure (Faraday cage), connected to signal ground can help. Enclosure need not directly to be connected to signal ground if high frequency interference is the issue. You can also connect it via 10...100nF capacitor.

Could you post a schematic of your circuit?

Kai