Actually you can get so close to precision with the resistors shown that the trim pot should not even be a consideration. I would only consider a trimpot for adjusting things that you have NO control over. In this simple circuit you are in total control through component selection. I agree that the idea to use two unity gains buffers like you suggest should work nicely. You could also look at a opamp summer circuit where you incorporate the attenuator and offset into the opamp input and feedback resistors. Use low offset and quality opamps. Linear Tech makes some of the ones I like best.

Also take a look at your bandwidth requirements. If you have a higher bandwidth and there is come capacitance on the traces going to the A/D input you may want to scale the attenuator resistors down by some factor to reduce the R/C settling time. On the other hand maybe that is not of concern and maybe you'll be adding additional filtering to prevent anti-aliasing.

Good Luck
Michael Karas