You are seeing an error of 173ppm. Unless you are using some strange bargain crystal there is no way the only problem is the crystal.

I had a look at a typical crystal (not high cost, not bargain basement) and it was specified at +-30ppm, so 173ppm is NOT the crystal.

Look at the RTC, does it run when you talk to it?
Look at your ISR, does it take overhead into consideration.
look at the rest of your code.

Erik