time drift is a fact of life! You can get cheap watch crystals and you can get expensive watch crystals - pay more and they will be more accurate (normally!). I used to have a trimmer cap to trim the frequency but that is labour intensive or you can correct it in software - add/subtract a fixed value each day/week whenever. You can have the user enter the correction.

The average PC has poor timekeeping due to the same problem so don't think only you have the problem.