The issue may be one of several clocks or timers are particularly inaccurate or possibly defective oscillator or timer / clock (including RTC) passive components (quartz crystal, and any related capacitors). If this is the cause you may not be able to entirely work around it, if accurate time is critical to you.
The nVidia nForce2 chipset for motherboards is known to be very erratic. There is also the possibility that APIC and ACPI, Variable Speed Processors, and Front Side Bus (FSB) Spread Spectrum (SS) oscillators / clock issues effecting accuracy. One potential software (kernel) issue is with some versions of 2.6.x kernel mis-detecting the TSC (timestamp counter) frequency, and some AMD processors adjust TSC based on the processor's core clock speed (i.e for power-saving, similar to SpeedStep in Intel processors).
If you haven't already, look at NTP documentation for Troubleshooting problems with NTP.
If none of the above reduce the clock's erratic behaviors, then you can Manually Calibrate it using adjtimex and NTP. Be sure to delete any existing NTP drift files
/var/lib/ntp/ntp.drift on Ubuntu, may be located in /etc/ntp/ on some other Unix/Linux systems) or entries in ntp.conf.
Apologies for possibly stating the obvious, but ensure you have 3 or 4 reachable NTP servers included in the NTP config file (typically
/etc/ntp.conf or similar). And please use (public) servers "close" to you. Most ISP or network providers offer free access to their own internal NTP server, use it. otherwise use public servers from http://pool.ntp.org/, preferably located in your province/state or at least country.