A few seconds per day of time drift is not particularly unusual or unexpected on a physical machine in my experience, especially if the drift is fairly stable (i.e. not 1s one day, 10s then next, -2s the day after, ...). In VMs, the clock drifting by only 4s/day would be quite an achievement without clock management like NTP.
Rather than jumping the time once per day with ntpdate, it is generally a much better idea to run
ntpd and have it keep the clock in sync using its much finer grained mechanism. Common "best practise" is to have one machine (or two) that you keep synced with external clock sources such as *.<local>.pool.ntp.org and then sync all your other machine against that - this reduces load on the public NTP servers (as only one or two of your machines are talking to them) and increases the accuracy of the sync between your machines (as they are syncing with the same, local, source).
If you Linux in any VMWare VMs and want to sync their time by ntp, make sure you are using a recent Linux kernel (that has the "tickless clock"), that VMWare's time sync is off, and that you have
tinker panic 0 at the top of the ntpd config so it doesn't give up if the clock skew changes fairly suddenly due to high load in the VM or on the host in general. I've found this the most reliable combination for keeping VM's clocks in order.