Looking at the chart, it seems to me like the periods of high CPU utilization happen on whole or half hours, which points toward something running through cron. If you want to start investigating through that route, that's where I would start.
That said, given that the bursts of high CPU utilization are fairly long (ten minutes in your case), you could simply dump system state through a cron job of your own to analyze in the morning.
I'd suggest using
top's batch mode (
-b) with an iteration count of 1 (
-n 1), so you'd add something like the following to a new file in /etc/cron.d:
* * * * * root top -b -n 1 > /var/log/processes/$(date +%F_%R)
Remember to create a directory /var/log/processes for this to write into.
In the morning, pick out one or a few of these files from a period of high CPU utilization and look more closely at them.
The default sort order appears to be highest current CPU utilization on top, which should work well in your case. Otherwise, you can simply import the file into a spreadsheet application (it should import just fine as fixed-width data) and sort on the %CPU field. Once you know the name of the binary, you can either look for it on disk (including using
dpkg -S to figure out what package it belongs to), or amend the above with something like
pstree to get an idea of what led to that particular binary being invoked.
That, in turn, should give you a good idea of which actual programs (not just processes) are hogging CPU, allowing you to determine whether it's even a problem, and if you think it is a problem then allow you to figure out how to turn it off.