Someone mentioned sar (system activity reporter) and I have to second that recommendation!
It sounds like you've done a boatload of work on your own with various tools. However, you might have found that most of the aggregate system data you're interested in has been collected by sadc (system activity data collector) cron jobs for as long as you've had the sysstat package installed on your systems.
'
Oddly enough these don't run as daemons.
The sadc scripts (named sa1 and sa2) collect information from various kernel counters ... some of them are the same that are queried by vmstat and others are pulled from the same sources that ifconfig uses to report on the amount of data transmitted and received on each interface, and so on. By default sa1 runs every 10 minutes.
The sar command can then be used to pull various subsets of this information from the collected data. To become familiar with all available tables run sar -A and redirect the output to some convenient file. Then open the file in your favorite editor, in one window, and bring up the sar man page in another. Now simply walk through the entire file, searching the man page for the various column heading identifiers.
I also like to use sar -A as an early step for troubleshooting unexplained performance and stability. (Often by the time such issues have been escalated to me the system has already been rebooted or the problem is intermittent). This data is often the best available since the user (including most of the junior and intermediate sysadmins I've worked with) don't know what sort of data to collect before rebooting.
The trick to using this data for post-mortem troubleshooting is to scan all those columns of numbers looking for the spikes. (Yes, if you're good with tools like GNUplot you could generate a bunch of graphs from them; but honestly you can often spot the spikes just as well just scanning along the raw numbers). Look at a few similar systems which are not exhibiting the issues, or look at the data on this system during intervals when the things seem to be fine. Once you spot the "spikes" (network traffic, paging activity, disk I/O) try to correlate them (I usually just make a copy of the data and delete all the "nominal" data to leave me with just the stuff that's probably probative). The first correlation to look for is in the timing of different resource spikes ... did the run queue suddenly spike just before paging climbed out of control? ... was there a huge increase in received network traffic just before the disk I/O numbers went through the roof? Then you correlate that to the output from the last command (who was logged in around then) ... and if you've enabled process accounting then you can look at the lastcomm command. And, naturally, you can also look at /var/log/* to focus on the same time interval (and scanning backwards from there, of course).
You'd be amazed at how much you can figure out from this exercise.
"Tom account logged in, a few minutes later that were gradually increasing data flows over eth0 and to the sdb3 disk; perhaps he was copying in a data set, probably from the Netapps over on Lan B ... then we see a huge spike in memory page allocations, but process creation remained nominal until this paging started; that's when the interrupts and run queue went through the room, and brought the system into OOM thrashing. Tom, you ignoramus! You have a memory leak in foobar.pl!" :)
However, for your purposes you can use sar -A just as a starting point. You find the bits that you need and then you call sar with various arguments to capture each table you want to parse, filter, and graph separately. There are options to render the output in a more machine friendly format, as well.
