Munin shows ram that is doing nothing most of the time. But swapping still happens. Quite regularly actually, bogging down the hdd.
Or am I seeing the ram graph in the wrong way? How would I go about diagnosing this kind of problem?
Ok, let's look at this in more detail then.
From you image, I get that your server has a couple of moments during the day, where it reports a couple of pages being swapped out.
A memory page is 4KiB in size. That is 4096 bytes. The maximum amount of pages I see swapped out per second in your graph is 2. That is 8192 bytes per second for a period of maybe 10 to 15 minutes. Your harddrive can handle this. Really. This is not bogging down your harddrive. If it is, you really need another drive. Even simple harddrives are generally able to handle dozens of megabytes per second.
Now, as for why your server is swapping: it is pretty normal for a Linux system that has been on for a prolonged period of time to use some swap. Only pages that are hardly ever accessed are swapped out and you should not notice any negative effects when this happens.
Again, in your case, only a couple of megabytes have ever been in use as swap. I cannot relate the white parts in your graph to the minute amount of swapping that occured. Are you looking for that?
Sometimes Linux swaps even when there's enough RAM, if this is causing a problem you can
b) Reduce the size of the swap partition or get rid of it altogether. (I personally like to have a very small swap partition, many sysadmins like none at all, it depends of what kind of failure -oom vs swapping- is worse for you).