I was in training in a class taught by Ted Ts'o, and the way he explained it to me when I asked the same question was like this...
By default, the kernel reserves most of the memory for caching things like filesystem metadata. That's why your "used" column shows 905MB. That's the total memory that's "used", meaning by programs and by cache. The actual amount of free memory is under the "free" column, beside "-/+ buffers/cache", in other words, 561MB. That's the amount available for use by programs.
Now, since the kernel has 462MB used for cache, it's leaving 118MB completely available. When a program says to the kernel, "hey, I need 50MB of memory RIGHT NOW!", the kernel draws the memory from this "free" pool. That makes the "free" pool at around 68MB, which is too small for comfort, so after that memory is drawn, the amount of stuff stored in cache is lowered, and the "free" pool goes back up.
But what happens if a program is a hog, and says "I need 120MB of memory RIGHT NOW!". You don't have that much "free" memory, so the kernel can't give it to the program, even temporarily, so the kernel digs into swap, just long enough to allocate enough free memory to give the program what it needs, then enough cache data is released to allow the "free" memory to go back to a comfortable level.
Through the settling process, the data in swap doesn't get freed from the swap immediately, even though it's cached data. The only time the swap memory is going to get used is if something requests the metadata (or whatever cached information) that is stored in that little segment of swap. So there's no harm in leaving it.
If it bothers you, you can run 'swapoff' then 'swapon' to get rid of it, but it isn't hurting anything.