(posted as an answer, as it got far too long to add as an additional comment to Avery Payne's answer)
Also, if swap has been used during a peak period which has since passed you may find that most of the data in swap is also currently in RAM. If Linux reads pages back from swap in to RAM it doesn't immediately deallocate the swap space unless it needs it for more data, that way if it needs to swap the pages back out (and it knows that the data in them has not changed) it doesn't actually need to write the pages to disk as they are already there.
See /proc/meminfo for details. On a little server of mine currently:
olm:/proc# free -m
total used free shared buffers cached
Mem: 1487 1457 30 0 15 1094
-/+ buffers/cache: 347 1139
Swap: 980 112 868
olm:/proc# cat meminfo
MemTotal: 1523572 kB
MemFree: 30688 kB
Buffers: 15724 kB
Cached: 1120884 kB
SwapCached: 67868 kB
SwapTotal: 1004052 kB
SwapFree: 888928 kB
So here ~66Mb of the 112Mb of swap space allocated is also currently present in RAM. There is no point it removing that 66Mb from swap as there is no demand for the space for other uses (there is plenty of completely free swap space). If swap gets full those pages will be reallocated, if the pages change in RAM they will be marked as dirty and free to be reallocated, but if they need to be swapped back out the kernel can save itself a bunch of disk writes.
If I force the disk caches and buffers to be cleared with
sync; echo 3 > /proc/sys/vm/drop_caches
the result stays the same:
olm:/proc# free -m
total used free shared buffers cached
Mem: 1487 1278 209 0 0 979
-/+ buffers/cache: 298 1189
Swap: 980 112 868
olm:/proc# cat meminfo
MemTotal: 1523572 kB
MemFree: 212320 kB
Buffers: 652 kB
Cached: 1005732 kB
SwapCached: 67868 kB
SwapTotal: 1004052 kB
SwapFree: 888928 kB
The "cached" reading stayed high after asking for the disk caches to be cleared as this is counts memory allocated to VMWare VMs too because of the way that is allocated. The SwapCached reading didn't alter as there is no point copying the pages back into RAM just because the RAM is now free - they might never be needed before the RAM gets allocated for something else again so those reads would be wasted.
The situation above is slightly different to yours in that this machine pretty much always has all its RAM allocated to something (VMs, other processes, I/O cache+buffers) but depending on your machine's load history since last boot it is not unlikely that a chunk of the pages in allocated space in your swap area are similarly also in RAM.