This is perfectly normal.
At system startup, a number of services start. These services initialize themselves, read in configuration files, create data structures and so on. They use some memory. Many of these services will never run again for the entire time the system is up because you're not using them. Some of them may run in hours, days, or weeks. Yet all this data is in physical memory.
Of course, the system can't throw this data away. It can't prove that it will literally never be accessed. One of those services, for example, might be the one that provides you remote access to the box. You may not have used it in a week, but if you do use it, it had better work.
But the system knows that it might like to use that physical memory for things like a disk cache or in other ways that will improve performance. So it does opportunistic swapping. When it has nothing better to do, it writes data that hasn't been used in a very long time to disk, using swap space. However, it still keeps the pages in physical memory. So they can still be accessed without having to swap them in.
Now, if the system later needs that physical memory for something else, it can simply throw those pages away because it has already written them to swap. This gives the system the best of both worlds. The data is still kept in memory, so it can be accessed without having to read it from disk. But if the system needs that memory for another purpose, it won't have to write it out first. Big win all around.