For the most part, memory management for Xen guests is no different than if you were running on bare metal. You're using swap when the various tools (
cat /proc/meminfo, etc.) tell you you are. In your case, your system is not using swap.
It sounds like you probably need to read Linux Ate My RAM! to brush up on the basics of how the linux kernel uses your system memory.
You really don't want un-used memory on your system - it's a waste of money. The kernel will use un-allocated pages for disk read cache, which greatly improves the IO performance of your system.
With regards to the "sluggish" feel of your system, that could be caused by a multitude of things, all of which would require you to gather data (IOps, IO latency, IOwait, load average, CPU load, etc.) over time. To ease this process, I'd highly suggest that you install a monitoring package like Munin. Munin excels at this sort of thing, and is very easy to set up.