If your running APC 3.0.6 like is suggested in W3 Total Cache installation instructions your probably getting the
potential cache slam avoided errors that fill up your logs very quickly.
Uninstall APC and rebuild using version 3.1.7 (3.1.9 is current stable) which fixes the cache slam bug and numerous other issues.
You will also get better results running latest PHP 5.3.6 every version gets better at interacting with APC (they will be completely integrated when PHP 6 comes out).
Also take a look at your config file and the apc.slam_defense setting and raise it to around 30 or 40.
For a real performance boost set apc.stat to 0 and it removes the process of checking for a new version of the file. If you make a lot of changes to your site they won't update till the cache expires. (My users get to frustrated so I have to keep it set at 1)
setting apc.slam_defense to 75 would
mean that there is a 75% chance that
the process will not cache an uncached
file. So, the higher the setting the
greater the defense against cache
slams. Setting this to 0 disables this
Lower the apc.user_ttl which is what W3 Total Cache uses and it the more dynamic stuff. Keep the apc.ttl high because this will be your WordPress core files and theme files that you don't really change.
I assume your setting object and db cache in W3 to APC. For your server size I would set page caching to disk enhanced. Also remember you can set the ttl for the dynamic stuff in W3.
Here's mine running a pretty large WordPress Multisite on a dedicated server with 12G Ram. Don't be afraid to increase the cache size even with limited memory. You can always go back down if needed. Mine is set at 512M but it never uses more than around 120M.
; configuration for php apc module
extension = apc.so
apc.enabled = 1
apc.shm_segments = 1
apc.shm_size = 512M
apc.optimization = 0
apc.num_files_hint = 2700
apc.user_entries_hint = 2700
apc.ttl = 7200
apc.user_ttl = 3600
apc.gc_ttl = 600
apc.cache_by_default = 1
apc.slam_defense = 1
apc.use_request_time = 1
apc.mmap_file_mask = /dev/zero
apc.file_update_protection = 2
apc.enable_cli = 0
apc.max_file_size = 2M
apc.stat = 1
apc.write_lock = 1
apc.report_autofilter = 0
apc.include_once_override = 0
apc.rfc1867 = 0
apc.rfc1867_prefix = "upload_"
apc.rfc1867_name = "APC_UPLOAD_PROGRESS"
apc.rfc1867_freq = 0
apc.localcache = 1
apc.localcache.size = 1350
apc.coredump_unmap = 0
apc.stat_ctime = 0