What hobodave said is half of the work. Do it then :
Install apc.php to monitor APC
Download http://pecl.php.net/get/APC and extract apc.php to a web folder on your server then point your browser at it. You will have a nice digram of APC memory usage.
In a correct configuration of APC this diagram must remain almost stable after hours of running (so all php pages have been cached). Memory Fragmentation will occur as developers updates php files.
Setup large amount of memory
APC is saving CPU cycles by using lot of memory. So give it lot of ram. A good practice is to give it 20 % more memory than it actually use.
Do this by increasing apc.shm_size in php.ini and restart httpd. If your server have enough ram start with 1000M (1GB).
APC could be compiled to either use Shared Segments memory or mmap memory. You can check it on apc.php > general information > Shared Memory
If your APC use mmap, then you can increase without problem. Otherwise the size of a segments is limited by OS. You'll have to set apc.shm_size to the maximum size allowed and increase the number of segments with apc.shm_segments option.
Type this command to see your system limit for each segment :
sysctl -a | grep -E "shmall|shmmax"
Once you've done that you must periodically check that APC still have enough ram.
Correctly configure TTL
Set apc.ttl to 7200 (recommended value). Lot of admins use a ttl of 0 to avoid the following common error
Warning: require_once() [function.require-once]: Unable to allocate memory for pool. in /path/to/file
Using a TTL of 0 means that APC will flush all the cache when it runs out of memory. The error don't appear anymore but it makes APC far less efficient. It's a no risk, no trouble, "I don't want to do my job" decision. APC is not meant to be used that way. You should choose a TTL high enough so the most accessed pages won't expire. The best is to give enough memory so APC don't need to flush cache.
Just read the manual to understand how ttl is used : http://www.php.net/manual/en/apc.configuration.php#ini.apc.ttl
Why APC use so much ram ?
APC store PHP opcode in memory. Opcodes are bigger than source PHP. Because one php instruction is translated in lot of opcodes.
The default of allowing only 32MB is ridiculously low. PHP was designed when servers had 64MB and most script were using one php file per page. Nowadays solutions like Magento require more than 10k files (~60Mb in APC). You should allow enough memory so most of php files are always cached. It's not a waste, it's more efficient to keep opcode in ram rather than having the corresponding raw php in file cache. Nowadays we can find dedicated servers with 24Gb of memory for as low as $150, so don't hesitate to allow several GB to APC. I put 2GB out of 24GB on a server hosting 5 Magento stores and ~40 wordpress website, APC uses 1.2GB. Count 64MB per Magento installation, 20MB per Wordpress with some plugins.
My server don't have enough ram, what should I do ?
First consider upgrading it ! :)
If you are using php applications that support multisites (Magento, Wordpress), you should merge them in one instance. So you need to cache php file only once.
You can use apc.fiters to decide what files must be chached. You should cache most frequently accessed files.
Why should I use APC and bother with all that ?
APC is cool, it make your websites faster. But it's more than this, APC save CPU cyles, APC save power, APC is environmentally friendly (that why I spent time writing this configuration guide).