Well, you have practically all the time over 8G either free (yellow, "unused") or as a cache (dark blue). In my opinion, your upper limit to allocate for memcached is maybe 6GB (give or take), which will still allow quite a lot of headroom for other processes and some disk cache without real risk of swapping. One could also argue you could allocate more. It's a matter of taste and real performance benefits will depend on your specific application.
Keep in mind that if you tell memcached to use up to 6GB memory, it won't use that much unless you store 6G in it. It will only consume 1GB if you only store 1GB in it, even if the limit is 6GB. I'd recommend you add memcached to your monitoring, and keep an eye on memcached's
bytes statistic and limit the memory usage accordingly to make sure the system will not do anything unexpected if there's surprise surge of users.
From shell you can use
free to determine memory usage, and it'll tell something like this:
total used free shared buffers cached
Mem: 4060096 3775976 284120 0 270828 1899720
-/+ buffers/cache: 1605428 2454668
Swap: 577528 0 577528
Free is unused memory, cached is disk cache. Free memory will get used as a disk cache over time, so it's quite safe to consider memory used as a cache as free, even if it's being used for something useful at the moment.