Can someone explain OPcache memory usage in layman terms ?
ldennison plus the various commenters have probably/mostly answered this part.
Another great write-up can be found here: http://www.sitepoint.com/understanding-opcache/
Allow me to summarise:
OPcache allows you to pre-allocate a certain amount of memory. You have set it to 1024, ie. 1GB. This is you telling your system: "cache up to 1GB of compiled php script so the computer doesn't have to turn it form human-readable PHP code to 1's and 0's the computer can understand, each and every time it gets run." Every first time such a script then is run, its pre-compiled version gets added to this cache library. It wasn't in the cache yet on its first run, so this was a cache miss. If it is requested again, and already/still in cache, it gets fetched from cache, saving one 'costly' computing stage. This constitutes a cache hit.
Memory usage just means how much of your 1GB is being used. You see 100% use. That means you have 1GB of php scripts cached and ready to be served from this cache. The cache is full so either no new scripts can be added, or unpopular cache items get pruned, depending on the settings/engine.
Should I be worried about this having 100% Memory usage ?
If your cache hit rate was low, meaning, it was looking in the cache for a file repeatedly but couldn't find it (because it didn't fit in there for example), then you should 'worry'. But your cache hit rate is high, meaning almost every time it wants something from cache, it finds it. That means your 1024 MB cache size, 1/8th of your total server RAM, is very good.
Doubling your cache size will probably not change your cache hit rate significantly. Add 100 MB and it might be enough to go from 96% to 98%.
So the 'problem' you are seeing (100% memory use) doesn't look like a true problem. Yes, it's full, but it isn't missing out on much of value. At this size, it can serve 96% of its needs. That's a great statistic.
Look at it this way...
If a library if full to the rafters, with no more capacity for additional books, but it already carries 96% of all books in the world, is it worth to build an extension? Will it add to the library visitors satisfaction? 96 out of every 100 visitors already find what they want.