Right now I'm using nginx together with APC for php cache and memcache for mysql cache. I was wondering what Varnish cache can do in my existing environment as I am thinking to implement varnish, any thoughts?
Varnish is typically setup to store cached copies of data in memory, and serve them directly from there. As a result, it should be allocated a significant amount of memory for it to function properly. Varnish can be setup to use a disk backed cache, but this loses some of the benefit that it offers.
In comparison to nginx, you will not notice a significant difference in performance for serving small, frequently accessed, static files if you use Varnish. The reason being that a well setup nginx will serve the static files directly (i.e. will not pass the request to PHP), and your operating system caches the disk requests (meaning that they are essentially being served from memory anyway).
Where you can get a significant benefit from Varnish is if you can cache your dynamic content. Consider an article or home page which does not change that often relative to the amount it is viewed (even if it changes every 5 minutes, but receives a 100 views in that time, the change is minimal). The advantage here is that Varnish can cache a copy of your dynamic page, and serve it directly to your users. This greatly reduces the necessity to regenerate dynamic pages, which tend to be a computationally expensive operation. (This not only results in less load going to PHP but also reduces the load on MySQL - the result being that requests that do need to be processed by the backend can be returned faster).
When well set up, Varnish will allow you to purge individual files from your cache as updates to them are available - which prevents you from serving stale copies. Moreover, Varnish allows for fairly fine grained control over how caches are stored - which allows you to not cache certain pages (e.g. those which are specific to logged in users, etc), cache pages based on requester IP addresses, change the headers, etc.
To recap: in your environment (nginx + php-fpm + APC), the benefit Varnish can offer is to reduce the requests for dynamic pages that reach the backend. If you are only going to use it for static assets, I would suggest you just stick with nginx.
In a different setup (e.g. Apache) serving the static assets from Varnish would make a lot of sense.
Even if you have the memory to spare, the performance improvement offered by Varnish for static assets, compared to nginx is not nearly as pronounced as it is for dynamic pages.
(As a point of mention, nginx does offer a fastcgi_cache, but it doesn't provide the extent of options that Varnish does, which, I feel makes Varnish better suited to caching dynamic pages).
PHP is an interpreted language - you write scripts that are then run through the interpreter and converted to a form understandable by the underlying operating system. APC is an opcode cache - it is specific to PHP (although, similar products may exist for other languages).
Essentially, APC stores the intermediate products of the 'compilation' process so that subsequent calls to the same code can be processed faster. This means that if you run the same code multiple times, the first call will go through the full process of interpretation, while the next requests will be able to use the preprocessed code - which is what results in the speed-up that APC yields.
Moreover APC does not operate on full PHP files - but rather parts of the file - so if you use (e.g. via an include) the same function in two separate PHP scripts, the cached copy of the function will be of benefit to both scripts.
APC uses memory mapping - which means that its cached files can be either backed by shared/anonymous memory, or by actual files. A memory backed cache usually offers better performance, but does not persist across restarts.
Unlike Varnish, which stores the final, 'static', output generated by a script (i.e. that which is being sent to the user), APC stores a much more dynamic version - a function which yields different outputs for each input will benefit from APC, but not from Varnish.
APC and Varnish perform different functions and you certainly can use them both together, for the best benefit.
Considered another way - it will always be faster to not involve PHP/MySQL - static pages (essentially what a cached copy amounts to) are faster than dynamic pages, but when you do need to serve a dynamic page, you want to make it as fast as possible.
(A word of caution though - Varnish can be tricky to setup properly - by default it does not cache any pages that have cookies - this is to prevent serving restricted information to the wrong users. Improperly setup, Varnish can easily cache a copy of a page that should only be displayed for certain (e.g. logged in/admin) users and serve it to everyone.)
Varnish is particularly good at serving 'static' content.
eg. images, js, css
So if you have lots of static content on your site that rarely changes, you can skip needing to go through your whole server stack and just hotcut to Varnish.