I am not sure if you have four CPU's. you can check cat /proc/cpuinfo for that.
the nginx worker_process should be set one less than the no of processors on your machine. So if you have two cpu's set it to 1.
each worker_process can handle the no of connections mentioned in the worker_connections that should be 1024 by default.
So max cleints that a nginx web server can handle is = worker_process * worker_connections. Also keep in mind the keep alive connections if you have enabled that in nginx.
So from my point of view nginx is not problematic here.
You should read carefully the max_children and parent processes that should be forked by php-fpm according to your server CPU/RAM. then set it accordingly in php-fpm.conf
If you have set 256M in varnish that means you are only left with 256M of RAM on server for other server tasks, which is very less.
You can set varnish's cache to file system instead of RAM. the performance won't be that bad but you will save your RAM on this.
Try this and see how it works.