As the question title suggests, I'm having an hard time to figure out what can be improved on my application(or tuned in the os, ubuntu) to achieve an acceptable performance. But first I'll explain the architecture:
I store persistent data(like users and scores) on a second server on mongodb(3.6) with 4gigs RAM and 2 cores.
The app is in production since few months(it's been running on a single box until few weeks ago) and it's being used by around 18k users per day. It has always worked very well apart of one main issue: performance degradation. With use, the amount of cpu used by each process grows until it staturates the worker(which won't serve requests anymore). I have temporarily solved it checking the cpu in use by each worker every minute, and restarting it if it reaches 98%. So the problem here is mainly cpu, and not RAM. The RAM is not an issue anymore since I've updated to socket.io 0.9.14(the earlier version was leaking memory) so I doubt it to be a memory leaking issue, especially because now it is the cpu that grows fairly quickly(I have to restart each worker around 10-12 times a day!). The RAM in use grows as well to be honest, but very slowly, 1 gig every 2-3 day of use, and the strange thing is that it is not released even when I completely restart the whole application. It is only released if I reboot the server! this I cannot really understand...
I've now discovered nodefly which is amazing, so I can finally see what's happening on my production server, and I'm collecting data since a couple of days. If anyone want to see the charts I can give you access, but basically I can see that I have between 80 and 200 concurrent connections! I was expecting node.js to handle thousands, not hundreds of requests. Also the average response time for http traffic floats between 500 and 1500 milliseconds which I think is really a lot. Also, in this very moment with 1300 users online, this is the output of "ss -s":
Total: 5013 (kernel 5533) TCP: 8047 (estab 4788, closed 3097, orphaned 139, synrecv 0, timewait 3097/0), ports 0 Transport Total IP IPv6 * 5533 - - RAW 0 0 0 UDP 0 0 0 TCP 4950 4948 2 INET 4950 4948 2 FRAG 0 0 0
which shows that I've got a lot of closed connections in timewait. I've increased the max open files to 999999, here is the output of ulimit -a:
core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 63724 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 999999 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 63724 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
So I thought the problem could be on http traffic that for some reasons saturates the available ports/sockets(?), but one thing does not make sense to me: why when I restart the workers, and all the clients reconnect within few seconds, the load on the cpu of the worker goes down to 1% and is capable of serving requests properly until it saturates after about 1 hour (at peak time)?
Hopefully there is something obvious that I'm doing wrong, and someone will help to spot it...feel free to ask me for more info, and I'm sorry for the length of the question but was necessary I believe...thanks in advance!