I'm currently developing application that has to send high number of outbound HTTP/s requests within very short time frame. This poses a serious problem in terms of relability of networking (TCP) that i'd like to ask about.
Mine implementation is following:
We have scheduled 5 processes to act as workers node.js in crontab and send 800 outgoing requests each - scheduled every minute. Node works is event-driven, therefore requests are sent async in non-blocking way.
* * * * * /usr/bin/node script.js 800 results_proc1_txt.txt * * * * * /usr/bin/node script.js 800 results_proc2_txt.txt * * * * * /usr/bin/node script.js 800 results_proc3_txt.txt * * * * * /usr/bin/node script.js 800 results_proc4_txt.txt * * * * * /usr/bin/node script.js 800 results_proc5_txt.txt
All requests are being executed just fine, but the results are totally unreliable in terms of networking.
One time a job has 10 successfull requests in the row, then suddenly we're having problem establishing outbound connection and some socket issues.
2019-07-17T19:37:11.547Z OK 200 2019-07-17T19:38:10.105Z OK 200 2019-07-17T19:39:17.155Z OK 200 2019-07-17T19:40:11.257Z OK 200 2019-07-17T19:41:13.161Z OK 200 2019-07-17T19:42:14.425Z OK 200 2019-07-17T19:43:19.444ZFAIL socket hang up 2019-07-17T19:44:13.355Z OK 200 2019-07-17T19:45:15.793Z OK 200
I have increased limit of open files using common advice found on the internet.
core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 15494 max locked memory (kbytes, -l) 65536 max memory size (kbytes, -m) unlimited open files (-n) 65535 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 65535 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
Is there any way how we can solve reliability issue? Like by tuning TCP parameters or similar?