I'm currently developing application that has to send high number of outbound HTTP/s requests within very short time frame. This poses a serious problem in terms of relability of networking (TCP) that i'd like to ask about.

Mine implementation is following:

We have scheduled 5 processes to act as workers node.js in crontab and send 800 outgoing requests each - scheduled every minute. Node works is event-driven, therefore requests are sent async in non-blocking way.

* * * * * /usr/bin/node script.js 800 results_proc1_txt.txt
* * * * * /usr/bin/node script.js 800 results_proc2_txt.txt
* * * * * /usr/bin/node script.js 800 results_proc3_txt.txt
* * * * * /usr/bin/node script.js 800 results_proc4_txt.txt
* * * * * /usr/bin/node script.js 800 results_proc5_txt.txt

All requests are being executed just fine, but the results are totally unreliable in terms of networking.

One time a job has 10 successfull requests in the row, then suddenly we're having problem establishing outbound connection and some socket issues.

2019-07-17T19:37:11.547Z OK 200
2019-07-17T19:38:10.105Z OK 200
2019-07-17T19:39:17.155Z OK 200
2019-07-17T19:40:11.257Z OK 200
2019-07-17T19:41:13.161Z OK 200
2019-07-17T19:42:14.425Z OK 200
2019-07-17T19:43:19.444ZFAIL socket hang up
2019-07-17T19:44:13.355Z OK 200
2019-07-17T19:45:15.793Z OK 200

I have increased limit of open files using common advice found on the internet.

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 15494
max locked memory       (kbytes, -l) 65536
max memory size         (kbytes, -m) unlimited
open files                      (-n) 65535
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 65535
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Is there any way how we can solve reliability issue? Like by tuning TCP parameters or similar?

  • 1
    I don't think there's enough information to reasonably guess at the issue here. How do you know that this isn't simply that whatever process / service these 4000 requests per minute are connecting to isn't simply overloaded by the sheer number of requests you're sending at once? That's over 65 requests/second sustained every minute. socket hang up simply means it didn't get a response at time. Not that the network is at fault (althought it could be). – yoonix Jul 17 at 21:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.