1

So I have been trying to figure this one out for about a week now. Here is the run down:

I use CURL in PHP to pull data from an API. As the response to the API call gets bigger (pulling over 15k records at once) I noticed that any API call that takes 5 or more minutes (within a few seconds) fails to return on my CentOS and Suse servers. So, I tested the API call from the CLI via CURL and I get the same issue. Oddly, if I run the CURL command via OS X the command runs fine and returns after ~7 or so minutes.

Here is the command (creds censored) I am running via CURL:

curl -m 0 -k --trace-ascii trace.txt --trace-time -X GET -H "tenant-code: 1cmPx7tqVDVTdN1GSelwycFUmICmASnLCmNQsV72" -H "Authorization: Basic JxHAsXeUiHMRkS8Msiu6pWb3PvY20p6am3QvXCY3knXTAntlxTBS3EyEDgly" -H "Content-Type: application/json" -H "Cache-Control: no-cache" 'https://api.endpoint.com/API/v1/system/users/search?groupid=555' > dump.txt

And here is the version output from CURL for each platform:

CentOS (this is where I really need this to work)-

curl 7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.19.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
Protocols: tftp ftp telnet dict ldap ldaps http file https ftps scp sftp 
Features: GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz 

Suse-

curl 7.19.7 (x86_64-suse-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8j zlib/1.2.7 libidn/1.10
Protocols: tftp ftp telnet dict ldap ldaps http file https ftps 
Features: GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz

OS X-

curl 7.37.1 (x86_64-apple-darwin14.0) libcurl/7.37.1 SecureTransport zlib/1.2.5
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smtp smtps telnet tftp 
Features: AsynchDNS GSS-Negotiate IPv6 Largefile NTLM NTLM_WB SSL libz 

And these are the error codes I get from Centos:

curl: (56) SSL read: errno -5961

I can't find that code referenced in the documentation. https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/SSL_functions/sslerr.html

I get a slightly different error from Suse:

curl: (52) SSL read: error:00000000:lib(0):func(0):reason(0), errno 104

The error 104 lead me to believe that the server is stopping/resetting the connection but the server side logs don't show it being dropped and OS X can pull the data without an issue. I even tried spoofing the User Agent to make sure that was not the issue.

So, at this point I assume that the SSL package SecureTransport is doing something that OpenSSL and NSS are not doing. The question is what and if not, what is the issue?

2

Run the curl command on the MacOSX machine but don't redirect the output, let it stream to your shell window. Watch to see if there appears to be any buffering involved, IE, do you get output from the start, a bit at a time, or do you get nothing for 5 minutes and then a flood of data all at once?

Run the curl command again on a machine where it times out, and compare the behavior. If your output is being buffered by some background process on the API server, you may not get results until it finishes its query. Something between your client application, your client's OS, the server's OS, the server's REST api, and the SSL between them likely has a timeout value of non-zero, and if that timer doesn't see any data flowing for 5 minutes, it may close your connection without saying much about why. I see this happen a lot in HTTP-based services. In perl I habitually put a $|=1; at the top of the code to disable output buffering on the server side.

It is also possible that a 3rd party device such as a Cisco ASA may have NAT rules timing out and triggering issues. I have this issue with AMANDA backups that are attempting to read from a client on the external side of an ASA. If the client takes too long to return its size estimates through the ASA back to the AMANDA server, the ASA drops its dynamic NAT rule and the backup fails. This suggestion is worth investigating if the MacOSX that works doesn't have a firewall between it and the API server but the ones that fail do have one.

It wouldn't surprise me in the least if MacOSX has a timeout value set somewhere to 0 (wait forever) where Linux defaults to something with a limit of 60 or 90 seconds.

  • I ran the CURL command on both boxes and I get silence before I get the error code or the returned results. I did spin up a VM of CentOS on OS X and the Suse server is in the same data center as the API server and both got the same result so I don't think it is a network or firewall issue. – Hybrid Aug 14 '15 at 19:18
  • Then my next experiment would be to use -o filename to get the content out of your way, and use --trace fileame and --trace-time to save debug output to a second file for inspection. I'd also experiment with the various time parameters on curl itself, see --connect-timeout, --max-time, and --keepalive-time. The last few lines of the trace file should be interesting. – user207998 Aug 14 '15 at 19:50
  • So, great progress here. I got a call to actually return the expected data. I added --max-time 600 -o dump.txt --connect-timeout 0 --keepalive-time 30 and removed -m 0 from the curl command. Now I just have to figure out how to transfer these to PHP CURL since setting just the max connect time and max time to 0 does not work. I suspect the keep alive time matters here. – Hybrid Aug 14 '15 at 21:01
  • Moved this to StackOverflow to figure that one out. stackoverflow.com/questions/32019448/… Thanks a ton @user207998 – Hybrid Aug 14 '15 at 22:21

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.