1

Problem description

I have an executable which repeatedly posts data to an HTTPS endpoint, using libcurl with libcares. This is getting occasional DNS resolution timeouts on some clients (not all). If I run the equivalent command in command-line curl, I never see any timeouts.

What makes this even more confusing is that the host is explicitly specified in /etc/hosts, so there shouldn't be any DNS resolution required.

The error from libcurl (with verbose mode) is:

* Adding handle: conn: 0xcbca20
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 88 (0xcbca20) send_pipe: 1, recv_pipe: 0
* Resolving timed out after 2002 milliseconds
* Closing connection 88

My libcurl executable is sending 2-3 queries a second, and I see this error about once every 300 requests. Using command-line curl, I have run 10000 queries without a single timeout.

Can anyone suggest anything I can try to resolve these errors from libcurl? Are there any settings which I need to add to my libcurl setup, or system configuration I might be missing?

I wasn't sure whether to put this in Stack Overflow, Server Fault, or Ask Ubuntu; apologies if it is in the wrong place.

Thanks for your time!

More detailed information

The client is Ubuntu 12.04, 64-bit. The same problem has been observed on several clients, all with the same OS.

Usernames/passwords/urls have been obfuscated in the following snippets.

Command-line curl tester (using v 7.22.0):

while true; do curl -v -u username:password "https://myhost.com/endpoint" -X POST --data "a=x&b=y" >> /tmp/commandLine.log 2>&1; sleep 0.1; done &

Libcurl source code (using curl 7.30.0, with c-ares 1.10.0):

#include <curl/curl.h>
#include <unistd.h>
#include <string>
#include <iostream>

using namespace std;

int main(int argc, char** argv) {

   while (1) {

       // Initialise curl
       CURL *curl = curl_easy_init();

       // Set endpoint
       string urlWithEndpoint = "https://myhost.com/endpoint";
       curl_easy_setopt(curl, CURLOPT_URL, urlWithEndpoint.c_str());

       // Set-up username and password for request
       curl_easy_setopt(curl, CURLOPT_USERPWD, "username:password");

       // Append POST data specific stuff
       string postData = "a=x&b=y";
       long postSize = postData.length();
       curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE, postSize);
       curl_easy_setopt(curl, CURLOPT_COPYPOSTFIELDS, postData.c_str());

       //set timeouts
       curl_easy_setopt(curl, CURLOPT_TIMEOUT, 10);
       curl_easy_setopt(curl, CURLOPT_CONNECTTIMEOUT, 2);
       curl_easy_setopt(curl, CURLOPT_DNS_CACHE_TIMEOUT, 60);

       cout << endl << endl << "=========================================================" << endl << endl;
       cout << "Making curl request to " << urlWithEndpoint << "(POST size " << postSize << "B)" << endl;

       // Set curl to log verbose information
       curl_easy_setopt(curl, CURLOPT_VERBOSE, 1);

       // Perform the request
       CURLcode curlRes = curl_easy_perform(curl);

       // Handle response
       bool success = false;
       if (curlRes == CURLE_OK) {
           long httpCode;
           curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, &httpCode);
           success = (httpCode==200);
           cout << "Received response " << httpCode << endl;
       } else if ( curlRes == CURLE_OPERATION_TIMEDOUT ) {
           cout << "Received timeout" << endl;
       } else {
           cout << "CURL error" << endl;
       }

       curl_easy_cleanup(curl);

       if (success) {
           cout << "SUCCESS! (" << time(0) << ")" << endl;
           usleep(0.1*1e6);
       } else {
           cout << "FAILURE!!!! (" << time(0) << ")" << endl;
           usleep(10*1e6);
       }

   }

}
0

Answering my own question...

It turned out that the problem was in lib c-ares (which gives thread-safe DNS resolution in libcurl). Having recompiled libcurl with --enable-threaded-resolver instead, the resolution timeouts stop.

I've tried updating to the very latest lib c-ares, and looking on the c-ares forums for similar bugs, but no luck on either front. So I'm ditching c-ares and using the Curl threaded resolver going forward.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.