1

We have an Elasticsearch cluster hosted on Amazon Elasticsearch Service (AWS).

We are using the Jest Java HTTP Rest client for ElasticSearch.

Every now and then (perhaps 1 in 10,000 requests), it appears to close the connection without a response.

The stack trace in our app looks like:

ERROR [2016-04-11 09:18:43,497] io.dropwizard.jersey.errors.LoggingExceptionMapper: Error handling a request: b9b9ee1e4eefadd2
! org.apache.http.NoHttpResponseException: search-xxx.eu-west-1.es.amazonaws.com:443 failed to respond
! at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143) ~[my-app-0.0.1.jar:0.0.1]
! at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) ~[my-app-0.0.1.jar:0.0.1]
! at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) ~[my-app-0.0.1.jar:0.0.1]
! at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) ~[my-app-0.0.1.jar:0.0.1]
! at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167) ~[my-app-0.0.1.jar:0.0.1]
! at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) ~[my-app-0.0.1.jar:0.0.1]
! at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) ~[my-app-0.0.1.jar:0.0.1]
! at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271) ~[my-app-0.0.1.jar:0.0.1]
! at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[my-app-0.0.1.jar:0.0.1]
! at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[my-app-0.0.1.jar:0.0.1]
! at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[my-app-0.0.1.jar:0.0.1]
! at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[my-app-0.0.1.jar:0.0.1]
! at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[my-app-0.0.1.jar:0.0.1]
! at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107) ~[my-app-0.0.1.jar:0.0.1]
! at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:48) ~[my-app-0.0.1.jar:0.0.1]

The relevant code from "org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead" looks like:

final int i = sessionBuffer.readLine(this.lineBuf);
if (i == -1 && count == 0) {
    // The server just dropped connection on us
    throw new NoHttpResponseException("The target server failed to respond");
}

As far as I can tell, Amazon doesn't give me access to the Elasticsearch server's logs.

So:

  1. How can I diagnose and fix the cause of this error?
  2. If the best fix is for my app to retry these failures, is there an easy way to retry using Jest? I don't see any config options to do this automatically.

TIA

0

1: (Don't know yet)

2: You can configure Jest to retry Elasticsearch operations which fail with network errors like:

new JestClientFactory() {
    @Override
    protected HttpClientBuilder configureHttpClient(HttpClientBuilder builder) {
        builder = super.configureHttpClient(builder);

        // See DefaultHttpRequestRetryHandler.requestSentRetryEnabled
        //
        // true if it's OK to retry non-idempotent requests that have been sent
        // and then fail with network issues (not HTTP failures).
        //
        // "true" here will retry POST requests which have been sent but where
        // the response was not received. This arguably is a bit risky.
        //
        // Retries are logged at INFO level to org.apache.http.impl.execchain.RetryExec
        boolean requestSentRetryEnabled = true;

        builder.setRetryHandler(new DefaultHttpRequestRetryHandler(
                3,
                requestSentRetryEnabled));

        return builder;
    }
}

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.