1

We have an Elasticsearch cluster hosted on Amazon Elasticsearch Service (AWS).

We are using the Jest Java HTTP Rest client for ElasticSearch.

Every now and then (perhaps 1 in 10,000 requests), it appears to close the connection without a response.

The stack trace in our app looks like:

ERROR [2016-04-11 09:18:43,497] io.dropwizard.jersey.errors.LoggingExceptionMapper: Error handling a request: b9b9ee1e4eefadd2
! org.apache.http.NoHttpResponseException: search-xxx.eu-west-1.es.amazonaws.com:443 failed to respond
! at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143) ~[my-app-0.0.1.jar:0.0.1]
! at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) ~[my-app-0.0.1.jar:0.0.1]
! at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) ~[my-app-0.0.1.jar:0.0.1]
! at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) ~[my-app-0.0.1.jar:0.0.1]
! at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167) ~[my-app-0.0.1.jar:0.0.1]
! at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) ~[my-app-0.0.1.jar:0.0.1]
! at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) ~[my-app-0.0.1.jar:0.0.1]
! at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271) ~[my-app-0.0.1.jar:0.0.1]
! at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[my-app-0.0.1.jar:0.0.1]
! at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[my-app-0.0.1.jar:0.0.1]
! at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[my-app-0.0.1.jar:0.0.1]
! at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[my-app-0.0.1.jar:0.0.1]
! at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[my-app-0.0.1.jar:0.0.1]
! at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107) ~[my-app-0.0.1.jar:0.0.1]
! at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:48) ~[my-app-0.0.1.jar:0.0.1]

The relevant code from "org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead" looks like:

final int i = sessionBuffer.readLine(this.lineBuf);
if (i == -1 && count == 0) {
    // The server just dropped connection on us
    throw new NoHttpResponseException("The target server failed to respond");
}

As far as I can tell, Amazon doesn't give me access to the Elasticsearch server's logs.

So:

  1. How can I diagnose and fix the cause of this error?
  2. If the best fix is for my app to retry these failures, is there an easy way to retry using Jest? I don't see any config options to do this automatically.

TIA

1 Answer 1

0

1: (Don't know yet)

2: You can configure Jest to retry Elasticsearch operations which fail with network errors like:

new JestClientFactory() {
    @Override
    protected HttpClientBuilder configureHttpClient(HttpClientBuilder builder) {
        builder = super.configureHttpClient(builder);

        // See DefaultHttpRequestRetryHandler.requestSentRetryEnabled
        //
        // true if it's OK to retry non-idempotent requests that have been sent
        // and then fail with network issues (not HTTP failures).
        //
        // "true" here will retry POST requests which have been sent but where
        // the response was not received. This arguably is a bit risky.
        //
        // Retries are logged at INFO level to org.apache.http.impl.execchain.RetryExec
        boolean requestSentRetryEnabled = true;

        builder.setRetryHandler(new DefaultHttpRequestRetryHandler(
                3,
                requestSentRetryEnabled));

        return builder;
    }
}

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .