1

From a server on AWS, I push data into AWS ES service. This data appears on Kibana just fine. I am also able to perform cleanup via cURL commands from this machine. However, no machine of mine is able to run ElasticSearch Curator successfully and connect to AWS ES.

...jects/elasticsearch-curator❯ curator --host HOSTNAME.us-east-1.es.amazonaws.com delete indices --older-than 12 --time-unit days --timestring %Y.%m.%d
2016-12-18 16:54:08,631 INFO      Job starting: delete indices
2016-12-18 16:58:10,143 ERROR     Connection failure.

My access policy allows me to access this machine:

~❯ curl HOSTNAME.us-east-1.es.amazonaws.com
{
  "name" : "Exodus",
  "cluster_name" : "########",
  "version" : {
    "number" : "2.3.2",
    "build_hash" : "##########",
    "build_timestamp" : "2016-11-14T15:59:50Z",
    "build_snapshot" : false,
    "lucene_version" : "5.5.0"
  },
  "tagline" : "You Know, for Search"
}

and run commands:

~❯ curl -XDELETE 'HOSTNAME.us-east-1.es.amazonaws.com/2016-11*'
{"acknowledged":true}%

I am using the version of ES curator stated to be working with AWS ES:

...jects/elasticsearch-curator❯ pip list | grep curator
elasticsearch-curator (3.5.1)

As per answer (1) I added a port setting: "--port 80" the connection still fails, just much faster:

curator --host HOSTNAME --port 80 delete indices --older-than 12 --time-unit days --timestring %Y.%m.%d
2016-12-19 08:28:18,309 INFO      Job starting: delete indices
2016-12-19 08:28:18,324 ERROR     Connection failure.

and here is the output for debug mode:

2016-12-19 11:25:59,247 DEBUG         curator.api.filter         get_date_regex:157  regex = \l\o\g\s\t\a\s\h\-\d{4}\.\d{2}\.\d{2}
2016-12-19 11:25:59,247 DEBUG          curator.cli.utils        filter_callback:195  REGEX = (?P<date>\l\o\g\s\t\a\s\h\-\d{4}\.\d{2}\.\d{2})
2016-12-19 11:25:59,247 DEBUG          curator.cli.utils        filter_callback:198  Added filter: {'pattern': '(?P<date>\\l\\o\\g\\s\\t\\a\\s\\h\\-\\d{4}\\.\\d{2}\\.\\d{2})', 'value': 12, 'groupname': 'date', 'time_unit': 'days', 'timestring': u'logstash-%Y.%m.%d', 'method': 'older_than'}
2016-12-19 11:25:59,247 DEBUG          curator.cli.utils        filter_callback:199  New list of filters: [{'pattern': '(?P<date>\\l\\o\\g\\s\\t\\a\\s\\h\\-\\d{4}\\.\\d{2}\\.\\d{2})', 'value': 12, 'groupname': 'date', 'time_unit': 'days', 'timestring': u'logstash-%Y.%m.%d', 'method': 'older_than'}]
2016-12-19 11:25:59,248 INFO      curator.cli.index_selection                indices:62   Job starting: delete indices
2016-12-19 11:25:59,248 DEBUG     curator.cli.index_selection                indices:65   Params: {'url_prefix': u'', 'http_auth': None, 'dry_run': False, 'certificate': None, 'loglevel': u'INFO', 'logformat': u'default', 'client_cert': None, 'host': u'HOSTNAME', 'quiet': False, 'timeout': 30, 'debug': True, 'use_ssl': False, 'logfile': None, 'master_only': False, 'port': 80, 'ssl_no_validate': False, 'client_key': None}
2016-12-19 11:25:59,248 DEBUG          curator.cli.utils             get_client:114  kwargs = {'url_prefix': u'', 'http_auth': None, 'dry_run': False, 'certificate': None, 'loglevel': u'INFO', 'host': u'HOSTNAME', 'quiet': False, 'port': 80, 'logformat': u'default', 'timeout': 30, 'debug': True, 'use_ssl': False, 'logfile': None, 'master_only': False, 'client_cert': None, 'ssl_no_validate': False, 'client_key': None}
2016-12-19 11:25:59,248 DEBUG         urllib3.util.retry               from_int:191  Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0)
2016-12-19 11:25:59,248 DEBUG     urllib3.connectionpool              _new_conn:212  Starting new HTTP connection (1): HOSTNAME
2016-12-19 11:25:59,584 DEBUG     urllib3.connectionpool          _make_request:400  http://HOSTNAME:80 "GET / HTTP/1.1" 200 None
2016-12-19 11:25:59,586 INFO               elasticsearch    log_request_success:66   GET http://HOSTNAME:80/ [status:200 request:0.338s]
2016-12-19 11:25:59,586 DEBUG              elasticsearch    log_request_success:68   > None
2016-12-19 11:25:59,587 DEBUG              elasticsearch    log_request_success:69   <
<!DOCTYPE html>
<html lang="en" charset="utf-8">
<head>
<title>Sign In</title>
<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1, user-scalable=no">
<style>
body {
         font-family: "Helvetica Neue",Helvetica,Arial,sans-serif;
font-size: 14px;
line-height: 1.42857143;
color: #333;
background: #f0f0f0;
}
.signin {
    display:block;
margin:20px auto;
max-width:400px;
background: #fff;
border:1px solid #ccc;
border-radius: 10px;
padding: 20px;
}
.center {
    text-align:center;
}
.btn {
    color: #fff;
        background-color: #428bca;
border: 1px solid #357ebd;
            -webkit-border-radius: 4;
-moz-border-radius: 4;
border-radius: 4px;
font-size: 14px;
padding: 6px 12px;
text-decoration: none;
cursor: pointer;
}

.btn:hover {
    background-color: #3071a9;
        border-color: #285e8e;
ext-decoration: none;
}
label {
    display: inline-block;
max-width: 100%;
margin-bottom: 5px;
font-weight: 700;
}
input {
    display: block;
width: 100%;
height: 34px;
padding: 6px 12px;
font-size: 14px;
line-height: 1.42857143;
color: #555;
background-color: #fff;
background-image: none;
border: 1px solid #ccc;
border-radius: 4px;
-webkit-box-shadow: inset 0 1px 1px rgba(0,0,0,.075);
box-shadow: inset 0 1px 1px rgba(0,0,0,.075);
-webkit-transition: border-color ease-in-out .15s,-webkit-box-shadow ease-in-out .15s;
-o-transition: border-color ease-in-out .15s,box-shadow ease-in-out .15s;
transition: border-color ease-in-out .15s,box-shadow ease-in-out .15s;
margin:0;
box-sizing: border-box;
}
footer {
    display:block;
font-size:10px;
color:#aaa;
text-align:center;
margin-bottom:10px;
}
footer a {
    display:inline-block;
height:25px;
line-height:25px;
color:#aaa;
text-decoration:underline;
}
footer a:hover {
    color:#aaa;
}
</style>
</head>
<body>
<div class="signin center">
<form method="GET" action="/oauth2/start">
<input type="hidden" name="rd" value="/">

<p>Authenticate using @crosswise.com</p>

<button type="submit" class="btn">Sign in with a Google Account</button><br/>
</form>
</div>


<div class="signin">
<form method="POST" action="/oauth2/sign_in">
<input type="hidden" name="rd" value="/">
<label for="username">Username:</label><input type="text" name="username" id="username" size="10"><br/>
<label for="password">Password:</label><input type="password" name="password" id="password" size="10"><br/>
<button type="submit" class="btn">Sign In</button>
</form>
</div>

<script>
if (window.location.hash) {
(function() {
var inputs = document.getElementsByName('rd');
for (var i = 0; i < inputs.length; i++) {
inputs[i].value += window.location.hash;
}
})();
}
</script>
<footer>

Secured with <a href="https://github.com/bitly/oauth2_proxy#oauth2_proxy">OAuth2 Proxy</a> version 2.2.0-alpha

</footer>
</body>
</html>

2016-12-19 11:25:59,587 ERROR          curator.cli.utils             get_client:140  Connection failure.

So I added a few diagnostic lines in the appropriate place in utils.py:

try:
    client = elasticsearch.Elasticsearch(**kwargs)
    check_version(client)
    # Verify the version is acceptable.
    logger.debug('version is acceptable')
    check_master(client, master_only=master_only)
    # Verify "master_only" status, if applicable
    logger.debug('Verify "master_only" status')
    return client
except Exception as e:
    import traceback
    logger.error('Connection failure.')
    logger.error(e)
    traceback.print_exc()
    sys.exit(1)

and we get:

2016-12-19 11:42:37,345 ERROR          curator.cli.utils             get_client:143  Connection failure.
2016-12-19 11:42:37,345 ERROR          curator.cli.utils             get_client:144  Unknown mimetype, unable to deserialize: text/html
Traceback (most recent call last):
  File "/Users/gilzellner/.virtualenvs/temp/lib/python2.7/site-packages/curator/cli/utils.py", line 134, in get_client
    check_version(client)
  File "/Users/gilzellner/.virtualenvs/temp/lib/python2.7/site-packages/curator/cli/utils.py", line 89, in check_version
    version_number = get_version(client)
  File "/Users/gilzellner/.virtualenvs/temp/lib/python2.7/site-packages/curator/api/utils.py", line 202, in get_version
    version = client.info()['version']['number']
  File "/Users/gilzellner/.virtualenvs/temp/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 69, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/Users/gilzellner/.virtualenvs/temp/lib/python2.7/site-packages/elasticsearch/client/__init__.py", line 220, in info
    return self.transport.perform_request('GET', '/', params=params)
  File "/Users/gilzellner/.virtualenvs/temp/lib/python2.7/site-packages/elasticsearch/transport.py", line 357, in perform_request
    data = self.deserializer.loads(data, headers.get('content-type'))
  File "/Users/gilzellner/.virtualenvs/temp/lib/python2.7/site-packages/elasticsearch/serializer.py", line 74, in loads
    raise SerializationError('Unknown mimetype, unable to deserialize: %s' % mimetype)
SerializationError: Unknown mimetype, unable to deserialize: text/html
0

Curator is attempting to connect to port 9200, the default ES port. AWS ES is on port 80. Add --port 80 to your command line.

  • curator --host HOSTNAME --port 80 delete indices --older-than 12 --time-unit days --timestring %Y.%m.%d 2016-12-19 08:28:18,309 INFO Job starting: delete indices 2016-12-19 08:28:18,324 ERROR Connection failure. – Gil Zellner Dec 19 '16 at 8:30
  • so err, didnt work. but it does fail faster, which is an improvement :) – Gil Zellner Dec 19 '16 at 8:32
  • Try running tcpdump while you run curator vs curl, see what it is doing different connection-wise. – Jason Martin Dec 19 '16 at 15:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.