3

I got this task handed over to by my colleage and this is the background.

He got ELK (Elasticsearch, Logstash and Kibana) stack working with our RHEL 6.2 servers, by using the regular method of configuring the Logstash on the server and Logstash-forwarder on all the agents. Everything is working fine as expected, except for two logstash agent nodes. It is not indexing data from these two nodes and these two agent nodes are slightly different from the other agents in the environment. All the agents (that are working) are running RHEL 6.2 machine with 'OpenSSL 1.0.0-fips', while the two troublesome agents are running RHEL 5.5 with 'OpenSSL 0.9.8e-fips-rhel5'. Now, let me explain the issues that these two nodes were facing when I started working on this.

When we restart the logstash forwarder service with '/etc/init.d/logstash-forwarder restart', it throws the following error:

2014/10/01 09:21:23.898917 Failed to tls handshake with 10.x.x.x x509: cannot validate certificate for 10.x.x.x because it doesn't contain any IP SANs 2014/10/01 09:21:24.900502 Connecting to [10.x.x.x]:5000 (10.x.x.x) 2014/10/01 09:21:24.902081 Failed to tls handshake with 10.x.x.x x509: cannot validate certificate for 10.x.x.x because it doesn't contain any IP SANs 2014/10/01 09:21:25.903970 Connecting to [10.x.x.x]:5000 (10.x.x.x) 2014/10/01 09:21:25.905708 Failed to tls handshake with 10.x.x.x x509: cannot validate certificate for 10.x.x.x because it doesn't contain any IP SANs

I understand that this is because the agent is trying to connect to the server through SSL using its IP address, so I recreated the certificates again with IP SANs and then it started throwing me following errors, not only with the 5.2 machines, but also with 6.2 machines:

2014/10/03 17:09:12.403253 Failed to tls handshake with 10.x.x.x x509: certificate signed by unknown authority 2014/10/03 17:09:13.404974 Connecting to [10.x.x.x]:5000 (10.x.x.x) 2014/10/03 17:09:13.428156 Failed to tls handshake with 10.x.x.x x509: certificate signed by unknown authority 2014/10/03 17:09:14.429648 Connecting to [10.x.x.x]:5000 (10.x.x.x) 2014/10/03 17:09:14.442006 Failed to tls handshake with 10.x.x.x x509: certificate signed by unknown authority

I tried to add the certificate to the ca-bundle.crt by appending the new logstash certificate to the CA file with:

'openssl x509 -in certs/logstash-forwarder.crt -text >> certs/ca-bundle.crt'.

I have a feeling that I will be able to fix the whole issue if I am able to make the 'IP SAN'd certificates add to the trusted certificates list of all the agent nodes. My questions are:

One more thing, the Logstash server is running RHEL 6.2. And all the agent nodes (working and non-working) have the same version of logstash-forwarder installed - logstash-forwarder-0.3.1-1.x86_64.rpm. What is the correct method to make these agent nodes accept the IP SANs certificates? Am I understanding issue correctly? Is it some incompatibility between the RHEL 6.x and RHEL 5.x machines?

I have the following entries in the /etc/logstash-forwarder

[root@name2 ~]# cat /etc/logstash-forwarder { "network": { "servers": [ "10.x.x.x:5000" ], "timeout": 15, "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt", "ssl strict verify": "false" }, "files": [ { "paths": [ "/var/log/maillog" ], "fields": { "type": "syslog" } } ] } [root@name2 ~]#

Thanks in advance.

5

The certificate you are using does not have any valid IP SAN's as mentioned in the message:

Failed to tls handshake with x.x.x.x x509: cannot validate certificate for x.x.x.x because 
it doesn't contain any IP SANs  

If you connect using an IP address then your certificate must contain a matching IP SAN to pass validation with Go 1.3 and higher. This is not (yet?) mentioned in any README file or documentation but there are some issues ( #226 , #221 ) open at the project github repo.
To permit IP address as the server name, the SSL cert must include IP address as a subjectAltName field.

To solve that you can use following procedure for the creation of the SSL cert and key:

  1. Create a file notsec.cfr (or any other name) containing output like:

    [req]
    distinguished_name = req_distinguished_name
    x509_extensions = v3_req
    prompt = no
    
    [req_distinguished_name]
    C = TG
    ST = Togo
    L =  Lome
    O = Private company
    CN = *
    
    [v3_req]
    subjectKeyIdentifier = hash
    authorityKeyIdentifier = keyid,issuer
    basicConstraints = CA:TRUE
    subjectAltName = @alt_names
    
    [alt_names]
    DNS.1 = *
    DNS.2 = *.*
    DNS.3 = *.*.*
    DNS.4 = *.*.*.*
    DNS.5 = *.*.*.*.*
    DNS.6 = *.*.*.*.*.*
    DNS.7 = *.*.*.*.*.*.*
    IP.1 = 192.168.1.1
    IP.2 = 192.168.69.14  
    

If you connect via host names, you can remove the IP SAN's, otherwise add your logstash server IP address.

  1. Create the certificate and key with following command (using the file from point 1):

    openssl req -x509 -batch -nodes -newkey rsa:2048 -keyout notsecure.key -out notsecure.crt -config notsec.cnf -days 1825
    

This will create a kind of wildcard certificate accepting any hostname and the IP addresses mentioned it that file. Of course this is just a simple example and you wil need to adjust the settings to your needs.

  • Thank you. I did generate a certificate with IP SANs, but then the error turned into ' Failed to tls handshake with 10.x.x.x x509: certificate signed by unknown authority'. Is it because I did not use the correct options while generating the certificate with IP SANs? – Sree Oct 5 '14 at 18:23
  • You have not mentioned how you generated the key/certifcate. There could be some problem hidden. As far as I remember GO's X.509 by default only supports certificates signed with a SHA1 hash, so if you are using certificates signed using SHA256 or SHA512 it could cause some problems. I'm pretty sure that I had not done anything else besides these 2 steps mentioned in the answer + copy the cert/keys to all nodes. Please provide the steps you have used to create the cert/key. – b13n1u Oct 5 '14 at 18:45
  • Thanks thanks thanks :) . May be I was using incorrect options or edited my xxx.cnf file wrong. I used the exact same command you have provided and created a new 'xxx.cnf' file with the contents provided by you (with the required changes to suit my company) and it worked!. Thanks a lot. – Sree Oct 5 '14 at 19:41
  • Hi, @b13n1u I tried this configuration but I get Failed to tls handshake with <my ip> x509: certificate signed by unknown authority, why is that? Am I supposed to put the ip logstash forwarder should connect in IP.1 ? Thanks a lot. – larryzhao Feb 18 '15 at 15:42
  • Yes, you should add the logstash server IP address in IP.1 – b13n1u Feb 19 '15 at 12:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.