11

I need to use openssl to perform some HTTP GET requests in a shell script. The line I'm using to do this right now is shown below. This is parsing the content of an XML response of the following formats.

<Result>success</Result>

<Result>failure</Result>

echo -e "GET /test HTTP/1.1\r\nHost:$(hostname)\r\n\r\n" | openssl 2>&1 s_client -quiet -connect server-url:443 | grep -o -P --color '(?<=Result\>).*(?=\</Result)'

This works and returns the string 'success' or 'failure' accordingly. The problem I'm facing is that the openssl command does not terminate after doing the GET request but instead sits there waiting on more input. I believe this is due to the implicit -ign_eof which prevents automatic termination caused by the -quiet option. I've tried using the -no_ign_eof option but that causes the openssl command to terminate before the GET request has received a response so I can't get the content of the response if I use that.

How can I modify this command so I can pass the GET request through stdin (required as I want to put this in a loop) but have the openssl command terminate after each request?

5
  • Why are you trying to abuse openssl for this, rather than using something more appropriate like curl? Oct 23, 2015 at 16:18
  • Because curl isn't available on the system and I don't have permission to install it Oct 23, 2015 at 16:18
  • 2
    Do yourself a favor, and go have a chat with the person who does have permission to install it. Oct 23, 2015 at 16:19
  • I intend to but that doesn't help me get this workaround up and running... Oct 23, 2015 at 16:20
  • If you can't get CURL there is wget you could abuse bit better for it you can use the info at stackoverflow.com/questions/12120935/… Oct 23, 2015 at 16:30

4 Answers 4

5

What you really should be doing is using a tool designed for fetching Web resources, such as curl, wget, or libwww-perl's GET command. If nothing is available, you should have your system administrator install something appropriate.

With that out of the way...

The openssl command does not terminate because the web server didn't close the connection.

Remember that by default HTTP keeps connections open after each request as a performance optimization. Once one request finishes, another request can be sent over the same connection, rather than closing and reopening a new connection.

If you want to instruct the server to close the connection instead, you can send the Connection: close HTTP header.

5
  • As mentioned in the comments on the question, unfortunately it is proving difficult to get open source tools installed on this AIX system and openssl seems to be the only tool available to me to do this right now. I've tried your suggestion and it works. Thanks Oct 23, 2015 at 16:48
  • 2
    As a technical matter it's very easy to install the tools. I expect your roadblocks are entirely political in nature. Oct 23, 2015 at 16:49
  • yes they are, IBM even supply a list of open source tools on their website that includes curl www-03.ibm.com/systems/power/software/aix/linux/toolbox/… Oct 23, 2015 at 16:50
  • 2
    Remember, and remind anyone who is in a position to listen, that workarounds like this ultimately cost money. Oct 23, 2015 at 16:51
  • Note: if you use POST, don't forget to include the request body (duh!)
    – Nickolay
    Aug 25, 2019 at 15:14
1

IBM uses Perl for bootstrapping downloads, and I would recommend using Perl until you can get approval to install curl properly, because it is bundled with the operating system. Specifically, their dnf_aixtoolbox.sh script uses lwp-download to download the rpm.rte package in lines like this:

    LDR_CNTRL=MAXDATA=0x80000000@DSA /usr/opt/perl5/bin/lwp-download https://public.dhe.ibm.com/aix/freeSoftware/aixtoolbox/ezinstall/ppc/dnf_bundle_aix_73.tar

While you're not trying to download a file, documentation on the LWP library provides a number of different examples for different uses.

0

Another simple (but probably worse) solution would be to use HTTP/1.0 instead of HTTP/1.1.

2
  • It might work, but new-ish http servers sometimes can't handle 1.0.
    – kubanczyk
    Oct 30, 2018 at 17:04
  • 1
    @kubanczyk I have yet to come across an http server that cannot handle HTTP/1.0. A much more likely problem would be an http server that requires a Host header and a client that only sends the Host header when using HTTP/1.1. But ultimately the reason using HTTP/1.0 would help here is that it has different default behavior regarding closing the connection compared to HTTP/1.1. Adding a Connection header to the request would address that.
    – kasperd
    Oct 30, 2018 at 17:59
-1

There is a new scenario that brings up several of the points discussed in this thread previously.
One is related to curl, the other related to tlsv1.3

First, I was using curl and found that curl will interpret a '#' as the start of a new URL. I have found others who have identified this same issue with curl using other special characters. In this case, the URL is truncated and curl fails.

Heading

The precise problem is using nonconforming characters to extend the URL name. Web developers are adding data after the ".html". My encounter is with the ---------.html#<date>.
As entered, curl will truncate the URL at the #. If I encode it -------.html%23<date>, then paste the encoded URL into the website via a browser, it encodes the encoded characters again with results in "no page found."

Web developers are attaching data to URLs so instead of sending a new webpage each time a user requests a table change (frequently a table) the webpage can be updated by sending just the table data via JSON and not the entire webpage.

I don't see getting curl changed anytime soon. Thus, using OpenSSL to bring up URLs is necessary if such features are to be tested. I was able to find an unsecure website that happens to use # in the server paths. I tested this using OpenSSL and it completed successfully.

echo -e "GET /#/Methods HTTP/1.1\r\nHost: eu.httpbin.org\r\nConnection: Close\r\n\r\n" | openssl s_client -quiet -state -connect eu.httpbin.org:443

Therefore, it can conclude that OpenSSL does not treat '#' as a special character in a URL. Curl parses input as HTML where '#' is a special character related to a HTML anchor.

The second issue, which is somewhat related to the initial topic, is using this same OpenSSL technique with tlsv1.3. There are no errors reported by OpenSSL, but it stops after the second "read R Block."

echo -e "GET / HTTP/1.1\r\nHost: cmegroup.com\r\nConnection: Close\r\n\r\n" | openssl 2>&1 s_client -CAfile firefoxCertBundle.pem -cert privClientCrt.pem -key privClient.key -tls1_3 -ciphersuites TLS_AES_256_GCM_SHA384 -quiet -state -connect cmegroup.com:443
    SSL_connect:before SSL initialization
    SSL_connect:SSLv3/TLS write client hello
    SSL_connect:SSLv3/TLS write client hello
    SSL_connect:SSLv3/TLS read server hello
    SSL_connect:TLSv1.3 read encrypted extensions
    depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global 
    Root CA
    verify return:1
    depth=1 C = US, O = DigiCert Inc, CN = DigiCert SHA2 Secure Server CA
    verify return:1
    depth=0 C = US, ST = Illinois, L = Chicago, O = Chicago Mercantile Exchange Inc, 
    CN = www.cmegroup.com
    verify return:1
    SSL_connect:SSLv3/TLS read server certificate
    SSL_connect:TLSv1.3 read server certificate verify
    SSL_connect:SSLv3/TLS read finished
    SSL_connect:SSLv3/TLS write change cipher spec
    SSL_connect:SSLv3/TLS write finished
    SSL_connect:SSL negotiation finished successfully
    SSL_connect:SSL negotiation finished successfully
    SSL_connect:SSLv3/TLS read server session ticket
    SSL_connect:SSL negotiation finished successfully
    SSL_connect:SSL negotiation finished successfully
    SSL_connect:SSLv3/TLS read server session ticket
2
  • 1
    The problem here is that # is a special character. Nothing after the # is intended to be sent to the server. It is interpreted client side only. It does not make sense to attempt to send it to the server. Jul 11, 2021 at 3:00
  • please check RFC3986, you'll find that fragments (portion of URI that's after the #) is dereferenced solely by the useragent (e.g. browser). second, try adding Connection: keep-alive and Accept: text/html;q=0.8 on your openssl s_client request, you'll find the server replies with 301.
    – mforsetti
    Jul 11, 2021 at 7:49

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .