0

I have been dealing with an aggressive bot crawler/attack for the last week. The bot is distributed with random ips, and agents string, so difficult to block, but I have another thread for that. This issue is that a flood of http requests can cause Tomcat to die.

The Tomcat process is still okay, and is not out of memory, it just stops taking http or https requests. It will timeout any http request, but will still take https requests (if http is being attacked, sometimes https dies as well).

I have seen error previously with "too many open files" so I changed the file limit from 10000 to 50000, that seemed to help, at least the https dying, but http still dies. I don't see the "too many open files" recently.

Seems like an extreme amount of open files, why would Tomcat open so many file, could it have a file leak under high load? The server sometimes is okay for 6 months (so can't be a leak under normal conditions), but has died under high load before.

The website is a large site with >1 million pages (dynamic content) and >1 million hits per day.

What happens when Tomcat gets a flood of http requests (like >100 per second for a prolonged duration), I assume requests will start backing up, if using a thread pool there will be no threads left, will it continue to pool requests until something breaks, or will it start rejecting requests?

Is there are way to start rejecting requests after a certain amount of backup? Seems like the only way to prevent a death or crash under extreme load.

My http and https config is different, so maybe that is related to why http dies. https is using maxThreads where as http is not, (how many threads is the default?)

<Connector port="80" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="443"
               URIEncoding="UTF-8" />

<Connector port="443" protocol="org.apache.coyote.http11.Http11NioProtocol"
               maxThreads="150" SSLEnabled="true" scheme="https" secure="true"
               clientAuth="false" sslProtocol="TLS"
               keystoreFile="..." keystorePass="..."
               URIEncoding="UTF-8"/>

Using Tomcat v8.5.47, CentOS 7.6, Oracle Java 1.8

  • 1
    In some circumstances network connections are considered file handles. It may not be all actual files. – Stephen Ostermiller Feb 29 at 17:31
  • Which ProtocolHandler gets selected for port 80? HTTP/1.1 automatically selects either org.apache.coyote.http11.Http11NioProtocol or org.apache.coyote.http11.Http11AprProtocol. Did you check the logs at the time the Connector stops responding? – Piotr P. Karwasz Mar 1 at 8:34
1

Both of those connectors will be using the NIO implementation.

For 8.5.47 the default value of maxConnections is 10000 and backlog is 100.

What will happen is the first 10000 connections will be accepted and will be allocated in turn to threads in the thread pool (default 200 threads) for processing. How long it takes to process those connections depends on the application.

If the are 10000 current connections, the next 100 requests will be held in the backlog. While Java tries to configure that to 100, the OS may choose to ignore that setting.

Once the backlog is full subsequent connections will be dropped.

You can (hopefully) find out why HTTP is non-responsive by taking three thread dumps ~5s apart when you see HTTP not responding.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.