We have an intermittent issue that involves a severe drop in network traffic (both IN and OUT) on our HAProxy load balancer. It happens a couple times a day.
HAProxy is configured to limit requests to our backend servers (and queue up overflow traffic). We have our web servers hitting their max often enough, so I don't believe this is a matter of lowering the backend limits.
The graph below is the most recent example, but we've seen it peak to 1200-1500 connections, so it's equally as strange that this peak (with subsequent drop in traffic) occurs at a low number of connections.
I've suspected abusive IPs, and tracked the most popular IPs each minute on these servers since we first started investigating. I've blocked some that were obviously abusive, but it has happened since, and that doesn't feel like the true root cause.
For reference, we're using HAProxy 1.4.