5

This is probably a continuation of my previous (unanswered) question because the underlying cause is probably the same.

I have a Linux server with nginx and sshd running on it. It's on a shared 100mbit/s unmetered link. During "peak times" (basically, during the day in the US), sftp performance becomes very bad, sometimes timing out before I can even connect. ssh is unaffected. I know it's nginx because when I stop nginx, the problem with sftp goes away instantly. However, nginx itself has essentially zero latency during these "episodes."

This is a long-standing problem with my server, and I set out recently to take care of it once and for all. Yesterday I began to suspect that the sheer volume of http traffic coupled with the greater latency induced by a lack of upstream bandwidth was crowding out my sftp traffic. I used tc to add some prioritization:

/sbin/tc qdisc add dev eth1 root handle 1: prio 
/sbin/tc filter add dev eth1 protocol ip parent 1: prio 1 u32 match ip dport 22 0xffff flowid 1:1
/sbin/tc filter add dev eth1 protocol ip parent 1: prio 1 u32 match ip sport 22 0xffff flowid 1:1
/sbin/tc filter add dev eth1 protocol ip parent 1: prio 1 u32 match ip protocol 1 0xff flowid 1:1

Unfortunately, even though I can see sftp packets accumulating in the first prio:

class prio 1:1 parent 1: 
 Sent 257065020 bytes 3548504 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
class prio 1:2 parent 1: 
 Sent 291943287326 bytes 206538185 pkt (dropped 615, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
class prio 1:3 parent 1: 
 Sent 22399809673 bytes 15525292 pkt (dropped 2334, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 

... latency is still unacceptable when connecting. Here are some pretty graphs I made just now while trying to correlate something with the sftp latency:

Here is sftp latency from a different location. I have the timeout set at 25 seconds. Anything greater than the normal 1-2 seconds it takes to connect and download a tiny file is unacceptable to me. You can see how it becomes OK during the night and then latency kicks in again during the day.

Content of /proc/net/sockstat. Note the apparent correlation of sftp latency with tcp memory use. No idea what that might mean.

Output of nginx's stub-status module. Nothing to see here ...

Output of netstat -tan | awk '{print $6}' | sort | uniq -c. Again, seems flat.

So why isn't tc working for me? Do I need to actually limit bandwidth rather than just prioritizing port 22 in and out? Or is tc the wrong tool for the job and I'm totally missing the real cause of the bad sftp performance?

Output of uname -a:

Linux [redacted] 3.2.0-0.bpo.2-amd64 #1 SMP Fri Jun 29 20:42:29 UTC 2012 x86_64 GNU/Linux

I'm running nginx 1.2.2 with the mp4 streaming module compiled in.

Edit 2012/07/31:

ewwhite asked whether I'm near or at my bandwidth limit. I checked, and there does seem to be a correlation (though not a perfect one) between the 100 mbit limit and the bad sftp latency:

Why, though, would sftp traffic (associated with port 22) not be prioritized higher than http traffic during these episodes?

Edit 2012/07/31 #2

In collecting sftp/scp latency data, I noticed a pattern, as shown in the below graph (the green lines I added):

Two clusters - subtracting "baseline" latency, they are at ~5 and ~10 seconds. You can also see them pretty clearly on the above sftp latency graph on a much larger timescale. Where is this 5 second number coming from?

2 Answers 2

2
+100

A couple of things jump out at me...

  • You're not maxing-out or approaching the bandwidth limits, are you?
  • Have you looked at system entropy pool levels during the period of slow sftp performance (check /proc/sys/kernel/random/entropy_avail)? E.g. are your nginx sessions doing a lot of SSL requests? That can have a clear effect on other services that use encryption.
  • There are some sysctl.conf tuning parameters that may help (tcp window size?), but sftp isn't terribly efficient. Is scp an option? How large are the files?
  • DNS? Are you encountering reverse-lookup delays? Do you have any control over who's connecting to you? If it's predictable, try a stub entry for the source IP's in /etc/hosts to see if that helps.
8
  • The system seemed to be low on entropy, even now during off-peak, so I installed haveged from Debian backports. I'm a bit confused as to where all the entropy is going. Virtually all of the http traffic should be unencrypted. I've had this problem on two totally unrelated boxes now (except for them sharing this specific workload) so I don't think it's hardware-related. I'm going to continue to monitor latency through tomorrow in order to determine whether the problem with sftp has been solved by installing haveged as an additional entropy provider.
    – njahnke
    Jul 31, 2012 at 4:54
  • Let me know if this helps. It was the first thing that jumped out at me. I've also heard of nginx + entropy issues, so perhaps the new random number generation source will resolve...
    – ewwhite
    Jul 31, 2012 at 8:37
  • Unfortunately, it wasn't a lack of entropy - today started out just like yesterday. I've edited my question with additional information and graphs related to bandwidth use. Also, a clarification - in the sftp latency graph, all it's measuring is connecting to my server and downloading a 5 byte file. It should obviously not take > 10 seconds to do this, thus my problem.
    – njahnke
    Jul 31, 2012 at 17:50
  • What's the CPU utilization during the slow periods? If you're looking at QoS, are there any upstream devices (firewalls, routers, etc.) that can assist?
    – ewwhite
    Jul 31, 2012 at 18:10
  • 2
    5 second is the usual DNS query timeout, but if this is the problem, all SSH connections should experience the same delay, not just sftp/scp. Or maybe your pure ssh connections happened to use the ControlMaster feature, avoiding the reverse DNS lookup when the connection is reused? Jul 31, 2012 at 19:46
1

So it turns out I had at least three different problems masking one another. Here's what I did to solve the problems:

  1. Prioritize ICMP and ingoing/outgoing traffic on port 22 (as shown in my question above). This boosts sftp responsiveness (e.g., ls) and also transmission throughput during peak times.

  2. Solve the entropy shortage by installing the haveged package via Debian backports. This solves the "hang for several minutes at select()" issue. ewwhite++

  3. Add UseDNS no to /etc/ssh/sshd_config and rehash sshd. This solves the sftp delay at 5 second intervals during peak times. Sergey Vlasov++

Remaining mysteries:

  • My host initially configured /etc/resolv.conf for me, adding two of their nameservers as primaries. It's understandable that one or more of these nameservers are overloaded during peak times (i.e., during the day in the US), resulting in the 5 second interval delays I noticed on my sftp latency graphs. However, why does sftp perform a reverse DNS lookup every time I transfer a file? Were these simply cases when the reverse lookup timed out on the initial connect, and then on the first transfer, the sftp subsystem tried again and again failed to reverse my IP? Does the system not try the secondary nameservers in this case? At any rate, I've now added some well-known public nameservers as primaries over my ISP's overloaded ones, so other possible applications running on this same server won't have problems with DNS during peak times.

  • What is consuming entropy on my server? I couldn't find any evidence that stock nginx (serving static files) calls rand(), and yet that seems to be exactly what is happening. Is it the filesystem (ext3/4) or is another part of the kernel involved somehow?

Anyway, this is good enough for now. Thanks to this community, I was able to solve one of the most annoying and persistent problems I've encountered in over ten years of unix web server administration.

1
  • Take a look at the extended options for your /etc/resolv.conf. You can set the resolver timeout in the file by adding options timeout 1. This will try the next DNS server if the first is not reachable after 1 second. The compiled-in default timeout is 5 seconds I've also set options rotate to perform round-robin selection of the listed nameservers. In some cases, enabling a caching daemon like nscd will speed things up. It's not as popular, but came in handy when I had application environments like yours.
    – ewwhite
    Aug 1, 2012 at 20:56

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .