0

Background: one of our Ubuntu servers is becoming randomly (?) unresponsive ie. some connections are dropped and the server is not accepting new ones for a while. At least this is what we see in logs of dependant services. The problem lasts for few minutes at best. It happens 1-3 times a day. We use munin to monitor this server, but nothing from standard graphs stands out for us.

After initial diagnosis nothing came up but our data center told us that when it happens there are short high spikes of traffic on their charts. No more useful info from them. How to identify what is causing them? (source IP, dest port)

I thought about capturing packets with tcpdump, but I have no idea what filter could be applied to avoid producing gigabytes of logs. Maybe there exist a tool to log all connections and print statistics (connection duration, bytes exchanged, rate)? Assuming these are not connectionless packets.

  • Monitoring software is your friend. It can capture the relevant data then can configure alerts to your desires. There are paid ones but if you are in a pinch Naigos is an old go to but kind dated. You can check out Sensu. I have used vROM before from VMware and that helped me to identify network traffic trends. If you are virtualizing vCenter collects network usage. What does this server host? What are your access logs showing. Monitoring is a very open ended it could be a burst traffic from a big transaction. I saw this a lot in email server management. Hope this helps... – Dion Pezzimenti Mar 25 at 0:59
  • This should be aittle more informative and hopefully helpful then my prior off the top of my head thoughts and prior experiences: acronis.com/en-us/blog/posts/… – Dion Pezzimenti Mar 25 at 1:11
0

Do not assume this events root cause is "high spikes of traffic" until you gather more data. It could be many kinds of problems, usually in the categories of utilization, saturation, or errors. Look at all the metrics, including your application's response time to user requests. Read your logs including syslog.


If the event is too brief for your monitoring interval, consider trying some high resolution monitoring. For example, Netdata can collect a broad range of metrics at 1 second intervals.


Maybe there exist a tool to log all connections and print statistics (connection duration, bytes exchanged, rate)?

Naturally, someone has hacked together BPF tracing scripts to do this. Brendan Greg's implementation is called tcplife. Source in the bcc repo.

# ./tcplife -t
TIME(s)   PID   COMM       LADDR           LPORT RADDR           RPORT TX_KB RX_KB MS
0.000000  5973  recordProg 127.0.0.1       47986 127.0.0.1       28527     0     0 0.25
0.000059  3277  redis-serv 127.0.0.1       28527 127.0.0.1       47986     0     0 0.29
1.022454  5996  recordProg 127.0.0.1       47988 127.0.0.1       28527     0     0 0.23
1.022513  3277  redis-serv 127.0.0.1       28527 127.0.0.1       47988     0     0 0.27
2.044868  6019  recordProg 127.0.0.1       47990 127.0.0.1       28527     0     0 0.24
2.044924  3277  redis-serv 127.0.0.1       28527 127.0.0.1       47990     0     0 0.28
3.069136  6042  recordProg 127.0.0.1       47992 127.0.0.1       28527     0     0 0.22
3.069204  3277  redis-serv 127.0.0.1       28527 127.0.0.1       47992     0     0 0.28

This shows that the recordProg process was connecting once per second.

Plenty of disclaimers: Linux 4.4 or later, only TCP sessions at close, the PID detection is sketchy. But you can show closed TCP sessions live with microsecond timestamps, without packet tracing.

Should the events not be caught at close, try tracing TCP connects.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.