18

We have a java server running in linux at a specific port that accepts persistent connections for thousands and thousands of users. Recently our clients are not able to connect with a time out error. We suspect the traffic is getting too high but our java log actually shows that not many are connected in per second.

We suspect that could it be that too many are trying at the same time and they are basically dropped at the OS level and therefore the java program never really gets a chance to accept the connection? Is there some sort of log in linux that can show someone trying to hit a socket?

13

iptables -I INPUT -p tcp --dport some_port -j LOG then
tail -f /var/log/messages
Afterwards, to see how much data has been hit by that rule: iptables -L -n -v
Or you could run tcpdump and grep out the ports.

2
  • 3
    +1. A slight modification might work better to catch just the new connection attempts: iptables -I INPUT -p tcp --dport some_port -m state --state NEW. Note that unless you're interested in the details of each connection attempt, omitting -j LOG avoids spamming the system log file with lots of unneeded data. Oct 21 '10 at 21:17
  • Upon setting this up, prepare to see a stomach-churningly high number of supposedly non-malicious security crawlers connecting to your server, like implant-scanner-victims-will-be-notified.threatsinkhole.com, stretchoid.com, jobqueue-listener.jobqueue.netcraft.com-digitalocean, ... good to know.
    – ijoseph
    Sep 6 '20 at 3:21
3

When I have really nasty network problems, I tend to fire up wireshark. For me, there's no better network diagnostic tool when I have to get down to the nitty-gritty details. And don't worry if you can't install it on either source or destination box; you can run tcpdump -w to write packet data to a file on start and/or endpoint, and feed file to wireshark on another box at your convenience.

1
  • Yup, I'd recommend putting a filter on it like tcpdump -nS dst port <some port>
    – gravyface
    Oct 21 '10 at 21:01
1
watch -n1 -d "netstat -an | grep ESTABLISHED | wc -l"

shows the number of currently established connections.

Compare this with your active ulimit settings, and of course with the maximum number of connections your java app can handle.

0

Would be good to see precisely what your Java threads are getting at a socket level. At the same time you would want to correlate that with OS network info. Take a look at AppFirst. They can do this kind of thing.

0

by the way, you may also want to delete it:

iptables -L INPUT --line-numbers
iptables -D INPUT <line num>

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.