0

I am using google api with PHP to query google's API from my server - which uses guzzle to send a regular http request. A high number of requests (~10%) are returning after exactly 63 seconds, while the rest are returning after about ~1s.

I tried from another server and the issue does not exist there.

I checked netstat and I see that I have about 500 time-wait connections:

netstat -ant | awk '{print $6}' | sort | uniq -c | sort -n

  1 established
  1 Foreign
  4 CLOSE_WAIT
  6 FIN_WAIT1
  6 SYN_RECV
 11 FIN_WAIT2
 11 LAST_ACK
 31 LISTEN
 66 SYN_SENT
 90 ESTABLISHED
456 TIME_WAIT

cat /proc/sys/fs/file-nr:

379008  0       500001

cat /proc/sys/net/ipv4/ip_local_port_range

24576   65534

How can I check which setting is causing the response to be so delayed.

Thanks

netstat -s

TcpExt:
26599 resets received for embryonic SYN_RECV sockets
147494 packets pruned from receive queue because of socket buffer overrun
158 ICMP packets dropped because they were out-of-window
1 ICMP packets dropped because socket was locked
1586409 TCP sockets finished time wait in fast timer
25630 packets rejects in established connections because of timestamp
2528562 delayed acks sent
2824 delayed acks further delayed because of locked socket
Quick ack mode was activated 160978 times
35706 times the listen queue of a socket overflowed
35706 SYNs to LISTEN sockets dropped
8532661 packets directly queued to recvmsg prequeue.
19911549 bytes directly in process context from backlog
3447934652 bytes directly received in process context from prequeue
60217414 packet headers predicted
4655144 packets header predicted and directly queued to user
80744319 acknowledgments not containing data payload received
42859898 predicted acknowledgments
26 times recovered from packet loss due to fast retransmit
184100 times recovered from packet loss by selective acknowledgements
292 bad SACK blocks received
Detected reordering 33568 times using FACK
Detected reordering 114073 times using SACK
Detected reordering 126806 times using time stamp
113848 congestion windows fully recovered without slow start
1980316 congestion windows partially recovered using Hoe heuristic
23835 congestion windows recovered without slow start by DSACK
25078 congestion windows recovered without slow start after partial ack
82939 TCP data loss events
TCPLostRetransmit: 4572
10 timeouts after reno fast retransmit
7421 timeouts after SACK recovery
4352 timeouts in loss state
838050 fast retransmits
108184 forward retransmits
87852 retransmits in slow start
182297 other TCP timeouts
3 classic Reno fast retransmits failed
4290 SACK retransmits failed
116 times receiver scheduled too late for direct processing
134674 packets collapsed in receive queue due to low socket buffer
164349 DSACKs sent for old packets
1551 DSACKs sent for out of order packets
775080 DSACKs received
1007 DSACKs for out of order packets received
153613 connections reset due to unexpected data
23137 connections reset due to early user close
58391 connections aborted due to timeout
TCPSACKDiscard: 33
TCPDSACKIgnoredOld: 10477
TCPDSACKIgnoredNoUndo: 457106
TCPSpuriousRTOs: 3193
TCPSackShifted: 670876
TCPSackMerged: 1268943
TCPSackShiftFallback: 661260
TCPBacklogDrop: 14555
TCPOFOQueue: 1173042
TCPOFOMerge: 919
TCPChallengeACK: 4644
TCPSYNChallenge: 432
TCPFromZeroWindowAdv: 17586
TCPToZeroWindowAdv: 17586
TCPWantZeroWindowAdv: 119758
TCPACKSkippedSynRecv: 4284
TCPACKSkippedPAWS: 11760
TCPACKSkippedSeq: 4434
TCPACKSkippedFinWait2: 107
TCPACKSkippedTimeWait: 249
TCPACKSkippedChallenge: 223
-1

Something is wrong with your network or application, It is better to tune your tcp settings or put proper tcp timeouts. Below are some examples.

echo 30 > /proc/sys/net/ipv4/tcp_fin_timeout
echo 30 > /proc/sys/net/ipv4/tcp_keepalive_intvl
echo 5 > /proc/sys/net/ipv4/tcp_keepalive_probes
  • Thanks, how did you see there was a problem by what I posted? – danielb Jun 1 at 20:37
  • 456 TIME_WAIT this count shows connection not closed, either due to network or application by tuning tcp, I think you can fix it or check application configuration. – user525856 Jun 2 at 5:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.