2

I have a somewhat complex setup that doesn't work, and I'm struggeling quite a bit to track down the root cause.

First of all the big picture:

Node backend is a medium size windows azure worker role running JBoss 5. It is streaming out HTTP multipart/chunked JSON data.

Node frontend is medium size windows azure worker role running JBoss 5. It is requesting the HTTP multipart/chunked JSON data from the backend node, and then publishing that again via http.

These streams are quite straigt forward and can be tested with a browser or wget.

We observe that the streams just stop working after some time (hours to days). In order to track that down I fired up a wget on the frontend (i.e. client) side and a wireshark to see whats goign on. Wireshark is set to dump to a ringbuffer of files. Stream speed is roughly 100KB/s, so performance should not be an issue. What I observe is: The Receive window of wget starts at 130000 and than decreases down to 517bytes. It eventually hits 0, and than wget sits there fore quite some time (sometimes a few secs, sometimes up to 100s) until it accepts data again. See this wireshark:

No.     Time            Source                Destination           Protocol Info
 138300 17:33:47.038970 10.211.148.222        94.245.90.204         TCP      52902 > 5284 [ACK] Seq=1 Ack=12610206 Win=27 Len=0
 138301 17:33:47.069346 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12610206 Ack=1 Win=514 Len=118
 138302 17:33:47.069353 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12610324 Ack=1 Win=514 Len=119
 138303 17:33:47.069358 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12610443 Ack=1 Win=514 Len=118
 138304 17:33:47.069362 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12610561 Ack=1 Win=514 Len=109
 138305 17:33:47.069366 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12610670 Ack=1 Win=514 Len=126
 138306 17:33:47.069370 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12610796 Ack=1 Win=514 Len=115
 138307 17:33:47.069374 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12610911 Ack=1 Win=514 Len=119
 138308 17:33:47.069378 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12611030 Ack=1 Win=514 Len=116
 138309 17:33:47.069382 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12611146 Ack=1 Win=514 Len=115
 138310 17:33:47.069386 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12611261 Ack=1 Win=514 Len=116
 138311 17:33:47.069390 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12611377 Ack=1 Win=514 Len=115
 138312 17:33:47.069499 10.211.148.222        94.245.90.204         TCP      52902 > 5284 [ACK] Seq=1 Ack=12611492 Win=22 Len=0
 138313 17:33:47.069569 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12611492 Ack=1 Win=514 Len=132
 138314 17:33:47.069573 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12611624 Ack=1 Win=514 Len=121
 138315 17:33:47.069577 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12611745 Ack=1 Win=514 Len=103
 138316 17:33:47.069581 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12611848 Ack=1 Win=514 Len=107
 138317 17:33:47.069585 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12611955 Ack=1 Win=514 Len=118
 138318 17:33:47.069589 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12612073 Ack=1 Win=514 Len=118
 138319 17:33:47.069593 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12612191 Ack=1 Win=514 Len=117
 138320 17:33:47.069597 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12612308 Ack=1 Win=514 Len=126
 138321 17:33:47.069601 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12612434 Ack=1 Win=514 Len=108
 138322 17:33:47.069605 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12612542 Ack=1 Win=514 Len=112
 138323 17:33:47.069609 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12612654 Ack=1 Win=514 Len=109
 138324 17:33:47.069613 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12612763 Ack=1 Win=514 Len=107
 138325 17:33:47.069617 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12612870 Ack=1 Win=514 Len=104
 138326 17:33:47.069621 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12612974 Ack=1 Win=514 Len=115
 138327 17:33:47.069625 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12613089 Ack=1 Win=514 Len=107
 138328 17:33:47.069629 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12613196 Ack=1 Win=514 Len=107
 138329 17:33:47.069633 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12613303 Ack=1 Win=514 Len=123
 138330 17:33:47.069638 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12613426 Ack=1 Win=514 Len=105
 138331 17:33:47.069641 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12613531 Ack=1 Win=514 Len=129
 138332 17:33:47.069712 10.211.148.222        94.245.90.204         TCP      52902 > 5284 [ACK] Seq=1 Ack=12613660 Win=14 Len=0
 138333 17:33:47.069756 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12613660 Ack=1 Win=514 Len=111
 138334 17:33:47.069761 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12613771 Ack=1 Win=514 Len=114
 138335 17:33:47.069765 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12613885 Ack=1 Win=514 Len=127
 138336 17:33:47.069769 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12614012 Ack=1 Win=514 Len=109
 138337 17:33:47.069773 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12614121 Ack=1 Win=514 Len=107
 138338 17:33:47.069777 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12614228 Ack=1 Win=514 Len=106
 138339 17:33:47.069781 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12614334 Ack=1 Win=514 Len=120
 138340 17:33:47.069785 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12614454 Ack=1 Win=514 Len=110
 138341 17:33:47.069789 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12614564 Ack=1 Win=514 Len=114
 138342 17:33:47.069793 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12614678 Ack=1 Win=514 Len=107
 138343 17:33:47.069797 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12614785 Ack=1 Win=514 Len=109
 138344 17:33:47.069801 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12614894 Ack=1 Win=514 Len=119
 138345 17:33:47.069805 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12615013 Ack=1 Win=514 Len=113
 138346 17:33:47.069809 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12615126 Ack=1 Win=514 Len=116
 138347 17:33:47.069813 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12615242 Ack=1 Win=514 Len=120
 138348 17:33:47.069817 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12615362 Ack=1 Win=514 Len=114
 138349 17:33:47.069870 10.211.148.222        94.245.90.204         TCP      52902 > 5284 [ACK] Seq=1 Ack=12615476 Win=7 Len=0
 138350 17:33:47.069908 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12615476 Ack=1 Win=514 Len=108
 138351 17:33:47.069912 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12615584 Ack=1 Win=514 Len=107
 138352 17:33:47.069916 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12615691 Ack=1 Win=514 Len=107
 138353 17:33:47.069919 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12615798 Ack=1 Win=514 Len=115
 138354 17:33:47.069923 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12615913 Ack=1 Win=514 Len=120
 138355 17:33:47.069927 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12616033 Ack=1 Win=514 Len=107
 138356 17:33:47.069931 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12616140 Ack=1 Win=514 Len=119
 138357 17:33:47.069935 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12616259 Ack=1 Win=514 Len=122
 138358 17:33:47.069939 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12616381 Ack=1 Win=514 Len=105
 138359 17:33:47.069943 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12616486 Ack=1 Win=514 Len=113
 138360 17:33:47.069947 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12616599 Ack=1 Win=514 Len=107
 138361 17:33:47.069950 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12616706 Ack=1 Win=514 Len=114
 138362 17:33:47.069954 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12616820 Ack=1 Win=514 Len=127
 138363 17:33:47.069959 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12616947 Ack=1 Win=514 Len=132
 138364 17:33:47.070004 10.211.148.222        94.245.90.204         TCP      [TCP ZeroWindow] 52902 > 5284 [ACK] Seq=1 Ack=12617079 Win=0 Len=0
 138365 17:33:47.390902 94.245.90.204         10.211.148.222        TCP      [TCP ZeroWindowProbe] 5284 > 52902 [ACK] Seq=12617079 Ack=1 Win=514 Len=1
 138366 17:33:47.599283 10.211.148.222        94.245.90.204         TCP      [TCP ZeroWindow] [TCP ACKed lost segment] 52902 > 5284 [ACK] Seq=1 Ack=12617080 Win=0 Len=0
 138367 17:33:48.203458 94.245.90.204         10.211.148.222        TCP      [TCP ZeroWindowProbe] 5284 > 52902 [ACK] Seq=12617080 Ack=1 Win=514 Len=1
 138368 17:33:48.411799 10.211.148.222        94.245.90.204         TCP      [TCP ZeroWindow] [TCP ACKed lost segment] 52902 > 5284 [ACK] Seq=1 Ack=12617081 Win=0 Len=0
 139055 17:33:50.015857 94.245.90.204         10.211.148.222        TCP      [TCP ZeroWindowProbe] 5284 > 52902 [ACK] Seq=12617081 Ack=1 Win=514 Len=1
 139056 17:33:50.224262 10.211.148.222        94.245.90.204         TCP      [TCP ZeroWindow] [TCP ACKed lost segment] 52902 > 5284 [ACK] Seq=1 Ack=12617082 Win=0 Len=0
 141485 17:33:54.234768 94.245.90.204         10.211.148.222        TCP      [TCP ZeroWindowProbe] 5284 > 52902 [ACK] Seq=12617082 Ack=1 Win=514 Len=1
 141972 17:33:54.442993 10.211.148.222        94.245.90.204         TCP      [TCP ZeroWindow] [TCP ACKed lost segment] 52902 > 5284 [ACK] Seq=1 Ack=12617083 Win=0 Len=0
 146212 17:34:03.422127 94.245.90.204         10.211.148.222        TCP      [TCP ZeroWindowProbe] 5284 > 52902 [ACK] Seq=12617083 Ack=1 Win=514 Len=1
 146214 17:34:03.630469 10.211.148.222        94.245.90.204         TCP      [TCP ZeroWindow] [TCP ACKed lost segment] 52902 > 5284 [ACK] Seq=1 Ack=12617084 Win=0 Len=0
 153400 17:34:22.203389 94.245.90.204         10.211.148.222        TCP      [TCP ZeroWindowProbe] 5284 > 52902 [ACK] Seq=12617084 Ack=1 Win=514 Len=1
 153967 17:34:22.411696 10.211.148.222        94.245.90.204         TCP      [TCP ZeroWindow] [TCP ACKed lost segment] 52902 > 5284 [ACK] Seq=1 Ack=12617085 Win=0 Len=0
 156977 17:34:59.547273 94.245.90.204         10.211.148.222        TCP      [TCP ZeroWindowProbe] 5284 > 52902 [ACK] Seq=12617085 Ack=1 Win=514 Len=1
 156978 17:34:59.755506 10.211.148.222        94.245.90.204         TCP      [TCP ZeroWindow] [TCP ACKed lost segment] 52902 > 5284 [ACK] Seq=1 Ack=12617086 Win=0 Len=0
 156983 17:35:49.424340 10.211.148.222        94.245.90.204         TCP      [TCP Window Update] 52902 > 5284 [ACK] Seq=1 Ack=12617086 Win=517 Len=0
 156984 17:35:49.427810 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12617086 Ack=1 Win=514 Len=1440
 156985 17:35:49.427893 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12618526 Ack=1 Win=514 Len=1440
 156986 17:35:49.427934 10.211.148.222        94.245.90.204         TCP      52902 > 5284 [ACK] Seq=1 Ack=12619966 Win=506 Len=0
 156987 17:35:49.427960 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12619966 Ack=1 Win=514 Len=1440
 156988 17:35:49.427991 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=12621406 Ack=1 Win=514 Len=1440 

I cannot see why wget would be busy so long. Also I see similar behaviour with our JBoss/java client. "Normally" the stream looks like this:

No.     Time            Source                Destination           Protocol Info
     46 17:31:04.858278 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=1 Ack=1 Win=514 Len=108
     50 17:31:04.858306 10.211.148.222        94.245.90.204         TCP      52902 > 5284 [ACK] Seq=1 Ack=109 Win=516 Len=0
     61 17:31:04.858528 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=109 Ack=1 Win=514 Len=95
    166 17:31:04.861793 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=204 Ack=1 Win=514 Len=107
    167 17:31:04.861813 10.211.148.222        94.245.90.204         TCP      52902 > 5284 [ACK] Seq=1 Ack=311 Win=515 Len=0
    320 17:31:04.869327 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=311 Ack=1 Win=514 Len=102
    326 17:31:04.869393 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=413 Ack=1 Win=514 Len=117
    327 17:31:04.869408 10.211.148.222        94.245.90.204         TCP      52902 > 5284 [ACK] Seq=1 Ack=530 Win=515 Len=0
    328 17:31:04.869419 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=530 Ack=1 Win=514 Len=96
    329 17:31:04.870048 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=626 Ack=1 Win=514 Len=113
    330 17:31:04.870069 10.211.148.222        94.245.90.204         TCP      52902 > 5284 [ACK] Seq=1 Ack=739 Win=514 Len=0
    337 17:31:04.870431 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=739 Ack=1 Win=514 Len=107
    383 17:31:04.871195 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=846 Ack=1 Win=514 Len=116
    388 17:31:04.871222 10.211.148.222        94.245.90.204         TCP      52902 > 5284 [ACK] Seq=1 Ack=962 Win=513 Len=0
    423 17:31:04.872362 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=962 Ack=1 Win=514 Len=99
    440 17:31:04.873151 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=1061 Ack=1 Win=514 Len=100
    446 17:31:04.873182 10.211.148.222        94.245.90.204         TCP      52902 > 5284 [ACK] Seq=1 Ack=1161 Win=512 Len=0
    508 17:31:04.875097 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=1161 Ack=1 Win=514 Len=98
    599 17:31:04.877254 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=1259 Ack=1 Win=514 Len=104
    604 17:31:04.877307 10.211.148.222        94.245.90.204         TCP      52902 > 5284 [ACK] Seq=1 Ack=1363 Win=517 Len=0
    639 17:31:04.878517 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=1363 Ack=1 Win=514 Len=115
    657 17:31:04.878662 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=1478 Ack=1 Win=514 Len=104
    660 17:31:04.878693 10.211.148.222        94.245.90.204         TCP      52902 > 5284 [ACK] Seq=1 Ack=1582 Win=516 Len=0
    683 17:31:04.879737 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=1582 Ack=1 Win=514 Len=106
    716 17:31:04.880392 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=1688 Ack=1 Win=514 Len=100
    720 17:31:04.880448 10.211.148.222        94.245.90.204         TCP      52902 > 5284 [ACK] Seq=1 Ack=1788 Win=515 Len=0
    767 17:31:04.882359 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=1788 Ack=1 Win=514 Len=104
    841 17:31:04.883507 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=1892 Ack=1 Win=514 Len=108
    843 17:31:04.883527 10.211.148.222        94.245.90.204         TCP      52902 > 5284 [ACK] Seq=1 Ack=2000 Win=515 Len=0
    851 17:31:04.884238 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=2000 Ack=1 Win=514 Len=101
    896 17:31:04.885225 94.245.90.204         10.211.148.222        TCP      5284 > 52902 [PSH, ACK] Seq=2101 Ack=1 Win=514 Len=116
    904 17:31:04.885276 10.211.148.222        94.245.90.204         TCP      52902 > 5284 [ACK] Seq=1 Ack=2217 Win=514 Len=0

Also I don't really get why the window is just 517 bytes.Any idea? The data packages are so small (100-200bytes) due to the current Level 7 protocol used.

Edit: Data transfer failed after 6GB, or 8GB. seems no 32bit-stuff. Link/Hardware is unknown, but supposed to be within one microsoft azure data center. Here is a connection setup of wget (remove a few bits for minimum privacy):

No.     Time            Source                Destination           Protocol Info
 157007 17:50:49.590140 10.211.148.222        94.245.90.204         TCP      58291 > 5284 [SYN] Seq=0 Win=8192 Len=0 MSS=1460 WS=8

Frame 157007 (66 bytes on wire, 66 bytes captured)
    Arrival Time: Apr 16, 2012 17:50:49.590140000
    Frame Number: 157007
    Frame Length: 66 bytes
    Capture Length: 66 bytes
Ethernet II, Src: 00:15:5d:36:03:6d (00:15:5d:36:03:6d), Dst: 02:f5:0a:d3:ff:0a (02:f5:0a:d3:ff:0a)
    Destination: 02:f5:0a:d3:ff:0a (02:f5:0a:d3:ff:0a)
        Address: 02:f5:0a:d3:ff:0a (02:f5:0a:d3:ff:0a)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default)
    Source: 00:15:5d:36:03:6d (00:15:5d:36:03:6d)
        Address: 00:15:5d:36:03:6d (00:15:5d:36:03:6d)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Type: IP (0x0800)
Internet Protocol, Src: 10.211.148.222 (10.211.148.222), Dst: 94.245.90.204 (94.245.90.204)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
        0000 00.. = Differentiated Services Codepoint: Default (0x00)
        .... ..0. = ECN-Capable Transport (ECT): 0
        .... ...0 = ECN-CE: 0
    Total Length: 52
    Identification: 0x17ac (6060)
    Flags: 0x02 (Don't Fragment)
        0.. = Reserved bit: Not Set
        .1. = Don't fragment: Set
        ..0 = More fragments: Not Set
    Fragment offset: 0
    Time to live: 128
    Protocol: TCP (0x06)
    Source: 10.211.148.222 (10.211.148.222)
    Destination: 94.245.90.204 (94.245.90.204)
Transmission Control Protocol, Src Port: 58291 (58291), Dst Port: 5284 (5284), Seq: 0, Len: 0
    Source port: 58291 (58291)
    Destination port: 5284 (5284)
    Sequence number: 0    (relative sequence number)
    Header length: 32 bytes
    Flags: 0x02 (SYN)
        0... .... = Congestion Window Reduced (CWR): Not set
        .0.. .... = ECN-Echo: Not set
        ..0. .... = Urgent: Not set
        ...0 .... = Acknowledgement: Not set
        .... 0... = Push: Not set
        .... .0.. = Reset: Not set
        .... ..1. = Syn: Set
            [Expert Info (Chat/Sequence): Connection establish request (SYN): server port 5284]
                [Message: Connection establish request (SYN): server port 5284]
                [Severity level: Chat]
                [Group: Sequence]
        .... ...0 = Fin: Not set
    Window size: 8192
    Options: (12 bytes)
        Maximum segment size: 1460 bytes
        NOP
        Window scale: 8 (multiply by 256)
        NOP
        NOP
        SACK permitted

No.     Time            Source                Destination           Protocol Info
 157008 17:50:49.591169 94.245.90.204         10.211.148.222        TCP      5284 > 58291 [SYN, ACK] Seq=0 Ack=1 Win=8192 Len=0 MSS=1440 WS=8

Frame 157008 (66 bytes on wire, 66 bytes captured)
    Arrival Time: Apr 16, 2012 17:50:49.591169000
    Frame Number: 157008
    Frame Length: 66 bytes
    Capture Length: 66 bytes
Ethernet II, Src: 02:f5:0a:d3:ff:0a (02:f5:0a:d3:ff:0a), Dst: 00:15:5d:36:03:6d (00:15:5d:36:03:6d)
    Destination: 00:15:5d:36:03:6d (00:15:5d:36:03:6d)
        Address: 00:15:5d:36:03:6d (00:15:5d:36:03:6d)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Source: 02:f5:0a:d3:ff:0a (02:f5:0a:d3:ff:0a)
        Address: 02:f5:0a:d3:ff:0a (02:f5:0a:d3:ff:0a)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default)
    Type: IP (0x0800)
Internet Protocol, Src: 94.245.90.204 (94.245.90.204), Dst: 10.211.148.222 (10.211.148.222)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
        0000 00.. = Differentiated Services Codepoint: Default (0x00)
        .... ..0. = ECN-Capable Transport (ECT): 0
        .... ...0 = ECN-CE: 0
    Total Length: 52
    Identification: 0x6cbb (27835)
    Flags: 0x02 (Don't Fragment)
        0.. = Reserved bit: Not Set
        .1. = Don't fragment: Set
        ..0 = More fragments: Not Set
    Fragment offset: 0
    Time to live: 127
    Protocol: TCP (0x06)
    Source: 94.245.90.204 (94.245.90.204)
    Destination: 10.211.148.222 (10.211.148.222)
Transmission Control Protocol, Src Port: 5284 (5284), Dst Port: 58291 (58291), Seq: 0, Ack: 1, Len: 0
    Source port: 5284 (5284)
    Destination port: 58291 (58291)
    Sequence number: 0    (relative sequence number)
    Acknowledgement number: 1    (relative ack number)
    Header length: 32 bytes
    Flags: 0x12 (SYN, ACK)
        0... .... = Congestion Window Reduced (CWR): Not set
        .0.. .... = ECN-Echo: Not set
        ..0. .... = Urgent: Not set
        ...1 .... = Acknowledgement: Set
        .... 0... = Push: Not set
        .... .0.. = Reset: Not set
        .... ..1. = Syn: Set
            [Expert Info (Chat/Sequence): Connection establish acknowledge (SYN+ACK): server port 5284]
                [Message: Connection establish acknowledge (SYN+ACK): server port 5284]
                [Severity level: Chat]
                [Group: Sequence]
        .... ...0 = Fin: Not set
    Window size: 8192
    Options: (12 bytes)
        Maximum segment size: 1440 bytes
        NOP
        Window scale: 8 (multiply by 256)
        NOP
        NOP
        SACK permitted
    [SEQ/ACK analysis]
        [This is an ACK to the segment in frame: 157007]
        [The RTT to ACK the segment was: 0.001029000 seconds]

No.     Time            Source                Destination           Protocol Info
 157009 17:50:49.591242 10.211.148.222        94.245.90.204         TCP      58291 > 5284 [ACK] Seq=1 Ack=1 Win=132352 Len=0

Frame 157009 (54 bytes on wire, 54 bytes captured)
    Arrival Time: Apr 16, 2012 17:50:49.591242000
    Frame Number: 157009
    Frame Length: 54 bytes
    Capture Length: 54 bytes
Ethernet II, Src: 00:15:5d:36:03:6d (00:15:5d:36:03:6d), Dst: 02:f5:0a:d3:ff:0a (02:f5:0a:d3:ff:0a)
    Destination: 02:f5:0a:d3:ff:0a (02:f5:0a:d3:ff:0a)
        Address: 02:f5:0a:d3:ff:0a (02:f5:0a:d3:ff:0a)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default)
    Source: 00:15:5d:36:03:6d (00:15:5d:36:03:6d)
        Address: 00:15:5d:36:03:6d (00:15:5d:36:03:6d)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Type: IP (0x0800)
Internet Protocol, Src: 10.211.148.222 (10.211.148.222), Dst: 94.245.90.204 (94.245.90.204)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
        0000 00.. = Differentiated Services Codepoint: Default (0x00)
        .... ..0. = ECN-Capable Transport (ECT): 0
        .... ...0 = ECN-CE: 0
    Total Length: 40
    Identification: 0x17ad (6061)
    Flags: 0x02 (Don't Fragment)
        0.. = Reserved bit: Not Set
        .1. = Don't fragment: Set
        ..0 = More fragments: Not Set
    Fragment offset: 0
    Time to live: 128
    Protocol: TCP (0x06)
    Source: 10.211.148.222 (10.211.148.222)
    Destination: 94.245.90.204 (94.245.90.204)
Transmission Control Protocol, Src Port: 58291 (58291), Dst Port: 5284 (5284), Seq: 1, Ack: 1, Len: 0
    Source port: 58291 (58291)
    Destination port: 5284 (5284)
    [Stream index: 2]
    Sequence number: 1    (relative sequence number)
    Acknowledgement number: 1    (relative ack number)
    Header length: 20 bytes
    Flags: 0x10 (ACK)
        0... .... = Congestion Window Reduced (CWR): Not set
        .0.. .... = ECN-Echo: Not set
        ..0. .... = Urgent: Not set
        ...1 .... = Acknowledgement: Set
        .... 0... = Push: Not set
        .... .0.. = Reset: Not set
        .... ..0. = Syn: Not set
        .... ...0 = Fin: Not set
    Window size: 132352 (scaled)
    [SEQ/ACK analysis]
        [This is an ACK to the segment in frame: 157008]
        [The RTT to ACK the segment was: 0.000073000 seconds]

No.     Time            Source                Destination           Protocol Info
 157010 17:50:49.591565 10.211.148.222        94.245.90.204         TCP      58291 > 5284 [PSH, ACK] Seq=1 Ack=1 Win=132352 Len=389

Frame 157010 (443 bytes on wire, 443 bytes captured)
    Arrival Time: Apr 16, 2012 17:50:49.591565000
    Frame Number: 157010
    Frame Length: 443 bytes
    Capture Length: 443 bytes
Ethernet II, Src: 00:15:5d:36:03:6d (00:15:5d:36:03:6d), Dst: 02:f5:0a:d3:ff:0a (02:f5:0a:d3:ff:0a)
    Destination: 02:f5:0a:d3:ff:0a (02:f5:0a:d3:ff:0a)
        Address: 02:f5:0a:d3:ff:0a (02:f5:0a:d3:ff:0a)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default)
    Source: 00:15:5d:36:03:6d (00:15:5d:36:03:6d)
        Address: 00:15:5d:36:03:6d (00:15:5d:36:03:6d)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Type: IP (0x0800)
Internet Protocol, Src: 10.211.148.222 (10.211.148.222), Dst: 94.245.90.204 (94.245.90.204)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
        0000 00.. = Differentiated Services Codepoint: Default (0x00)
        .... ..0. = ECN-Capable Transport (ECT): 0
        .... ...0 = ECN-CE: 0
    Total Length: 429
    Identification: 0x17ae (6062)
    Flags: 0x02 (Don't Fragment)
        0.. = Reserved bit: Not Set
        .1. = Don't fragment: Set
        ..0 = More fragments: Not Set
    Fragment offset: 0
    Time to live: 128
    Protocol: TCP (0x06)
    Source: 10.211.148.222 (10.211.148.222)
    Destination: 94.245.90.204 (94.245.90.204)
Transmission Control Protocol, Src Port: 58291 (58291), Dst Port: 5284 (5284), Seq: 1, Ack: 1, Len: 389
    Source port: 58291 (58291)
    Destination port: 5284 (5284)
    Sequence number: 1    (relative sequence number)
    [Next sequence number: 390    (relative sequence number)]
    Acknowledgement number: 1    (relative ack number)
    Header length: 20 bytes
    Flags: 0x18 (PSH, ACK)
        0... .... = Congestion Window Reduced (CWR): Not set
        .0.. .... = ECN-Echo: Not set
        ..0. .... = Urgent: Not set
        ...1 .... = Acknowledgement: Set
        .... 1... = Push: Set
        .... .0.. = Reset: Not set
        .... ..0. = Syn: Not set
        .... ...0 = Fin: Not set
    Window size: 132352 (scaled)
    [SEQ/ACK analysis]
        [Number of bytes in flight: 389]
Data (389 bytes)

0000  47 45 54 20 2f 69 6e 74 65 72 66 61 63 65 73 2f   GET /interfaces/
...
0100  54 54 50 2f 31 2e 30 0d 0a 52 61 6e 67 65 3a 20   TTP/1.0..Range: 
0110  62 79 74 65 73 3d 37 35 35 37 30 32 33 35 39 36   bytes=7557023596
0120  2d 0d 0a 55 73 65 72 2d 41 67 65 6e 74 3a 20 57   -..User-Agent: W
0130  67 65 74 2f 31 2e 31 31 2e 34 0d 0a 41 63 63 65   get/1.11.4..Acce
0140  70 74 3a 20 2a 2f 2a 0d 0a 48 6f 73 74 3a 20 73   pt: */*.
0170  63 74 69 6f 6e 3a 20 4b 65 65 70 2d 41 6c 69 76   ction: Keep-Aliv
0180  65 0d 0a 0d 0a                                    e....

Edit: Fixed title

5
  • 1
    I doubt the window is just 517 bytes. A window of 517 does not necessarily mean 517 bytes -- see RFC1323. We'd have to see the connection setup to know what it means. Apr 17, 2012 at 10:56
  • And by 'connection setup' we mean the first set of packets as the TCP connection is set up.
    – sysadmin1138
    Apr 17, 2012 at 11:18
  • There are some packets listed there with len=1440. Are they getting lost on your unusual link? What sort of link is it? Another suspicion: is it failing after 2GB / 4GB of data?
    – pjc50
    Apr 17, 2012 at 12:01
  • Ah, right! Obviously due to the ring buffer wireshark didn't know about the window scaling, therefore it reports a just started conenction as 132352, but in all following files as just 517. Thanks David for pointing me to that again. Apr 17, 2012 at 12:57
  • The len=1440 seems to be a TCP send buffer flush once the receive buffer is free again. I don't think they are lost, this wireshark is running on the receivers side. Apr 17, 2012 at 13:07

2 Answers 2

2

Window size at zero means the OS has received all the data it can in its receive buffer for this connection and is waiting for the application to read() it. In other words, if you don't read the data the server is sending you, as fast as it's sending it, at some point it will block like this. How are you using wget, can you paste the exact script/command? Same with your java program, are you sure it's reading all the data from the server, and never getting backed up, or missing an event/notification that data is ready, or having a problem where it can't keep up with its read()s?

Also, check netstat -anp and look at the Recv-Q column for this connection, if it's not 0 then there's data buffered up, waiting for your app to read it. Edit: argh, Windows, this link might help.

1

Sorry for the delayed update guys. Long story short, reason behind all this was a broken NAT in the Microsoft Azure cloud. It is ignoring RFC 5382. Their timeout on a TCP connection is 60s. When that timeout is hit, the connection is not actively closed (RST or so, as suggested in the RFC), or just ignored. Instead a very weird translation happening, making the packages arrive with garbeld source/destination IPs on the correct VM. The VM throws those away, but this made caputing them in wireshark quite tricky. It will see them as a totally different connection.

Solution woraround was a 60s application level heartbeat.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .