Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I'm running nginx in a OpenVZ container on Debian 6 and when proxying I get truncated responses. Most of the time it happens when I proxy another proxy like varnish or apache (for svn servers). I already checked tcp send and receive buffer sizes and after tweaking them I don't get errors for them in /proc/user_beancounter anymore and the truncation happens less, but it still happens on some files and on those it does it consistently. Today I even had truncation while curling a Python wsgi server directly inside the OpenVZ container, so I'm leaning more and more towards a problem with OpenVZ or the container config. I already ruled out issues with the temporary buffering to a file of responses by nginx (got that from a related question). Would be great if someone could give me some tips on narrowing down the problem or even having a solution :)

edit 1.11.2011:

Here is the output of a tcp-dump between a nginx and a varnish instance where truncation happens. The point of truncation varies.

# tcpdump port 6081
tcpdump: WARNING: venet0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on venet0, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
13:47:52.429724 IP 10.1.10.10.34318 > extrem-test.6081: Flags [S], seq 3858311307, win 5840, options [mss 1460,sackOK,TS val 628244007 ecr 0,nop,wscale 7], length 0
13:47:52.429751 IP extrem-test.6081 > 10.1.10.10.34318: Flags [S.], seq 4285975234, ack 3858311308, win 5792, options [mss 1460,sackOK,TS val 628244007 ecr 628244007,nop,wscale 7], length 0
13:47:52.429779 IP 10.1.10.10.34318 > extrem-test.6081: Flags [.], ack 1, win 46, options [nop,nop,TS val 628244007 ecr 628244007], length 0
13:47:52.429801 IP 10.1.10.10.34318 > extrem-test.6081: Flags [P.], seq 1:970, ack 1, win 46, options [nop,nop,TS val 628244007 ecr 628244007], length 969
13:47:52.429812 IP extrem-test.6081 > 10.1.10.10.34318: Flags [.], ack 970, win 61, options [nop,nop,TS val 628244007 ecr 628244007], length 0
13:47:52.430004 IP extrem-test.6081 > 10.1.10.10.34318: Flags [.], seq 1:1449, ack 970, win 61, options [nop,nop,TS val 628244007 ecr 628244007], length 1448
13:47:52.430026 IP 10.1.10.10.34318 > extrem-test.6081: Flags [.], ack 1449, win 69, options [nop,nop,TS val 628244007 ecr 628244007], length 0
13:47:52.430036 IP extrem-test.6081 > 10.1.10.10.34318: Flags [.], seq 1449:2897, ack 970, win 61, options [nop,nop,TS val 628244007 ecr 628244007], length 1448
13:47:52.430052 IP 10.1.10.10.34318 > extrem-test.6081: Flags [.], ack 2897, win 91, options [nop,nop,TS val 628244007 ecr 628244007], length 0
13:47:52.430060 IP extrem-test.6081 > 10.1.10.10.34318: Flags [P.], seq 2897:4345, ack 970, win 61, options [nop,nop,TS val 628244007 ecr 628244007], length 1448
13:47:52.430085 IP extrem-test.6081 > 10.1.10.10.34318: Flags [.], seq 4345:5793, ack 970, win 61, options [nop,nop,TS val 628244007 ecr 628244007], length 1448
13:47:52.430084 IP 10.1.10.10.34318 > extrem-test.6081: Flags [.], ack 4345, win 114, options [nop,nop,TS val 628244007 ecr 628244007], length 0
13:47:52.430090 IP extrem-test.6081 > 10.1.10.10.34318: Flags [.], seq 5793:7241, ack 970, win 61, options [nop,nop,TS val 628244007 ecr 628244007], length 1448
13:47:52.430098 IP extrem-test.6081 > 10.1.10.10.34318: Flags [P.], seq 7241:8689, ack 970, win 61, options [nop,nop,TS val 628244007 ecr 628244007], length 1448
13:47:52.430102 IP extrem-test.6081 > 10.1.10.10.34318: Flags [.], seq 8689:10137, ack 970, win 61, options [nop,nop,TS val 628244007 ecr 628244007], length 1448
13:47:52.430109 IP extrem-test.6081 > 10.1.10.10.34318: Flags [P.], seq 10137:11585, ack 970, win 61, options [nop,nop,TS val 628244007 ecr 628244007], length 1448
13:47:52.430162 IP 10.1.10.10.34318 > extrem-test.6081: Flags [.], ack 5793, win 137, options [nop,nop,TS val 628244007 ecr 628244007], length 0
13:47:52.430165 IP 10.1.10.10.34318 > extrem-test.6081: Flags [.], ack 7241, win 159, options [nop,nop,TS val 628244007 ecr 628244007], length 0
13:47:52.430167 IP 10.1.10.10.34318 > extrem-test.6081: Flags [.], ack 8689, win 182, options [nop,nop,TS val 628244007 ecr 628244007], length 0
13:47:52.430170 IP 10.1.10.10.34318 > extrem-test.6081: Flags [.], ack 10137, win 204, options [nop,nop,TS val 628244007 ecr 628244007], length 0
13:47:52.430173 IP 10.1.10.10.34318 > extrem-test.6081: Flags [.], ack 11585, win 227, options [nop,nop,TS val 628244007 ecr 628244007], length 0
13:47:52.430202 IP extrem-test.6081 > 10.1.10.10.34318: Flags [.], seq 11585:13033, ack 970, win 61, options [nop,nop,TS val 628244007 ecr 628244007], length 1448
13:47:52.430218 IP 10.1.10.10.34318 > extrem-test.6081: Flags [.], ack 13033, win 250, options [nop,nop,TS val 628244007 ecr 628244007], length 0
13:47:52.430227 IP extrem-test.6081 > 10.1.10.10.34318: Flags [.], seq 13033:14481, ack 970, win 61, options [nop,nop,TS val 628244007 ecr 628244007], length 1448
13:47:52.430243 IP 10.1.10.10.34318 > extrem-test.6081: Flags [.], ack 14481, win 272, options [nop,nop,TS val 628244007 ecr 628244007], length 0
13:47:52.430251 IP extrem-test.6081 > 10.1.10.10.34318: Flags [.], seq 14481:15929, ack 970, win 61, options [nop,nop,TS val 628244007 ecr 628244007], length 1448
13:47:52.430267 IP 10.1.10.10.34318 > extrem-test.6081: Flags [.], ack 15929, win 295, options [nop,nop,TS val 628244007 ecr 628244007], length 0
13:47:52.430275 IP extrem-test.6081 > 10.1.10.10.34318: Flags [.], seq 15929:17377, ack 970, win 61, options [nop,nop,TS val 628244007 ecr 628244007], length 1448
13:47:52.430297 IP 10.1.10.10.34318 > extrem-test.6081: Flags [.], ack 17377, win 318, options [nop,nop,TS val 628244007 ecr 628244007], length 0
13:47:52.430310 IP extrem-test.6081 > 10.1.10.10.34318: Flags [.], seq 17377:18825, ack 970, win 61, options [nop,nop,TS val 628244007 ecr 628244007], length 1448
13:47:52.430326 IP 10.1.10.10.34318 > extrem-test.6081: Flags [.], ack 18825, win 340, options [nop,nop,TS val 628244007 ecr 628244007], length 0
13:47:52.430966 IP 10.1.10.10.34318 > extrem-test.6081: Flags [.], ack 46337, win 392, options [nop,nop,TS val 628244007 ecr 628244007,nop,nop,sack 1 {60817:62265}], length 0
13:47:52.430992 IP extrem-test.6081 > 10.1.10.10.34318: Flags [.], seq 46337:47785, ack 970, win 61, options [nop,nop,TS val 628244007 ecr 628244007], length 1448
13:47:52.430968 IP 10.1.10.10.34318 > extrem-test.6081: Flags [.], ack 47785, win 381, options [nop,nop,TS val 628244007 ecr 628244007,nop,nop,sack 1 {60817:62265}], length 0
13:47:52.431001 IP extrem-test.6081 > 10.1.10.10.34318: Flags [.], seq 47785:49233, ack 970, win 61, options [nop,nop,TS val 628244007 ecr 628244007], length 1448
13:47:52.430970 IP 10.1.10.10.34318 > extrem-test.6081: Flags [.], ack 49233, win 370, options [nop,nop,TS val 628244007 ecr 628244007,nop,nop,sack 1 {60817:62265}], length 0
13:47:52.431009 IP extrem-test.6081 > 10.1.10.10.34318: Flags [.], seq 49233:50681, ack 970, win 61, options [nop,nop,TS val 628244007 ecr 628244007], length 1448
13:47:52.430971 IP 10.1.10.10.34318 > extrem-test.6081: Flags [.], ack 50681, win 359, options [nop,nop,TS val 628244007 ecr 628244007,nop,nop,sack 1 {60817:62265}], length 0
13:47:52.430972 IP 10.1.10.10.34318 > extrem-test.6081: Flags [.], ack 52129, win 348, options [nop,nop,TS val 628244007 ecr 628244007,nop,nop,sack 1 {60817:62265}], length 0
13:47:52.431037 IP extrem-test.6081 > 10.1.10.10.34318: Flags [.], seq 53577:55025, ack 970, win 61, options [nop,nop,TS val 628244007 ecr 628244007], length 1448
13:47:52.430975 IP 10.1.10.10.34318 > extrem-test.6081: Flags [.], ack 53577, win 337, options [nop,nop,TS val 628244007 ecr 628244007,nop,nop,sack 1 {60817:63713}], length 0
13:47:52.431045 IP extrem-test.6081 > 10.1.10.10.34318: Flags [.], seq 55025:56473, ack 970, win 61, options [nop,nop,TS val 628244007 ecr 628244007], length 1448
13:47:52.431058 IP extrem-test.6081 > 10.1.10.10.34318: Flags [.], seq 57921:59369, ack 970, win 61, options [nop,nop,TS val 628244007 ecr 628244007], length 1448
13:47:52.431064 IP extrem-test.6081 > 10.1.10.10.34318: Flags [.], seq 65161:66609, ack 970, win 61, options [nop,nop,TS val 628244007 ecr 628244007], length 1448
13:47:52.431161 IP 10.1.10.10.34318 > extrem-test.6081: Flags [.], ack 65161, win 248, options [nop,nop,TS val 628244007 ecr 628244007,nop,nop,sack 1 {46337:47785}], length 0
13:47:52.431201 IP extrem-test.6081 > 10.1.10.10.34318: Flags [R.], seq 68057, ack 970, win 61, options [nop,nop,TS val 628244007 ecr 628244007], length 0
13:47:52.431164 IP 10.1.10.10.34318 > extrem-test.6081: Flags [.], ack 65161, win 248, options [nop,nop,TS val 628244007 ecr 628244007,nop,nop,sack 1 {47785:49233}], length 0
13:47:52.431222 IP extrem-test.6081 > 10.1.10.10.34318: Flags [R], seq 4286040395, win 0, length 0
13:47:52.431167 IP 10.1.10.10.34318 > extrem-test.6081: Flags [.], ack 65161, win 248, options [nop,nop,TS val 628244007 ecr 628244007,nop,nop,sack 1 {49233:50681}], length 0
13:47:52.431231 IP extrem-test.6081 > 10.1.10.10.34318: Flags [R], seq 4286040395, win 0, length 0
13:47:52.431170 IP 10.1.10.10.34318 > extrem-test.6081: Flags [.], ack 65161, win 248, options [nop,nop,TS val 628244007 ecr 628244007,nop,nop,sack 1 {50681:52129}], length 0
13:47:52.431239 IP extrem-test.6081 > 10.1.10.10.34318: Flags [R], seq 4286040395, win 0, length 0
13:47:52.431172 IP 10.1.10.10.34318 > extrem-test.6081: Flags [.], ack 65161, win 248, options [nop,nop,TS val 628244007 ecr 628244007,nop,nop,sack 1 {52129:53577}], length 0
13:47:52.431246 IP extrem-test.6081 > 10.1.10.10.34318: Flags [R], seq 4286040395, win 0, length 0
13:47:52.431174 IP 10.1.10.10.34318 > extrem-test.6081: Flags [.], ack 65161, win 248, options [nop,nop,TS val 628244007 ecr 628244007,nop,nop,sack 1 {53577:55025}], length 0
13:47:52.431253 IP extrem-test.6081 > 10.1.10.10.34318: Flags [R], seq 4286040395, win 0, length 0
13:47:52.431175 IP 10.1.10.10.34318 > extrem-test.6081: Flags [.], ack 65161, win 248, options [nop,nop,TS val 628244007 ecr 628244007,nop,nop,sack 1 {55025:56473}], length 0
13:47:52.431260 IP extrem-test.6081 > 10.1.10.10.34318: Flags [R], seq 4286040395, win 0, length 0
13:47:52.431178 IP 10.1.10.10.34318 > extrem-test.6081: Flags [.], ack 65161, win 248, options [nop,nop,TS val 628244007 ecr 628244007,nop,nop,sack 1 {56473:57921}], length 0
13:47:52.431265 IP extrem-test.6081 > 10.1.10.10.34318: Flags [R], seq 4286040395, win 0, length 0
13:47:52.431180 IP 10.1.10.10.34318 > extrem-test.6081: Flags [.], ack 65161, win 248, options [nop,nop,TS val 628244007 ecr 628244007,nop,nop,sack 1 {57921:59369}], length 0
share|improve this question
    
It sounds like you can reproduce the problem without the proxy? Can you capture when the truncation happens with tcpdump? –  Aleksandr Levchuk Oct 11 '11 at 19:59
    
I have added tcpdump output. –  fschulze Nov 1 '11 at 12:53
    
If you run the werbserver on the hardware node does the truncation happen? –  Aleksandr Levchuk Nov 4 '11 at 13:17
    
I want to avoid messing with the hardware node. I found this tech.blog.aknin.name/2011/11/04/… and it's exactly my issue. Now I just have to find out why the backend fails, I suspect some configuration/limits issue with openvz, but I don't see any failures in /proc/user_beancounters. –  fschulze Nov 6 '11 at 17:34

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.