Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We're seeing some very strange mangling of HTTP responses, and we can't figure out what is doing it. We have an app server handling JSON requests. Occasionally, the response is returned gzipped, but with incorrect headers that prevent the browser from interpreting it correctly.

The problem is intermittent, and changes behavior over time. Yesterday morning it seemed to fail 50% of the time, and in fact, seemed tied to one of our two load-balanced servers. Later in the afternoon, it was failing only 20 times out of 1000, and didn't correlate with an app server.

The two app servers are running Apache 2.2 with mod_wsgi and a Django app stack. They have identical Apache configs and source trees, and even identical packages installed on Red Hat. There's a hardware load balancer in front, I don't know the make or model.

Akamai is also part of the food chain, though we removed Akamai and still had the problem.

Here's a good request and response:

* Connected to example.com (97.7.79.129) port 80 (#0)
> POST /claim/ HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-pc-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8k zlib/1.2.3.3 libidn/1.15
> Host: example.com
> Accept: */*
> Referer: http://example.com/apps/
> Accept-Encoding: gzip,deflate
> Content-Length: 29
> Content-Type: application/x-www-form-urlencoded
> 
} [data not shown]
< HTTP/1.1 200 OK
< Server: Apache/2
< Content-Language: en-us
< Content-Encoding: identity
< Content-Length: 47
< Content-Type: application/x-javascript
< Connection: keep-alive
< Vary: Accept-Encoding
< 
{ [data not shown]
* Connection #0 to host example.com left intact
* Closing connection #0
{"msg": "", "status": "OK", "printer_name": ""}

And here's a bad one:

* Connected to example.com (97.7.79.129) port 80 (#0)
> POST /claim/ HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-pc-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8k zlib/1.2.3.3 libidn/1.15
> Host: example.com
> Accept: */*
> Referer: http://example.com/apps/
> Accept-Encoding: gzip,deflate
> Content-Length: 29
> Content-Type: application/x-www-form-urlencoded
> 
} [data not shown]
< HTTP/1.1 200 OK
< Server: Apache/2
< Content-Language: en-us
< Content-Encoding: identity
< Content-Type: application/x-javascript
< Content-Encoding: gzip
< Content-Length: 59
< Connection: keep-alive
< Vary: Accept-Encoding
< X-N: S
< 
{ [data not shown]
* Connection #0 to host example.com left intact
* Closing connection #0
�V�-NW�RPR�QP*.I,)-���A���̼�Ԣ����T��Z�
��/

There are two things to notice about the bad response:

  1. It has two Content-Encoding headers, and the browsers seem to use the first. So they see an identity encoding header, and gzipped content, so they can't interpret the response.

  2. The bad response has an extra "X-N: S" header.

Perhaps if I could find out what intermediary adds "X-N: S" headers to responses, I could track down the culprit...

share|improve this question
    
Which exact versions of Apache and mod_wsgi? Are the exact same versions used on each? Are they the latest or some older version? –  Graham Dumpleton Apr 18 '10 at 0:11
    
They are the exact same versions on each machine (the entire population of packages is exactly the same). –  Ned Batchelder Apr 18 '10 at 12:30
    
Ned eventually found out where the "X-N: S" response header came from. See his blog post for details. –  ulty4life Oct 26 '12 at 19:11
add comment

3 Answers

up vote 2 down vote accepted

Some additional clues

According to http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.5:

  • identity The default (identity) encoding; the use of no transformation whatsoever. This content-coding is used only in the Accept- Encoding header, and SHOULD NOT be used in the Content-Encoding header.

Akamai seems to be ignoring the fact that a server could include this header in their response and is not removing it when the change the encoding to "gzip".

Since the upstream server "should" not be adding the header in the first place, that is another way to fix this problem.

share|improve this answer
    
We got confirmation from Akamai that they are gzipping the response and adding a "Content-Encoding: gzip" without first removing the existing Content-Encoding. We've identified the place in our code that adds the identity header, and removed it. Still no word on what "X-N:S" means! :) –  Ned Batchelder Apr 22 '10 at 11:55
add comment

What does your architecture look like?

Any reverse proxy in the request path? What modules are loaded in apache? What does the apache config look like? If you disable mod_deflate, does it still occur? Is a php script processing your json output that uses ob_start and handles its own compression?

share|improve this answer
add comment

I suspect caching at the load-balancer. Try adding anti-caching headers in the response and see if the problem persists. This may increase your server load alot.

share|improve this answer
    
These are JSON posts, so I don't expect to have any caching. There's no evidence that something is caching, judging from the Apache logs. –  Ned Batchelder Apr 17 '10 at 23:55
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.