5

We're seeing some very strange mangling of HTTP responses, and we can't figure out what is doing it. We have an app server handling JSON requests. Occasionally, the response is returned gzipped, but with incorrect headers that prevent the browser from interpreting it correctly.

The problem is intermittent, and changes behavior over time. Yesterday morning it seemed to fail 50% of the time, and in fact, seemed tied to one of our two load-balanced servers. Later in the afternoon, it was failing only 20 times out of 1000, and didn't correlate with an app server.

The two app servers are running Apache 2.2 with mod_wsgi and a Django app stack. They have identical Apache configs and source trees, and even identical packages installed on Red Hat. There's a hardware load balancer in front, I don't know the make or model.

Akamai is also part of the food chain, though we removed Akamai and still had the problem.

Here's a good request and response:

* Connected to example.com (97.7.79.129) port 80 (#0)
> POST /claim/ HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-pc-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8k zlib/1.2.3.3 libidn/1.15
> Host: example.com
> Accept: */*
> Referer: http://example.com/apps/
> Accept-Encoding: gzip,deflate
> Content-Length: 29
> Content-Type: application/x-www-form-urlencoded
> 
} [data not shown]
< HTTP/1.1 200 OK
< Server: Apache/2
< Content-Language: en-us
< Content-Encoding: identity
< Content-Length: 47
< Content-Type: application/x-javascript
< Connection: keep-alive
< Vary: Accept-Encoding
< 
{ [data not shown]
* Connection #0 to host example.com left intact
* Closing connection #0
{"msg": "", "status": "OK", "printer_name": ""}

And here's a bad one:

* Connected to example.com (97.7.79.129) port 80 (#0)
> POST /claim/ HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-pc-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8k zlib/1.2.3.3 libidn/1.15
> Host: example.com
> Accept: */*
> Referer: http://example.com/apps/
> Accept-Encoding: gzip,deflate
> Content-Length: 29
> Content-Type: application/x-www-form-urlencoded
> 
} [data not shown]
< HTTP/1.1 200 OK
< Server: Apache/2
< Content-Language: en-us
< Content-Encoding: identity
< Content-Type: application/x-javascript
< Content-Encoding: gzip
< Content-Length: 59
< Connection: keep-alive
< Vary: Accept-Encoding
< X-N: S
< 
{ [data not shown]
* Connection #0 to host example.com left intact
* Closing connection #0
�V�-NW�RPR�QP*.I,)-���A���̼�Ԣ����T��Z�
��/

There are two things to notice about the bad response:

  1. It has two Content-Encoding headers, and the browsers seem to use the first. So they see an identity encoding header, and gzipped content, so they can't interpret the response.

  2. The bad response has an extra "X-N: S" header.

Perhaps if I could find out what intermediary adds "X-N: S" headers to responses, I could track down the culprit...

  • Which exact versions of Apache and mod_wsgi? Are the exact same versions used on each? Are they the latest or some older version? – Graham Dumpleton Apr 18 '10 at 0:11
  • They are the exact same versions on each machine (the entire population of packages is exactly the same). – Ned Batchelder Apr 18 '10 at 12:30
  • Ned eventually found out where the "X-N: S" response header came from. See his blog post for details. – ulty4life Oct 26 '12 at 19:11
3

Some additional clues

According to http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.5:

  • identity The default (identity) encoding; the use of no transformation whatsoever. This content-coding is used only in the Accept- Encoding header, and SHOULD NOT be used in the Content-Encoding header.

Akamai seems to be ignoring the fact that a server could include this header in their response and is not removing it when the change the encoding to "gzip".

Since the upstream server "should" not be adding the header in the first place, that is another way to fix this problem.

  • We got confirmation from Akamai that they are gzipping the response and adding a "Content-Encoding: gzip" without first removing the existing Content-Encoding. We've identified the place in our code that adds the identity header, and removed it. Still no word on what "X-N:S" means! :) – Ned Batchelder Apr 22 '10 at 11:55
  • X-N: community.akamai.com/community/web-performance/blog/2016/05/24/…. "This header is added to accommodate a bug in Netscape when the response headers have a certain length. If the terminating \r\n of the header begins on character 256, 257, or 258, then Netscape used to display a broken image. Akamai fix this by padding the headers with X-N: S." – Jason Martin Jul 17 '17 at 1:15
0

What does your architecture look like?

Any reverse proxy in the request path? What modules are loaded in apache? What does the apache config look like? If you disable mod_deflate, does it still occur? Is a php script processing your json output that uses ob_start and handles its own compression?

0

I suspect caching at the load-balancer. Try adding anti-caching headers in the response and see if the problem persists. This may increase your server load alot.

  • These are JSON posts, so I don't expect to have any caching. There's no evidence that something is caching, judging from the Apache logs. – Ned Batchelder Apr 17 '10 at 23:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.