Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Tried multiple forums to get to the bottom of this. I hope I can get some direction here:

Here is the stack I am working with: Red Hat Enterprise Linux Server release 5.6 (Tikanga) Liferay 6.0.6 on Glassfish 3.0.1 MySQL 5.0.77 Apache 2.2.3

The Liferay portal provides a variety of portlets to end users. Static content (web pages), static resources (primarily pdf and mp3 files 1mb - 80mb in size), File upload and download capabilities (primarily 40-60mb mp3 files) and online streaming of those MP3 files.

Here is the strange end user experiences: Under normal load: (20-30) users uploading, downloading or streaming files and 20-30 accessing static content (some of it downloads), we see the following:

1) Clicking a link triggers the download of a portion of an MP3 (the portion is a few seconds long).

2) Clicking on a link tiggers the download of the page content rather than rendering.

3) Clicking a link causes the page to dump binary data to the end user rather than the expected content.

4) Clicking a link returns the text of a javascript file rather than rendering the page.

Each occurrence is totally random (or appears so). Sometimes it works, sometimes it doesn't. It seems to have no relation to browser or client OS. The strange events seem to occur much more frequently when using an SSL connection rather than regular http.

Apache serves as a proxy server only (reverse). It basically passes all the requests through to Glassfish. There isn't any static content proxy served by Apache.

We rebuilt the entire stack from scratch and redeployed the portlet wars and still have the same issues. Liferay is running as a single server (not clustered). We disabled mod_cache in Apache. The problems are more frequent as the server load grows. This morning the load is pretty light and we are seeing few problems but the use of the site will grow, particularly tonight around 9pm CST through Wednesday morning. You could try the site (http://preview.bsfinternational.org) during those times and I would expect that you might experience one of the weirdnesses as you randomly click links on the site (https is invoked only when signed in). Again, https seems to exacerbate the issue.

This seems very much like a caching issue but I don't know where in the stack to start peeling the onion. Apache? Liferay? Glassfish? MySQL? Maybe even Redhat? We are stumped and most forums we have posted to (LifeRay and Glassfish) have returned very few suggestions. I just need an idea of where to start looking. I understand that we could have a portlet

EDIT: Opening the files in a Hex editor that appear to be pages that download rather than render, we see that the first 4000 characters are "junk" and then the "HTTP/1.1 ...." 'normal' header is seen. So something is dumping a jumble of characters up to offset 4000 (when viewing it in a Hex editor). Perhaps a clue?

EDIT2: The 4000h offset is 16k (16384). I am thinking that this IS a cache issue but don't know where to look for cache settings of this size. I see references to 16k LRU cache buffers in Apache but I know that Glassfish (or perhaps Liferay) uses ehcache. Does THIS trigger any thoughts for someone? Ideas?

share|improve this question
    
Can you post links to the other places where you already asked? This way one could see what else you already tried or if it's been solved already. –  Olaf Oct 26 '12 at 9:31
add comment

2 Answers

Key to finding out where your problem is located is to remove different pieces from the equation - or intercept traffic where you can't.

E.g.:

  • You can try to leave Apache out. If you can't do that for public access, you can try to reproduce by accessing glassfish from behind the firewall while it's also serving content through Apache - or set up a mirror system completely separated from the production system and try to reproduce on that. Another alternative is to wireshark/record the traffic between apache and glassfish
  • If you can reproduce that way (without apache), set up the same system on tomcat, put that under load (maybe artificial load, as you don't want to do that in production, with hardening etc.)
  • If you cannot reproduce without apache, strip down the apache configuration - eliminate gzip, eliminate (or force) https, connect Apache/Glassfish with either mod_jk or mod_proxy (the other one from what you're using right now)

As the problem looks to be rather on the HTTP side (generating illegal HTTP responses), I'd expect the database to be ruled out as the cause.

And some less technical suggestions:

Another general approach for this kind of problem is to have someone else take a look without you telling them what you already did - just sit there and let them explain what they're looking at, maybe you'll find something that didn't yet cross your mind or that you didn't check before.

And, to play tricks on your mind: Use a different computer or desk to conduct the testing: Changing the physical perspective to observe a system sometimes is a good cause for different thoughts. Sounds stupid, but sometimes the brain wants to be tricked.

share|improve this answer
add comment

So here is the answer (not the complete answer but at least the solution). We had a third party portlet that enforced an SSL connection on certain URL's that had \secure in the URL. We removed that portlet and the issue has gone away.

When I get the chance to examine the code, I might be able to figure out what in the portlet was causing the issue. But we have run trouble free for a couple of weeks and the portlet was definitely the cause (AFAIK).

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.