Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Right now, I have an Apache setup sitting in front of Tomcat to handle caching. This setup has been given to an external service to manage, and since the transition, I've noticed odd behavior. Specifically, when I request a swf file from the web server, I hit the Apache cache (good), but occasionally I'll receive a truncated file. Once I receive this truncated file, the cache will NOT refresh until I manually delete the cache and let the swf pull down from tomcat again.

The external service claims that the configuration is fine, but I don't see any way this could be happening aside from improper configuration. Now, there are two apache and two tomcat servers under a load balancer, and occasionally one apache cache will break while another does not (leading to 50% of all requests getting bad, truncated data).

Where should I start looking to debug this issue? What could POSSIBLY be causing this odd behavior?

Edit: Inspecting the logs, tomcat throws this:

java.io.IOException: Bad file number
        at java.io.FileInputStream.readBytes(Native Method)
        at java.io.FileInputStream.read(FileInputStream.java:199)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
        at java.io.FilterInputStream.read(FilterInputStream.java:90)
        at org.apache.catalina.servlets.DefaultServlet.copyRange(DefaultServlet.java:1968)
        at org.apache.catalina.servlets.DefaultServlet.copy(DefaultServlet.java:1714)
        at org.apache.catalina.servlets.DefaultServlet.serveResource(DefaultServlet.java:809)
        at org.apache.catalina.servlets.DefaultServlet.doGet(DefaultServlet.java:325)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:568)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.ha.session.JvmRouteBinderValve.invoke(JvmRouteBinderValve.java:209)
        at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:347)
        at org.terracotta.modules.tomcat.tomcat_5_5.SessionValve55.invoke(SessionValve55.java:57)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
        at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)
        at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:283)
        at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:767)
        at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:697)
        at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889)
        at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
        at java.lang.Thread.run(Thread.java:619)

followed by 

access_log.2009-12-14.txt:1.2.3.4 - - [14/Dec/2009:00:27:32 -0500] "GET /myApp/mySwf.swf HTTP/1.1" 304 -
access_log.2009-12-14.txt:1.2.3.4 - - [14/Dec/2009:01:27:33 -0500] "GET /myApp/mySwf.swf HTTP/1.1" 304 -
access_log.2009-12-14.txt:1.2.3.4 - - [14/Dec/2009:01:39:53 -0500] "GET /myApp/mySwf.swf HTTP/1.1" 304 -
access_log.2009-12-14.txt:1.2.3.4 - - [14/Dec/2009:02:27:38 -0500] "GET /myApp/mySwf.swf HTTP/1.1" 304 -

So apache is caching the bad file size. What could possibly be causing this, and possibly separate, how do I ensure that this exception does not get written to cache?

share|improve this question
    
How's the memory usage on this machine? –  Matt Simmons Jan 8 '10 at 19:43
    
Very low. The server has 32GB of ram. –  Stefan Kendall Jan 8 '10 at 20:06
add comment

2 Answers

up vote 0 down vote accepted
+150

wget, curl or file save as this file?

Do other large files work?

If there's a support contract, I would consider requesting the vendor debug this collaboratively.

share|improve this answer
    
This seems to be the answer: mail-archives.apache.org/mod_mbox/tomcat-dev/200808.mbox/… –  Barry Jan 8 '10 at 20:17
    
Is that still not fixed in 6.0.20? –  Stefan Kendall Jan 8 '10 at 20:21
    
wget and save as both pull down the cached apache file and fail. There aren't many other files on the server that get hit often as large as this file (in fact this file probably gets hit most frequently). –  Stefan Kendall Jan 9 '10 at 3:08
    
If it's still occuring, I would ask the service provider run a sandbox with apache interactively with strace enabled. Also I won't mention a related service we're launching, that would be douchetabular. –  Barry Apr 29 '11 at 21:20
add comment

Tomcat's inability to serve large files has been fixed in trunk, 5.5.x, and the fix fixing the bug is committed to 6.0.27.

share|improve this answer
    
That's great news. I hear 6.0.32 seems the most stable. –  Barry Apr 29 '11 at 21:22
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.