Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We have configured a web site with SSI for .html files on Apache/2.2.19 on CentOS 5.5 Linux. Works OK, but occasionally get 403 for the root page (index.html served as a directory index). No file system changes between when page works, doesn't work and then works again.

When it doesn't work it is as though Apache has decided that the index.html file does not exist so tries to serve a directory listing, but that is intentionally denied by our configuration.

Apache virtualhost specific error log contains entries like:

[Tue Aug 09 03:10:47 2011] [error] [client 66.249.72.3] Directory index forbidden by Options directive: /main/directory/

Apache general error log contains nothing.

Apache virtualhost specific access log contains entries like:

66.249.72.3 - - [09/Aug/2011:03:10:47 +0100] "GET / HTTP/1.1" 403 230 "-" "SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)"

Overall distribution:

# fgrep 'GET / ' `ls -tr domain-access_log*` | cut -d ' ' -f 9 | sort | uniq -c
   1339 200
      1 206
     31 304
     29 403

Apache config looks lke:

<VirtualHost ip.address:80>
  ServerName domain.here

  DocumentRoot /main/directory
  ErrorLog logs/domain-error_log
  CustomLog logs/domain-access_log combined
  DirectoryIndex index.html

  AliasMatch ^/(robots\.txt|sitemap.*\.xml(?:\.gz)?)$ /other/directory/$1

  CacheEnable disk /
  CacheDisable /sitemap.xml.gz
  CacheDisable /robots.txt
  CacheIgnoreHeaders Set-Cookie
  CacheIgnoreNoLastMod On

  AddOutputFilterByType DEFLATE text/html text/plain text/xml text/css application/x-javascript

  <Directory /main/directory>
    Options Includes
    AddOutputFilter INCLUDES .html
    Order allow,deny
    Allow from all
  </Directory>

  <Directory /other/directory>
    Order allow,deny
    Allow from all
  </Directory>
</VirtualHost>
share|improve this question
    
What does grep -lir "\-indexes" /etc/httpd/ say? –  quanta Aug 23 '11 at 12:31
    
Are all the denies to bots? Do you have a robots.txt? –  Andrew Case Aug 24 '11 at 23:10
    
I've updated the question. I should have made it clearer: normal behaviour is for Apache to serve the index.html file as a directory index. When it doesn't work it is as though Apache has decided that the index.html file does not exist so tries to serve a directory listing, but that is intentionally denied by our configuration. –  Duncan Harris Aug 25 '11 at 19:36
    
Can you get a full packet capture on one of these requests? And do any real, actual users ever trip this error? I'd suspect something flaky being sent as part of this bot's request. –  Shane Madden Sep 7 '11 at 16:36
add comment

2 Answers 2

Since this is a virtual site we also have to take into account the settings at the httpd.conf level... Do you have Options Multiviews set anywhere? SSI plus language negotiation can lead to it attempting to serve alternate versions of your index.html that don't exist. And then the same thing can lead to it failing to find the ErrorDocument that it wants and giving you even stranger results.

Are there lower-level rewrites taking place?

old The log line you included is a cell phone browser. Do you have other Apache handling that deals with mobile web browsers? Are you doing any Rewrites that match on HTTP_USER_AGENT and send the request elsewhere?

share|improve this answer
    
No, don't have any other configuration. I know it was a cell phone. That is merely a coincidence. I have other examples that aren't from mobile phones. –  Duncan Harris Sep 7 '11 at 6:23
    
Added to my Answer. –  Mark Sep 7 '11 at 13:27
add comment

It looks like you probably just need to add the "Indexes" option in your directory declarations specifically /main/directory.

share|improve this answer
    
I've updated the question. I should have made it clearer: normal behaviour is for Apache to serve the index.html file as a directory index. When it doesn't work it is as though Apache has decided that the index.html file does not exist so tries to serve a directory listing, but that is intentionally denied by our configuration. –  Duncan Harris Aug 25 '11 at 19:37
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.