What happened:

I developed a LAMP website and client asked me to install this to their EC2 instance, Amazon Linux AMI release 2017.03

Normally I host them in my Centos server or with some normal webhosting provider.

I installed the site on 17 Aug and everything was working, about 2000 visitors per day.

Client called today, that website is down. Error 500.

I logged in and saw at once that recursively all files in web root folder had lost their all r bits. All other rights, like w, x, and sticky bits were unaffected.

I chmoded +r for user and group and everything started working again.

Error log:

[Wed Sep 06 16:01:01.734067 2017] [mpm_prefork:notice] [pid 14111] AH00171: Graceful restart requested, doing restart
[Wed Sep 06 16:01:02.327167 2017] [auth_digest:notice] [pid 14111] AH01757: generating secret for digest authentication ...
[Wed Sep 06 16:01:02.329737 2017] [lbmethod_heartbeat:notice] [pid 14111] AH02282: No slotmem from mod_heartmonitor
[Wed Sep 06 16:01:02.699353 2017] [mpm_prefork:notice] [pid 14111] AH00163: Apache/2.4.25 (Amazon) configured -- resuming normal operations
[Wed Sep 06 16:01:02.699380 2017] [core:notice] [pid 14111] AH00094: Command line: '/usr/sbin/httpd'
[Wed Sep 06 16:01:02.730492 2017] [:error] [pid 18035] [client x.x.x.x:60180] PHP Warning:  Unknown: failed to open stream: Permission denied in Unknown on line 0
[Wed Sep 06 16:01:02.730559 2017] [:error] [pid 18035] [client x.x.x.x:60180] PHP Fatal error:  Unknown: Failed opening required '/var/www/html/index.php' (include_path='.:/usr/share/pear7:/usr/share/php7') in Unknown on line 0

and the last two lines repeating until the end ...

Nothing special in access log or messages.

Funny thing is, that happened exactly at webserver access logs rotation time. But these logs rotate weekly and without problems before.

Edit: I checked that there was no logins in the last two weeks and it was always my IP even before that.

My question is: What caused the read access to be removed from the files and how to avoid this?

Edit 2: Appears that this was human error. See comment below.

  • Check with dmesg maybe the whole file-system got mounter read-only after a disk error... – HBruijn Sep 7 '17 at 11:27
  • @HBruijn No messages there ... – Rauli Rajande Sep 7 '17 at 11:32
  • Found that amazon issue Linux has following in logrotate: postrotate /etc/init.d/httpd graceful > /dev/null 2>/dev/null || true This explains restart, but not the read bits. – Rauli Rajande Sep 8 '17 at 9:22
  • 1
    Case closed. Appears, that client logged in using my username and messed with file rights during the week before. As AMI Apache2 is configured to restart after logrotate, it seems that it discovered the changed file rights only after that. – Rauli Rajande Sep 11 '17 at 10:37

Monitor your log rotation (even though I doubt that's the cause), and check as soon as it runs next time. Check your cron tasks to make sure that there's no weirdness going on there either.

If that's not the problem, chances are someone ran it accidentally and borked it!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.