Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

PROBLEM:

I have several missing directories on an offsite Linux/Apache server. I've tracked down about when they went missing from the httpd error logs, and I have a list of SSL logins to the machine (complete with network addresses.)

There are several individuals who have access to my account on the machine, so I am trying to confirm whether the files were removed accidentally or intentionally. If I can determine that the session came from an external network address, then I can be sure it was intentional. Otherwise I will need additional information.

QUESTION:

Is there any way to determine what user account (mine vs. root), network session, or process removed the files? I'm not worried about restoring them as backups are recent.

RELATED QUESTION:

I've tried looking at the bash history for my own account (the only one other than root with permission) and it only seems to extend back to the beginning of my SSL session. Does this imply the bash history was tampered with, or is this normal behavior?

share|improve this question

5 Answers 5

up vote 5 down vote accepted

By default, this isn't logged (imagine how big the logs would get). You can add logging with auditctl or inotify/iwatch (interesting options, check them out), or your control panel logging or ftp logs (usually all in /var/log somewhere) or bash history might help you track it down a little. Check your HTTP access logs in case some GET request managed to do this (unlikely, but possible)

If your history isn't there, it has been cleared. This is either done manually, or a command in your .bash_logout or .logout files.

Also, it's SSH not SSL :)

share|improve this answer
5  
It doesn't operate over SSL, it uses The Secure Shell Transport Layer Protocol: tools.ietf.org/html/rfc4253 –  James Lawrie Nov 6 '10 at 2:11
1  
! ... wow, I was 99% percent positive SSH operated over SSL; I have no idea where such a strong misconception came from. Thanks for the heads up. –  robjb Nov 6 '10 at 7:09

Cleanup is always a hard. Resources you can use:

To determine who was logged in, and from where:

  • last / lastlog
  • /var/log/secure (shows successful and failed logins most of the time)

To determine what was going on at the time:

  • ausearch (assuming you are auditing; if you aren't, you might consider starting)
  • sar (to figure out if the system was idle or busy at the time)

If there hasn't been much churn since the incident, the mtime on the directory might have useful info (probably not, but I generally like to collect that kind of data either way)

For the future, get every single person with access to the machine an account of their very own. What you have now is very silly for the reason you have just discovered; You can't know who did what.

share|improve this answer
    
Unfortunately, my boss was who decided to give out my account info to the rest of the IT team... until yesterday, I wasn't even aware. I will have to discuss this with him. –  robjb Nov 6 '10 at 20:01

The bash history only gets written when you close the shell (you can deactivate that for the current session). In a running bash session, you can't see the commands you typed in that session in the bash history file - close the session, start a new one and you should be able to see it.

share|improve this answer
    
The history command will show .bash_history plus any history from the current session, but you're right in that it won't appear in .bash_history –  James Lawrie Nov 6 '10 at 2:12

If you are running process accounting, you may be able to find the call(s) to rm in the processs accounting log.

Shell history may have been removed, or may be configured for session private history. Concurrent sessions can also cause problems with shell history.

share|improve this answer

The only reliable way to do this is to run system accounting (like sar or similar.) Other things might give you the answer, but are not guaranteed:

  1. bash history is not guarateed because (a) there is a history limit; (b) by default it is not written instantly; (c) you have to search through all bash history of all user accounts; (d) history options affect the behavior; (e) A user can clear their own bash history (prevent it from ever being written to file.)

  2. sftp by default does not log transactions, but this can be enabled on the sftp server if needed. files added or removed via sftp or ftp are not logged into a shell history file or any other place.

In your particular case, if you are not already running accounting, then your only recourse is to search through the shell command histories for all the user accounts, in the hopes that it is preserved.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.