So I setup a linux server and forgot to disable clear text ssh password or install denyhosts or enable any kind of password policy. Usually I have deny hosts and it works well. As a result of missing this vital step (yes I should automate the process) a user with a weak password has been hacked. Now on the assumption that the general permissions are good what can I do to work out what they did and remove it?
By the way I am a programmer by nature not a system admin so please be kind!
You can never be completely sure what they did on the user account. But places to start are the .*history files in the home directory.
My advice would be to copy out the known good/important data and then blow the rest away. The intruder could have left any sorts of nasty surprises in configuration files, .bashrc, etc.
You should also check to see if any files owned by the user are on the system and look for running processes:
# find / -user USERNAME
# ps -a -u USERNAME
For the future, I would advise turning on process accounting. You can then check previously run commands using 'lastcomm'.
If you have a backup handy, you could compare that to the current filesystem to see what has changed. Pay particular attention to directories that would normally be in the path, such as /bin, /usr/bin, /usr/local/bin, /sbin, /usr/sbin, /usr/local/sbin, /opt/bin, etc.
Also look for a rootkit: A list of Windows rootkit detection and removal tools
But you can't know that you found everything that they did. Better to go back to a known good state (e.g., last backup) and bring the carefully checked data that has changed since then with you. Even better would be to wipe the system, and install an IDS before hooking it up to the network.
Unless you are running aide or tripwire or something similar, you don't have too many other options.
If the suspected compromised account is unprivileged, there are only a handful of places the attacker may hide a persistent malware. Clean these places properly and there is no need to blow the account completely.
She'll initialization files: .bashrc, .bash_profile, .profile, .bash_login, etc. For non-bash shells, check the corresponding man page.
If the system uses systemD then there is another vector via systemd --user instance. The user-defined systemD units go into ~/.config and ~/.local [1].
Cron jobs. Depends on your cron flavor, but typically somewhere in /var/lib/...
Desktop config file. Typically in ~/.config and ~/.local. To be safe, remove these dirs.
Pam_env config in ~/.pam_environment.
Also note that it is NOT safe to do the cleanup from a running system because systemD allows lingering of users via login to, which means that user units may be started on every boot. Fortunately, one needs polkit installed for the attacker to be able to linger the compromised user... if you have polkit, you are in a world of pain :)
Hence, it is best to do the cleanup from initramfs by passing init=/bin/bash to the kernel command line.
You should check the users history and roots history.
Look into /tmp for suspicious files (source code, executables, files owned by that user).
Use http://www.chkrootkit.org/ and/or http://rkhunter.sourceforge.net to check the system.
pstree/top/ps aux to check for running processes.
Look at the logfiles in /var/log for the specific time of the hack, if you have it.
Clear the user account by moving important data if any. Check for users command history to see if any script was run or any non-required command was run. Delete the user from the system and remove the home directory. Check if any cronjobs were set. Check all processes in detail to see if there was any background processes set by the compromised account.
Hope it helps.
