1

I was doing an rsync and accidently reversed the command so it overwrote a bunch a files from our backup server on the main server.

The backup server is in a chroot jail so it overwrote these files:

bin dir:

bash  chroot-jail  cp  ls  mkdir  mv  rm  rmdir  sh  su

dev dir:

null  tty  urandom  zero

lib dir:

pam_access.so    pam_ecryptfs.so   pam_issue.so    pam_listfile.so   pam_passwdqc.so 
pam_rps.so         pam_tally.so      pam_unix_auth.so
pam_ccreds.so    pam_env.so        pam_keyinit.so  pam_localuser.so  pam_permit.so       pam_securetty.so   pam_tally2.so     pam_unix_passwd.so
pam_chroot.so    pam_exec.so       pam_krb5        pam_loginuid.so   pam_pkcs11.so       pam_selinux.so     pam_time.so       pam_unix_session.so
pam_console.so   pam_faildelay.so  pam_krb5.so     pam_mail.so       pam_postgresok.so   pam_shells.so      pam_timestamp.so  pam_userdb.so
pam_cracklib.so  pam_filter        pam_krb5afs.so  pam_mkhomedir.so  pam_pwhistory.so    pam_smb_auth.so    pam_tty_audit.so  pam_warn.so
pam_debug.so     pam_filter.so     pam_lastlog.so  pam_motd.so       pam_rhosts.so       pam_stack.so       pam_umask.so      pam_wheel.so
pam_deny.so      pam_ftp.so        pam_ldap.so     pam_namespace.so  pam_rhosts_auth.so  pam_stress.so      pam_unix.so       pam_xauth.so
pam_echo.so      pam_group.so      pam_limits.so   pam_nologin.so    pam_rootok.so       pam_succeed_if.so  pam_unix_acct.so

lib64:

ld-linux-x86-64.so.2  libaudit.so.0    libcrypt.so.1   libglib-2.0.so.0  
libpam.so.0       libresolv.so.2   libsepol.so.1    libz.so.1
libacl.so.1           libc.so.6        libcrypto.so.6  libkeyutils.so.1  libpam_misc.so.0  librt.so.1       libtermcap.so.2
libattr.so.1          libcom_err.so.2  libdl.so.2      libnsl.so.1       libpthread.so.0   libselinux.so.1  libutil.so.1

sbin:

unix_chkpwd

After doing this, on our main server now I can only login as root, none of the secondary accounts work as the permissions seem to be totally screwed up. Apache is just serving up 403 pages, etc. Any ideas on how I can fix this or is the system hosed?

Accounts still seem to be listed in passwd and shadow.

2
  • As no one has leaped to answer yet, I'll offer a stab in the dark. Maybe you have something like tripwire and it could provide sufficient information that you can reverse the changes.
    – minopret
    Mar 18, 2012 at 4:05
  • 4
    Sounds like time to use those backups.
    – Bill Weiss
    Mar 18, 2012 at 5:05

2 Answers 2

0

As Bill Weiss says, you might find it easier and safer to restore from a known good backup on a clean system. You haven't just messed up the permissions, but you've replaced the files, too. Even if both the source and target machines were the same distribution, there's no guarantee they'll be identical files, and that can cause all sorts of wacky problems.

If you feel you must attempt a restore, there's two ways I an think of going about it. The first is to use your distro's package manager to to a force-reinstall of the packages owning the affected files. The second is the more manual approach:

  • Use your system's package manager to find out what owns every one of those files.
  • Set up a new system and ensure you install exactly the same versions of those packages on it
  • Upload a copy of the files to the server
  • Boot the server in single-user mode and carefully replace all the files.
  • Set the permissions/ownership to match.
  • Reboot

You might then find you have a working system again, but I wouldn't trust it and it would probably take longer than the clean/restore route.

Here's a useful command you can run on a list of the full paths of the affected files to get a list of affected packages (Debian/Ubuntu):

for a in `cat files.txt`; do dpkg -S $a |cut -d':' -f1; done |sort -u
1
  • Turns out all that required was a simple chmod fix. When I ran the rsync, I discovered it changed "/" to match that of the chroot, as well as the UID which caused my permission issues. Crisis adverted. Thanks for the tips though!
    – relm
    Mar 18, 2012 at 21:08
0

For the future you may wish to look into rdiff-backup as an alternative to rsync. Specifically because rdiff-backup has a flag to only increment an existing backup, at least then the risk of repeating is gone.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .