1

I have a weird issue on an Ubuntu 10.04 LTS virtual server. A few days ago I started getting these alerts:

***** Nagios *****

Notification Type: PROBLEM

Service: Current Load
Host: localhost
Address: 127.0.0.1
State: CRITICAL

Date/Time: Mon Aug 13 13:09:58 EDT 2012

Now logging in to the server does not work. It seems like I am able to authenticate with ssh but then seems to fail at getting a prompt. I get:

/bin/bash: Exec format error
Connection to 192.172.132.20 closed.

I can't login from any account. Can't try single user as it is remotely hosted. I can connect without starting a shell using ssh -N.

I've received this from cron. It seems like all my /bin files are gone. Is there any way to recover?

/etc/cron.daily/apt:
/bin/sleep: 2: /bin/sleep: ‰Æ‰ø‰ó¹ è›ýÿÿ¸ »: not found
/bin/sleep: 1: /bin/sleep: ‰Æ‰ß‰øèØÿÿÿ ÆN»: not found
/bin/sleep: 2: /bin/sleep: ‰ø1Ò÷ó Â0ˆ N=wìaÃVSP¸-S»Í€[‰Æ[ ø-Í€9Øu ‰ðé 1Àé[^ÃSQRV‰Æ1É   A8 uø‰ÈH^ZY[Ã: not found
/bin/sleep: 1: /bin/sleep: 1ÒŠ ˆ I@C ùwñaÃSQRVW‰Æ‰ßŠ Š 8Øu FGI ùwí¸ _^ZY[ø_^ZY[ÃQSP1À°f1Û³ ‰áÍ€Y[YÃQSP1À°f1Û³ ‰áÍ€X[YÃQSP1À°f1Û³ ‰áÍ€X[YÃQSP1À°f1Û³ ‰áÍ€Y[YÃRQSP1À°f1Û³ ‰áÍ€X[YZÃRQSP1À°f1Û³‰áÍ€X[YZÃQSP1À°f1Û³ ‰áÍ€X[YÃSQRV1ö¸@èÈþÿÿ=t ‰ÆSQ¸·‰ó¹@Í€Y[‰ð^ZY[Ã/dev/hdxSQRVW P01ÿ¸2èˆþÿÿ‰Æ= „†ë Xÿàèøÿÿÿ-5‰ÃˆP SQR¸ ‰Û¹BºÍ€ZY[=ÿÿÿÿtG‰ÂfÇ  fÇF ÇF ÇF ¸ Í€‰FSQR¸7‰Ó¹ ‰òÍ€ZY[=u ¿ ¸-S‰óÍ€[‰ø_^ZY[ÃSQRVWë X‰Ã ë ÿàèðÿÿÿSQR¸ »¹ºÍ€ZY[‰ÙSQ¸0» ‰ÉÍ€Y[_^ZY[Ã: not found
/bin/sleep: 1: /bin/sleep:  ELF    : not found
/bin/sleep: 1: /bin/sleep: Syntax error: word unexpected (expecting ")")
/bin/pidof: 2: /bin/pidof: ‰Æ‰ø‰ó¹ è›ýÿÿ¸ »: not found
/bin/pidof: 1: /bin/pidof: ‰Æ‰ß‰øèØÿÿÿ ÆN»: not found
/bin/pidof: 2: /bin/pidof: ‰ø1Ò÷ó Â0ˆ N=wìaÃVSP¸-S»Í€[‰Æ[ ø-Í€9Øu ‰ðé 1Àé[^ÃSQRV‰Æ1É   A8 uø‰ÈH^ZY[Ã: not found
/bin/pidof: 1: /bin/pidof: 1ÒŠ ˆ I@C ùwñaÃSQRVW‰Æ‰ßŠ Š 8Øu FGI ùwí¸ _^ZY[ø_^ZY[ÃQSP1À°f1Û³ ‰áÍ€Y[YÃQSP1À°f1Û³ ‰áÍ€X[YÃQSP1À°f1Û³ ‰áÍ€X[YÃQSP1À°f1Û³ ‰áÍ€Y[YÃRQSP1À°f1Û³ ‰áÍ€X[YZÃRQSP1À°f1Û³‰áÍ€X[YZÃQSP1À°f1Û³ ‰áÍ€X[YÃSQRV1ö¸@èÈþÿÿ=t ‰ÆSQ¸·‰ó¹@Í€Y[‰ð^ZY[Ã/dev/hdxSQRVW P01ÿ¸2èˆþÿÿ‰Æ= „†ë Xÿàèøÿÿÿ-5‰ÃˆP SQR¸ ‰Û¹BºÍ€ZY[=ÿÿÿÿtG‰ÂfÇ  fÇF ÇF ÇF ¸ Í€‰FSQR¸7‰Ó¹ ‰òÍ€ZY[=u ¿ ¸-S‰óÍ€[‰ø_^ZY[ÃSQRVWë X‰Ã ë ÿàèðÿÿÿSQR¸ »¹ºÍ€ZY[‰ÙSQ¸0» ‰ÉÍ€Y[_^ZY[Ã: not found
/bin/pidof: 1: /bin/pidof:  ELF    : not found
/bin/pidof: 1: /bin/pidof: Syntax error: word unexpected (expecting ")")
/etc/cron.daily/dpkg:
/bin/cp: 2: /bin/cp: ‰Æ‰ø‰ó¹ è›ýÿÿ¸ »: not found
/bin/cp: 1: /bin/cp: ‰Æ‰ß‰øèØÿÿÿ ÆN»: not found
/bin/cp: 2: /bin/cp: ‰ø1Ò÷ó Â0ˆ N=wìaÃVSP¸-S»Í€[‰Æ[ ø-Í€9Øu ‰ðé 1Àé[^ÃSQRV‰Æ1É   A8 uø‰ÈH^ZY[Ã: not found
/bin/cp: 1: /bin/cp: 1ÒŠ ˆ I@C ùwñaÃSQRVW‰Æ‰ßŠ Š 8Øu FGI ùwí¸ _^ZY[ø_^ZY[ÃQSP1À°f1Û³ ‰áÍ€Y[YÃQSP1À°f1Û³ ‰áÍ€X[YÃQSP1À°f1Û³ ‰áÍ€X[YÃQSP1À°f1Û³ ‰áÍ€Y[YÃRQSP1À°f1Û³ ‰áÍ€X[YZÃRQSP1À°f1Û³‰áÍ€X[YZÃQSP1À°f1Û³ ‰áÍ€X[YÃSQRV1ö¸@èÈþÿÿ=t ‰ÆSQ¸·‰ó¹@Í€Y[‰ð^ZY[Ã/dev/hdxSQRVW P01ÿ¸2èˆþÿÿ‰Æ= „†ë Xÿàèøÿÿÿ-5‰ÃˆP SQR¸ ‰Û¹BºÍ€ZY[=ÿÿÿÿtG‰ÂfÇ  fÇF ÇF ÇF ¸ Í€‰FSQR¸7‰Ó¹ ‰òÍ€ZY[=u ¿ ¸-S‰óÍ€[‰ø_^ZY[ÃSQRVWë X‰Ã ë ÿàèðÿÿÿSQR¸ »¹ºÍ€ZY[‰ÙSQ¸0» ‰ÉÍ€Y[_^ZY[Ã: not found
/bin/cp: 1: /bin/cp:  ELF    : not found
/bin/cp: 1: /bin/cp: Syntax error: word unexpected (expecting ")")
/etc/cron.daily/standard:
/bin/cp: 2: /bin/cp: ‰Æ‰ø‰ó¹ è›ýÿÿ¸ »: not found
/bin/cp: 1: /bin/cp: ‰Æ‰ß‰øèØÿÿÿ ÆN»: not found
/bin/cp: 2: /bin/cp: ‰ø1Ò÷ó Â0ˆ N=wìaÃVSP¸-S»Í€[‰Æ[ ø-Í€9Øu ‰ðé 1Àé[^ÃSQRV‰Æ1É   A8 uø‰ÈH^ZY[Ã: not found
/bin/cp: 1: /bin/cp: 1ÒŠ ˆ I@C ùwñaÃSQRVW‰Æ‰ßŠ Š 8Øu FGI ùwí¸ _^ZY[ø_^ZY[ÃQSP1À°f1Û³ ‰áÍ€Y[YÃQSP1À°f1Û³ ‰áÍ€X[YÃQSP1À°f1Û³ ‰áÍ€X[YÃQSP1À°f1Û³ ‰áÍ€Y[YÃRQSP1À°f1Û³ ‰áÍ€X[YZÃRQSP1À°f1Û³‰áÍ€X[YZÃQSP1À°f1Û³ ‰áÍ€X[YÃSQRV1ö¸@èÈþÿÿ=t ‰ÆSQ¸·‰ó¹@Í€Y[‰ð^ZY[Ã/dev/hdxSQRVW P01ÿ¸2èˆþÿÿ‰Æ= „†ë Xÿàèøÿÿÿ-5‰ÃˆP SQR¸ ‰Û¹BºÍ€ZY[=ÿÿÿÿtG‰ÂfÇ  fÇF ÇF ÇF ¸ Í€‰FSQR¸7‰Ó¹ ‰òÍ€ZY[=u ¿ ¸-S‰óÍ€[‰ø_^ZY[ÃSQRVWë X‰Ã ë ÿàèðÿÿÿSQR¸ »¹ºÍ€ZY[‰ÙSQ¸0» ‰ÉÍ€Y[_^ZY[Ã: not found
/bin/cp: 1: /bin/cp:  ELF    : not found
/bin/cp: 1: /bin/cp: Syntax error: word unexpected (expecting ")")
/bin/awk: 2: /bin/awk: ‰Æ‰ø‰ó¹ è›ýÿÿ¸ »: not found
/bin/awk: 1: /bin/awk: ‰Æ‰ß‰øèØÿÿÿ ÆN»: not found
/bin/awk: 2: /bin/awk: ‰ø1Ò÷ó Â0ˆ N=wìaÃVSP¸-S»Í€[‰Æ[ ø-Í€9Øu ‰ðé 1Àé[^ÃSQRV‰Æ1É   A8 uø‰ÈH^ZY[Ã: not found
/bin/awk: 1: /bin/awk: 1ÒŠ ˆ I@C ùwñaÃSQRVW‰Æ‰ßŠ Š 8Øu FGI ùwí¸ _^ZY[ø_^ZY[ÃQSP1À°f1Û³ ‰áÍ€Y[YÃQSP1À°f1Û³ ‰áÍ€X[YÃQSP1À°f1Û³ ‰áÍ€X[YÃQSP1À°f1Û³ ‰áÍ€Y[YÃRQSP1À°f1Û³ ‰áÍ€X[YZÃRQSP1À°f1Û³‰áÍ€X[YZÃQSP1À°f1Û³ ‰áÍ€X[YÃSQRV1ö¸@èÈþÿÿ=t ‰ÆSQ¸·‰ó¹@Í€Y[‰ð^ZY[Ã/dev/hdxSQRVW P01ÿ¸2èˆþÿÿ‰Æ= „†ë Xÿàèøÿÿÿ-5‰ÃˆP SQR¸ ‰Û¹BºÍ€ZY[=ÿÿÿÿtG‰ÂfÇ  fÇF ÇF ÇF ¸ Í€‰FSQR¸7‰Ó¹ ‰òÍ€ZY[=u ¿ ¸-S‰óÍ€[‰ø_^ZY[ÃSQRVWë X‰Ã ë ÿàèðÿÿÿSQR¸ »¹ºÍ€ZY[‰ÙSQ¸0» ‰ÉÍ€Y[_^ZY[Ã: not found
/bin/awk: 1: /bin/awk:  ELF    : not found
/bin/awk: 1: /bin/awk: Syntax error: word unexpected (expecting ")")
/etc/cron.daily/thedailycommandline:
run-parts: failed to exec /etc/cron.daily/thedailycommandline: Exec format error
run-parts: /etc/cron.daily/thedailycommandline exited with return code 1

Any help would be greatly appreciated.

2 Answers 2

1

Can you login to a different account (if one exists)? Can you login to the console? Or reboot in single user mode and login? Can you connect with something like WinSCP, which shouldn't start a shell?

That error may mean you have a bad login script.

It could also mean that bash has been replaced with a corrupt or different architecture's bash, which might be a sign that someone tried to install a rootkit.

2
  • Whatever happened to /bin/bash has also happened to sleep, cp, awk, and pidof. The cron scripts are getting "Exec format error" on them and falling back on the "interpret as shell script" behavior, which blows up because /bin/cp is not a shell script.
    – Alan Curry
    Aug 15, 2012 at 0:05
  • Yep...looking like either incompetent hackers or hard drive corruption. Either way time to restore from your backups.
    – Grant
    Aug 15, 2012 at 2:40
0

That paste looks like severe filesystem corruption.

At this point, were it my server, I would:

  • Check the disks for hardware failure and replace any disk that has failed, is failing, or even looks like it might fail in the near future.
  • Reinstall the OS and restore my data from the most recent good backup, checking them carefully to ensure that I'm not restoring corrupt data. This could have been going on for quite a long time without your knowledge, so check your backup carefully before restoring it.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .