Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a weird issue on an Ubuntu 10.04 LTS virtual server. A few days ago I started getting these alerts:

***** Nagios *****

Notification Type: PROBLEM

Service: Current Load
Host: localhost
Address: 127.0.0.1
State: CRITICAL

Date/Time: Mon Aug 13 13:09:58 EDT 2012

Now logging in to the server does not work. It seems like I am able to authenticate with ssh but then seems to fail at getting a prompt. I get:

/bin/bash: Exec format error
Connection to 192.172.132.20 closed.

I can't login from any account. Can't try single user as it is remotely hosted. I can connect without starting a shell using ssh -N.

I've received this from cron. It seems like all my /bin files are gone. Is there any way to recover?

/etc/cron.daily/apt:
/bin/sleep: 2: /bin/sleep: ‰Æ‰ø‰ó¹ è›ýÿÿ¸ »: not found
/bin/sleep: 1: /bin/sleep: ‰Æ‰ß‰øèØÿÿÿ ÆN»: not found
/bin/sleep: 2: /bin/sleep: ‰ø1Ò÷ó Â0ˆ N=wìaÃVSP¸-S»Í€[‰Æ[ ø-Í€9Øu ‰ðé 1Àé[^ÃSQRV‰Æ1É   A8 uø‰ÈH^ZY[Ã: not found
/bin/sleep: 1: /bin/sleep: 1ÒŠ ˆ I@C ùwñaÃSQRVW‰Æ‰ßŠ Š 8Øu FGI ùwí¸ _^ZY[ø_^ZY[ÃQSP1À°f1Û³ ‰áÍ€Y[YÃQSP1À°f1Û³ ‰áÍ€X[YÃQSP1À°f1Û³ ‰áÍ€X[YÃQSP1À°f1Û³ ‰áÍ€Y[YÃRQSP1À°f1Û³ ‰áÍ€X[YZÃRQSP1À°f1Û³‰áÍ€X[YZÃQSP1À°f1Û³ ‰áÍ€X[YÃSQRV1ö¸@èÈþÿÿ=t ‰ÆSQ¸·‰ó¹@Í€Y[‰ð^ZY[Ã/dev/hdxSQRVW P01ÿ¸2èˆþÿÿ‰Æ= „†ë Xÿàèøÿÿÿ-5‰ÃˆP SQR¸ ‰Û¹BºÍ€ZY[=ÿÿÿÿtG‰ÂfÇ  fÇF ÇF ÇF ¸ Í€‰FSQR¸7‰Ó¹ ‰òÍ€ZY[=u ¿ ¸-S‰óÍ€[‰ø_^ZY[ÃSQRVWë X‰Ã ë ÿàèðÿÿÿSQR¸ »¹ºÍ€ZY[‰ÙSQ¸0» ‰ÉÍ€Y[_^ZY[Ã: not found
/bin/sleep: 1: /bin/sleep:  ELF    : not found
/bin/sleep: 1: /bin/sleep: Syntax error: word unexpected (expecting ")")
/bin/pidof: 2: /bin/pidof: ‰Æ‰ø‰ó¹ è›ýÿÿ¸ »: not found
/bin/pidof: 1: /bin/pidof: ‰Æ‰ß‰øèØÿÿÿ ÆN»: not found
/bin/pidof: 2: /bin/pidof: ‰ø1Ò÷ó Â0ˆ N=wìaÃVSP¸-S»Í€[‰Æ[ ø-Í€9Øu ‰ðé 1Àé[^ÃSQRV‰Æ1É   A8 uø‰ÈH^ZY[Ã: not found
/bin/pidof: 1: /bin/pidof: 1ÒŠ ˆ I@C ùwñaÃSQRVW‰Æ‰ßŠ Š 8Øu FGI ùwí¸ _^ZY[ø_^ZY[ÃQSP1À°f1Û³ ‰áÍ€Y[YÃQSP1À°f1Û³ ‰áÍ€X[YÃQSP1À°f1Û³ ‰áÍ€X[YÃQSP1À°f1Û³ ‰áÍ€Y[YÃRQSP1À°f1Û³ ‰áÍ€X[YZÃRQSP1À°f1Û³‰áÍ€X[YZÃQSP1À°f1Û³ ‰áÍ€X[YÃSQRV1ö¸@èÈþÿÿ=t ‰ÆSQ¸·‰ó¹@Í€Y[‰ð^ZY[Ã/dev/hdxSQRVW P01ÿ¸2èˆþÿÿ‰Æ= „†ë Xÿàèøÿÿÿ-5‰ÃˆP SQR¸ ‰Û¹BºÍ€ZY[=ÿÿÿÿtG‰ÂfÇ  fÇF ÇF ÇF ¸ Í€‰FSQR¸7‰Ó¹ ‰òÍ€ZY[=u ¿ ¸-S‰óÍ€[‰ø_^ZY[ÃSQRVWë X‰Ã ë ÿàèðÿÿÿSQR¸ »¹ºÍ€ZY[‰ÙSQ¸0» ‰ÉÍ€Y[_^ZY[Ã: not found
/bin/pidof: 1: /bin/pidof:  ELF    : not found
/bin/pidof: 1: /bin/pidof: Syntax error: word unexpected (expecting ")")
/etc/cron.daily/dpkg:
/bin/cp: 2: /bin/cp: ‰Æ‰ø‰ó¹ è›ýÿÿ¸ »: not found
/bin/cp: 1: /bin/cp: ‰Æ‰ß‰øèØÿÿÿ ÆN»: not found
/bin/cp: 2: /bin/cp: ‰ø1Ò÷ó Â0ˆ N=wìaÃVSP¸-S»Í€[‰Æ[ ø-Í€9Øu ‰ðé 1Àé[^ÃSQRV‰Æ1É   A8 uø‰ÈH^ZY[Ã: not found
/bin/cp: 1: /bin/cp: 1ÒŠ ˆ I@C ùwñaÃSQRVW‰Æ‰ßŠ Š 8Øu FGI ùwí¸ _^ZY[ø_^ZY[ÃQSP1À°f1Û³ ‰áÍ€Y[YÃQSP1À°f1Û³ ‰áÍ€X[YÃQSP1À°f1Û³ ‰áÍ€X[YÃQSP1À°f1Û³ ‰áÍ€Y[YÃRQSP1À°f1Û³ ‰áÍ€X[YZÃRQSP1À°f1Û³‰áÍ€X[YZÃQSP1À°f1Û³ ‰áÍ€X[YÃSQRV1ö¸@èÈþÿÿ=t ‰ÆSQ¸·‰ó¹@Í€Y[‰ð^ZY[Ã/dev/hdxSQRVW P01ÿ¸2èˆþÿÿ‰Æ= „†ë Xÿàèøÿÿÿ-5‰ÃˆP SQR¸ ‰Û¹BºÍ€ZY[=ÿÿÿÿtG‰ÂfÇ  fÇF ÇF ÇF ¸ Í€‰FSQR¸7‰Ó¹ ‰òÍ€ZY[=u ¿ ¸-S‰óÍ€[‰ø_^ZY[ÃSQRVWë X‰Ã ë ÿàèðÿÿÿSQR¸ »¹ºÍ€ZY[‰ÙSQ¸0» ‰ÉÍ€Y[_^ZY[Ã: not found
/bin/cp: 1: /bin/cp:  ELF    : not found
/bin/cp: 1: /bin/cp: Syntax error: word unexpected (expecting ")")
/etc/cron.daily/standard:
/bin/cp: 2: /bin/cp: ‰Æ‰ø‰ó¹ è›ýÿÿ¸ »: not found
/bin/cp: 1: /bin/cp: ‰Æ‰ß‰øèØÿÿÿ ÆN»: not found
/bin/cp: 2: /bin/cp: ‰ø1Ò÷ó Â0ˆ N=wìaÃVSP¸-S»Í€[‰Æ[ ø-Í€9Øu ‰ðé 1Àé[^ÃSQRV‰Æ1É   A8 uø‰ÈH^ZY[Ã: not found
/bin/cp: 1: /bin/cp: 1ÒŠ ˆ I@C ùwñaÃSQRVW‰Æ‰ßŠ Š 8Øu FGI ùwí¸ _^ZY[ø_^ZY[ÃQSP1À°f1Û³ ‰áÍ€Y[YÃQSP1À°f1Û³ ‰áÍ€X[YÃQSP1À°f1Û³ ‰áÍ€X[YÃQSP1À°f1Û³ ‰áÍ€Y[YÃRQSP1À°f1Û³ ‰áÍ€X[YZÃRQSP1À°f1Û³‰áÍ€X[YZÃQSP1À°f1Û³ ‰áÍ€X[YÃSQRV1ö¸@èÈþÿÿ=t ‰ÆSQ¸·‰ó¹@Í€Y[‰ð^ZY[Ã/dev/hdxSQRVW P01ÿ¸2èˆþÿÿ‰Æ= „†ë Xÿàèøÿÿÿ-5‰ÃˆP SQR¸ ‰Û¹BºÍ€ZY[=ÿÿÿÿtG‰ÂfÇ  fÇF ÇF ÇF ¸ Í€‰FSQR¸7‰Ó¹ ‰òÍ€ZY[=u ¿ ¸-S‰óÍ€[‰ø_^ZY[ÃSQRVWë X‰Ã ë ÿàèðÿÿÿSQR¸ »¹ºÍ€ZY[‰ÙSQ¸0» ‰ÉÍ€Y[_^ZY[Ã: not found
/bin/cp: 1: /bin/cp:  ELF    : not found
/bin/cp: 1: /bin/cp: Syntax error: word unexpected (expecting ")")
/bin/awk: 2: /bin/awk: ‰Æ‰ø‰ó¹ è›ýÿÿ¸ »: not found
/bin/awk: 1: /bin/awk: ‰Æ‰ß‰øèØÿÿÿ ÆN»: not found
/bin/awk: 2: /bin/awk: ‰ø1Ò÷ó Â0ˆ N=wìaÃVSP¸-S»Í€[‰Æ[ ø-Í€9Øu ‰ðé 1Àé[^ÃSQRV‰Æ1É   A8 uø‰ÈH^ZY[Ã: not found
/bin/awk: 1: /bin/awk: 1ÒŠ ˆ I@C ùwñaÃSQRVW‰Æ‰ßŠ Š 8Øu FGI ùwí¸ _^ZY[ø_^ZY[ÃQSP1À°f1Û³ ‰áÍ€Y[YÃQSP1À°f1Û³ ‰áÍ€X[YÃQSP1À°f1Û³ ‰áÍ€X[YÃQSP1À°f1Û³ ‰áÍ€Y[YÃRQSP1À°f1Û³ ‰áÍ€X[YZÃRQSP1À°f1Û³‰áÍ€X[YZÃQSP1À°f1Û³ ‰áÍ€X[YÃSQRV1ö¸@èÈþÿÿ=t ‰ÆSQ¸·‰ó¹@Í€Y[‰ð^ZY[Ã/dev/hdxSQRVW P01ÿ¸2èˆþÿÿ‰Æ= „†ë Xÿàèøÿÿÿ-5‰ÃˆP SQR¸ ‰Û¹BºÍ€ZY[=ÿÿÿÿtG‰ÂfÇ  fÇF ÇF ÇF ¸ Í€‰FSQR¸7‰Ó¹ ‰òÍ€ZY[=u ¿ ¸-S‰óÍ€[‰ø_^ZY[ÃSQRVWë X‰Ã ë ÿàèðÿÿÿSQR¸ »¹ºÍ€ZY[‰ÙSQ¸0» ‰ÉÍ€Y[_^ZY[Ã: not found
/bin/awk: 1: /bin/awk:  ELF    : not found
/bin/awk: 1: /bin/awk: Syntax error: word unexpected (expecting ")")
/etc/cron.daily/thedailycommandline:
run-parts: failed to exec /etc/cron.daily/thedailycommandline: Exec format error
run-parts: /etc/cron.daily/thedailycommandline exited with return code 1

Any help would be greatly appreciated.

share|improve this question

2 Answers 2

Can you login to a different account (if one exists)? Can you login to the console? Or reboot in single user mode and login? Can you connect with something like WinSCP, which shouldn't start a shell?

That error may mean you have a bad login script.

It could also mean that bash has been replaced with a corrupt or different architecture's bash, which might be a sign that someone tried to install a rootkit.

share|improve this answer
    
Whatever happened to /bin/bash has also happened to sleep, cp, awk, and pidof. The cron scripts are getting "Exec format error" on them and falling back on the "interpret as shell script" behavior, which blows up because /bin/cp is not a shell script. –  Alan Curry Aug 15 '12 at 0:05
    
Yep...looking like either incompetent hackers or hard drive corruption. Either way time to restore from your backups. –  Grant Aug 15 '12 at 2:40

That paste looks like severe filesystem corruption.

At this point, were it my server, I would:

  • Check the disks for hardware failure and replace any disk that has failed, is failing, or even looks like it might fail in the near future.
  • Reinstall the OS and restore my data from the most recent good backup, checking them carefully to ensure that I'm not restoring corrupt data. This could have been going on for quite a long time without your knowledge, so check your backup carefully before restoring it.
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.