My server's been hacked EMERGENCY
We've got a production server running Ubuntu 11.10. We're encountering segfaults that appear under various conditions. The simplest reproducible case is when we login to an ssh session as our administrative user and enter a bogus command. You'd expect the standard "command not found" error message. Instead, we get a segfault in python. The user's default shell is /bin/bash.
$ asdf Segmentation fault Info from /var/log/syslog: Jul 6 15:39:20 PROD001 kernel: [2155960.605695] python: segfault at 0 ip (null) sp 00007fffd030b808 error 14 in python2.7[400000+233000]
Some details about the server:
$ uname -a Linux PROD001 3.0.0-16-server #29-Ubuntu SMP Tue Feb 14 13:08:12 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux $ cat /etc/issue Ubuntu 11.10 \n \l
Before we ask the IT department to reinstall the O.S., I'd like to understand what got us here. The system and/or this particular user's environment is suspect. Many people have touched this server over the past year, so I'm wondering if it is missing libraries, incorrectly installed packages, etc. I'm hoping that if we can understand what's going wrong in this case, it will help explain why we're getting segfaults in a couple of other scenarios.
Any tips on troubleshooting this segfault will be appreciated!