Error "Illegal instruction (core dumped)" while running Puppet agent (version 3.2.1)

Question

I just upgraded puppet to version to 3.2.1, and when running the agent, I get an error saying "Illegal instruction (core dumped)". Here is some details:

Both master and agent is run on the same machine

#cat /etc/redhat-release
CentOS release 6.2 (Final)

#uname -r
3.9.3-x86_64-linode33

#ruby --version
ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux]

RPM installed is "puppet-3.2.1-1.el6.noarch.rpm"

However when I run puppet agent with the same version on another host, everything goes fine. Problem is only while running agent on the same machine as the master. Any idea what is going on ?

Answer 1

The solution was to upgrade glibc. See https://bugzilla.redhat.com/show_bug.cgi?id=752122 for details

Answer 2

If your Ruby installation and its dependent libraries are unmodified, there's a good chance that you have a library in your library search path that conflicts with the version Ruby was compiled against. This is really common if, for example, you've built a library like zlib or openssl and dropped it into /usr/local/lib instead of putting it in a custom prefix and setting an RPATH/LD_LIBRARY_PATH for the programs you need compiled against it.

Generically, the best way to diagnose a segfault is to get the kernel to dump a core file (see: ulimit and the core_pattern sysctl), then run it through a debugger like GDB. It can give you a backtrace that, if nothing else, can pretty readily point you at exactly which library's function call is to blame.