0

I have some box that was handed to me and I can't even start the basic apache server. It's installed via yum (I've uninstalled and installed it) and when I start the service (service httpd start) its says "OK", but there's no pid. Status shows not running.

In the error_log there is the following:

[Wed Feb 13 16:16:36 2013] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Wed Feb 13 16:16:36 2013] [notice] Digest: generating secret for digest authentication ...
[Wed Feb 13 16:16:36 2013] [notice] Digest: done
/usr/sbin/httpd: symbol lookup error: /usr/lib64/libaprutil-1.so.0: undefined symbol: apr_os_uuid_get

I assume it's not starting because of the last line. Suggestions?

ldd $(which httpd)
    linux-vdso.so.1 =>  (0x00007fff9edff000)
    libm.so.6 => /lib64/libm.so.6 (0x00007f4c8815c000)
    libpcre.so.0 => /lib64/libpcre.so.0 (0x00007f4c87f30000)
    libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f4c87d10000)
    libaprutil-1.so.0 => /usr/lib64/libaprutil-1.so.0 (0x00007f4c87aec000)
    libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f4c878b5000)
    libexpat.so.1 => /lib64/libexpat.so.1 (0x00007f4c8768c000)
    libdb-4.7.so => /lib64/libdb-4.7.so (0x00007f4c87318000)
    libapr-1.so.0 => /usr/lib64/libapr-1.so.0 (0x00007f4c870e9000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f4c86ecb000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f4c86b38000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f4c86934000)
    /lib64/ld-linux-x86-64.so.2 (0x000000369be00000)
    libuuid.so.1 => /lib64/libuuid.so.1 (0x00007f4c8672f000)
    libfreebl3.so => /lib64/libfreebl3.so (0x00007f4c864cd000)
    librt.so.1 => /lib64/librt.so.1 (0x00007f4c862c5000)
2
  • It looks like your httpd binary is compiled against libraries which no longer exist, or it was compiled against a version different than the one currently on the disk. What's the output of ldd $(which httpd)
    – Scrivener
    Feb 13, 2013 at 21:31
  • Updated in original. Feb 13, 2013 at 21:34

4 Answers 4

1

After taking a look at the ldd information, it looks like your httpd binary is compiled against different libraries than are currently installed. I would purge as many of the apache dependencies as is safe to do so and reinstall them. Before you do that, though, I would also check to see if you have any non-standard yum repositories installed. Certain of the third-party yum repositories are famous for causing some terrible conflicts that look somewhat similar to this.

For starters:

yum remove apr-devel apr-util-devel httpd

Unfortunately, even if this does solve the problem, you may already be neck deep in system incompatibility. It looks like whoever administered this system before you either compiled things from source, installed RPMs that were incompatible, or misused third-party repos. You may spend less time reinstalling the system than trying to clean it up.

4
  • That didn't get it working, but I understand your point and will probably have to reinstall the system. I even tried a few times to uninstall apache, but no success. Feb 13, 2013 at 21:49
  • I don't think this is the case. ldd is almost the same as in my recently installed CentOS 6.3. Only librt reference is absent in my system, but this is a special kind of library.
    – Tometzky
    Feb 13, 2013 at 21:51
  • any other suggestions to try? Feb 13, 2013 at 21:54
  • I'd do a careful vetting of the yum.repos.d and check the versions to make sure the installed versions of the packages are the same as the latest available in the main CentOS repos... and if they're newer, you can attempt a downgrade.
    – Scrivener
    Feb 13, 2013 at 21:56
2

a) Update all os provided packages:

yum clean all   # Clean yum cache so all fresh data will be downloaded
yum --disablerepo='*' --enablerepo=base --enablerepo=updates update

b) Check if there are installed any additional packages besides available from default CentOS repositories — base and updates:

yum clean all   # Clean yum cache so all fresh data will be downloaded
yum --disablerepo='*' --enablerepo=base --enablerepo=updates list extras

If there are any, which look relevant, then disable all repositories other than these two and downgrade/upgrade these packages to distribution provided versions.

c) Verify files of all packages:

rpm -Va

Only modified configuration files should be printed as modified (look for "5" at 3rd character of each line printed). Certainly nothing in /lib, /lib64, /usr/lib, /usr/lib64, /bin or /usr/bin. If anything there is modified then it looks like somebody has overwritten distribution-provided binaries — either administrator with no clue or a black hat hacker with a rootkit. In that case you can reinstall packages that contain these modified binaries, but a full reinstall is recommended, as it's very hard to find out what else was modified (maybe installing the same set of packages on another computer or virtual machine and comparing).

d) If nothing above helps then I'd suspect a sophisticated rootkit. A full reinstall is strongly recommended then.

1
  • Thank you for your comment. We decided to do a full reinstall. Feb 14, 2013 at 20:04
1

I ran into this issue on centos 5.8. Before removing packages, check your /usr/local/lib - someone installed a later version of apr onto this server and caused this problem (subversion is run on this server with dav, so I needed the httpd to start up).

You can check this by: /usr/sbin/httpd -V

output Server loaded: APR 1.3.3, APR-Util 1.2.7 Compiled using: APR 1.2.7, APR-Util 1.2.7

APR 1.3.3 was in /usr/local/lib. I moved the library of /usr/local/lib (I know this is bad) and then httpd started up.

0

I had a similar problem on centos 6.5 today. error message was:

"/usr/sbin/httpd: symbol lookup error: /usr/lib64/libapr-1.so.0: undefined symbol: dlopen"

I recompiled apache 2.4.10 with apr-1.5.1 and apr-util 1.5.3 from source according to INSTALL instructions.

I downloaded the source to /root. moved apr and apr-util source to srclib/ as directed in INSTALL.

./configure --with-included-apr
make
make install

this created a new httpd and lib in /usr/local/apache2/lib

I compared the lib files in /usr/lib64 with /usr/local/apache2/lib and saw they differed in size.

I copied the newly compiled libapr-1.so.0 into /usr/lib64 and this solved the issue.

service httpd start

[OK}

I also run php-fpm, so started that too:

service php-fpm start

[OK]

back in business! This might help someone else solve this problem.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .