Server still vulnerable to HeartBleed after Openssl update

Question

On a Centos 6.5 Minimal install, I have compiled Apache, PHP, and rpm installed Percona.

After updating OpenSSL days ago, my site that uses SSL on this server is vulnerable to Heartbleed somehow.

My Apache binary doesn't show that it is using libssl.so* at all, yet when I check it through https://filippo.io/Heartbleed/ it says my site is in fact vulnerable.

Am I checking something incorrectly?

[root@centos user]# ldd /usr/local/apache2/bin/httpd |grep -i ssl
[root@centos user]#

I've updated OpenSSL through yum (yum update openssl openssl-devel), I've recompiled Apache (four times since the openssl update), I've recompiled PHP (since the update), and I've reinstalled Percona.

I still receive the "Website is vulnerable" message, in spite of the fact that everything is up to date.

[root@centos user]# openssl version -a
OpenSSL 1.0.1e-fips 11 Feb 2013
built on: Tue Apr  8 02:39:29 UTC 2014
platform: linux-x86_64
options:  bn(64,64) md2(int) rc4(16x,int) des(idx,cisc,16,int) idea(int) blowfish(idx) 
compiler: gcc -fPIC -DOPENSSL_PIC -DZLIB -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -DKRB5_MIT -m64 -DL_ENDIAN -DTERMIO -Wall -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -Wa,--noexecstack -DPURIFY -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM
OPENSSLDIR: "/etc/pki/tls"
engines:  dynamic 

I compile apache with the following options if this is of any help:

./configure --prefix=/usr/local/apache2 \
            --with-apxs2=/usr/local/apache2/bin/apxs \
            --with-mysql=/opt \
            --with-mysqli=/usr/bin/mysql_config \
            --with-gd --with-ttf=/opt \
            --with-xpm-dir=/opt \
            --with-freetype-dir=/opt \
            --enable-gd-native-ttf \
            --with-zlib-dir=/opt \
            --with-curl=/opt/include/curl \
            --enable-mbstring \
            --with-xsl=/opt \
            --with-libexpat-dir=/opt \
            --with-jpeg-dir=/opt \
            --with-png-dir=/opt \
            --enable-soap \
            --with-openssl \
            --with-ssl=/usr/local/openssl \
            --with-included-apr \
            --with-pcre=/usr/local/pcre/ \
            --with-mpm=worker \
            --enable-rewrite \
            --enable-ssl; 

I have compiled a version of OpenSSL 1.0.1g to /usr/local/openssl using:

export CFLAGS="-fPIC";
openssl-1.0.1g]# ./config shared no-ssl2 no-ssl3 --openssldir=/usr/local/openssl

I have tried updating OpenSSL, Recompiling Apache, Recompiling PHP, Reinstalling Percona, rebooting, compiling OpenSSL (admittedly incorrectly and with no luck), and am still not able to patch this vulnerability.

I have rebooted the server already to ensure no libraries are still loaded in Ram.

Here is output regarding the libraries:

[root@centos php-5.5.9]# grep 'libssl.*(deleted)' /proc/*/maps
/proc/4497/maps:7faf80018000-7faf80079000 r-xp 00000000 fd:00 1591034                    /usr/lib64/libssl.so.6 (deleted)
/proc/4497/maps:7faf80079000-7faf80278000 ---p 00061000 fd:00 1591034                    /usr/lib64/libssl.so.6 (deleted)
/proc/4497/maps:7faf80278000-7faf8027c000 r--p 00060000 fd:00 1591034                    /usr/lib64/libssl.so.6 (deleted)
/proc/4497/maps:7faf8027c000-7faf80283000 rw-p 00064000 fd:00 1591034                    /usr/lib64/libssl.so.6 (deleted)
/proc/4506/maps:7f21d735d000-7f21d73be000 r-xp 00000000 fd:00 1591034                    /usr/lib64/libssl.so.6 (deleted)
/proc/4506/maps:7f21d73be000-7f21d75bd000 ---p 00061000 fd:00 1591034                    /usr/lib64/libssl.so.6 (deleted)
/proc/4506/maps:7f21d75bd000-7f21d75c1000 r--p 00060000 fd:00 1591034                    /usr/lib64/libssl.so.6 (deleted)
/proc/4506/maps:7f21d75c1000-7f21d75c8000 rw-p 00064000 fd:00 1591034                    /usr/lib64/libssl.so.6 (deleted)
/proc/4594/maps:311b000000-311b061000 r-xp 00000000 fd:00 1591034                        /usr/lib64/libssl.so.6 (deleted)
/proc/4594/maps:311b061000-311b260000 ---p 00061000 fd:00 1591034                        /usr/lib64/libssl.so.6 (deleted)
/proc/4594/maps:311b260000-311b264000 r--p 00060000 fd:00 1591034                        /usr/lib64/libssl.so.6 (deleted)
/proc/4594/maps:311b264000-311b26b000 rw-p 00064000 fd:00 1591034                        /usr/lib64/libssl.so.6 (deleted)

To see what version I am using:

[root@centos php-5.5.9]# strings /usr/lib64/libssl.so.6|grep -i openssl
OpenSSLDie
OPENSSL_cleanse
OPENSSL_DIR_read
OPENSSL_DIR_end
OPENSSL_init_library
OPENSSL_1.0.1
OPENSSL_1.0.1_EC
SSLv2 part of OpenSSL 1.0.1e-fips 11 Feb 2013
SSLv3 part of OpenSSL 1.0.1e-fips 11 Feb 2013
TLSv1 part of OpenSSL 1.0.1e-fips 11 Feb 2013
DTLSv1 part of OpenSSL 1.0.1e-fips 11 Feb 2013
OpenSSL 1.0.1e-fips 11 Feb 2013
OPENSSL_DIR_read(&ctx, '
OPENSSL_DEFAULT_ZLIB
OPENSSL_malloc Error

I am unaware if OpenSSL 1.0.1e-fips 11 Feb 2013 relates to the above output where openssl version -a reports being OpenSSL 1.0.1e-fips 11 Feb 2013 but patch on the 8th, for heartbleed, or if its vulnerable.

On the same server, I am running Tomcat and GlassFish, but even when these are off, the server flags as vulnerable. Any ideas? Thank you in advance for any advice.

Answer 1

You likely still have a process that is using the old library.

You can test it like so:

grep 'libssl.*(deleted)' /proc/*/maps

You could alternatively bounce the entire system.

https://unix.stackexchange.com/questions/123711/how-do-i-recover-from-the-heartbleed-bug-in-openssl

Answer 2

You seem to compile apache with local version of openssl (--with-ssl=/usr/local/openssl), that might differ from the one you have upgraded through yum.