Unsure as to whether this is a SF or SO problem.
Very (very) busy LAMP servers in load balanced environment Apache 2.2.26 on Centos 5.10.
I'm trying to trace down a code or systems issue with hanging httpd processes. These calls sit in W, "Sending Reply" status forever, the tcp connections remain in keepalive and the apache's timeout never fires. Eventually we accumulate enough hanging processes that I have to bounce the httpd process.
This is the end of an strace that seems to surround all of hung calls. I'm really unsure as to where to go next with this. Seems Apache is trying to write out to the console, and I'm not sure if that's normal. But the malloc error definitely points to something wrong. Any help (even wild ideas) appreciated.
I'm not sure there are interesting moudles, but here are the LoadModules. LoadModule rewrite_module modules/mod_rewrite.so LoadModule info_module modules/mod_info.so LoadModule auth_digest_module modules/mod_auth_digest.so LoadModule proxy_module modules/mod_proxy.so LoadModule proxy_balancer_module modules/mod_proxy_balancer.so LoadModule proxy_connect_module modules/mod_proxy_connect.so LoadModule proxy_http_module modules/mod_proxy_http.so LoadModule ldap_module modules/mod_ldap.so LoadModule authnz_ldap_module modules/mod_authnz_ldap.so LoadModule deflate_module modules/mod_deflate.so LoadModule php4_module modules/libphp4.so
lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=12288, ...}) = 0 lstat("/tmp/promo.log", {st_mode=S_IFREG|0644, st_size=14778558, ...}) = open("/tmp/promo.log", O_WRONLY|O_CREAT|O_APPEND, 0666) = 27 fstat(27, {st_mode=S_IFREG|0644, st_size=14778558, ...}) = 0 lseek(27, 0, SEEK_CUR) = 0 lseek(27, 0, SEEK_CUR) = 0 write(27, "20140314065931 : cartitem::calcu"..., 68) = 68 close(27) = 0 open("/dev/tty", O_RDWR|O_NOCTTY|O_NONBLOCK) = -1 ENXIO (No such device or address) writev(2, [{"*** glibc detected *** ", 23}, {"/usr/local/apache-2.2.26/bin/htt"..., 34}, {": ", 2}, {"double free or corruption (!prev"..., 33}, {": 0x", 4}, {"0000000005ebfb20", 16}, {" ***\n", 5}], 7) = 117 open("/usr/local/apache-2.2.26/lib/libgcc_s.so.1", O_RDONLY) = -1 ENOENT (No such file or directory) open("/usr/lib64/tls/libgcc_s.so.1", O_RDONLY) = -1 ENOENT (No such file or directory) open("/usr/lib64/libgcc_s.so.1", O_RDONLY) = -1 ENOENT (No such file or directory) open("/usr/local/apache-2.2.26/lib/libgcc_s.so.1", O_RDONLY) = -1 ENOENT (No such file or directory) open("/etc/ld.so.cache", O_RDONLY) = 27 fstat(27, {st_mode=S_IFREG|0644, st_size=93743, ...}) = 0 mmap(NULL, 93743, PROT_READ, MAP_PRIVATE, 27, 0) = 0x2ba82af8b000 close(27) = 0 open("/lib64/libgcc_s.so.1", O_RDONLY) = 27 read(27, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P\36`T8\0\0\0"..., 832) = 832 fstat(27, {st_mode=S_IFREG|0755, st_size=58400, ...}) = 0 open("/dev/tty", O_RDWR|O_NOCTTY|O_NONBLOCK) = -1 ENXIO (No such device or address) writev(2, [{"*** glibc detected *** ", 23}, {"/usr/local/apache-2.2.26/bin/htt"..., 34}, {": ", 2}, {"malloc(): memory corruption", 27}, {": 0x", 4}, {"0000000005fc1e70", 16}, {" ***\n", 5}], 7) = 111 futex(0x2ba822f1dfc0, FUTEX_WAIT_PRIVATE, 9, NULL) = -1 EINTR (Interrupted system call) --- SIGTERM (Terminated) @ 0 (0) --- futex(0x2ba822f1b9e0, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EINTR (Interrupted system call) --- SIGCONT (Continued) @ 0 (0) --- futex(0x2ba822f1b9e0, FUTEX_WAIT_PRIVATE, 2, NULL
-s99999
to yourstrace
command so you can see the rest of the strings. And show how you've set up/tmp/promo.log
in the apache config. And let us know if you have any intereseting apache modules loaded. And compare those addresses (5ebfb20 and 5fc1e70) with the contents of/proc/PID/maps
to see if they belong to a module.promo.log
is always being written to just before the memory corruption error, that's a possible clue that whatever comes after the write to the log is the cause. Another thing you might try is changing the glibcMALLOC_CHECK_
mode (gnu.org/software/libc/manual/html_node/…). If you set it to 3 you'll get a backtrace and core dump instead of just a warning message.