2

Unsure as to whether this is a SF or SO problem.

Very (very) busy LAMP servers in load balanced environment Apache 2.2.26 on Centos 5.10.

I'm trying to trace down a code or systems issue with hanging httpd processes. These calls sit in W, "Sending Reply" status forever, the tcp connections remain in keepalive and the apache's timeout never fires. Eventually we accumulate enough hanging processes that I have to bounce the httpd process.

This is the end of an strace that seems to surround all of hung calls. I'm really unsure as to where to go next with this. Seems Apache is trying to write out to the console, and I'm not sure if that's normal. But the malloc error definitely points to something wrong. Any help (even wild ideas) appreciated.

I'm not sure there are interesting moudles, but here are the LoadModules. LoadModule rewrite_module modules/mod_rewrite.so LoadModule info_module modules/mod_info.so LoadModule auth_digest_module modules/mod_auth_digest.so LoadModule proxy_module modules/mod_proxy.so LoadModule proxy_balancer_module modules/mod_proxy_balancer.so LoadModule proxy_connect_module modules/mod_proxy_connect.so LoadModule proxy_http_module modules/mod_proxy_http.so LoadModule ldap_module modules/mod_ldap.so LoadModule authnz_ldap_module modules/mod_authnz_ldap.so LoadModule deflate_module modules/mod_deflate.so LoadModule php4_module modules/libphp4.so

lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=12288, ...}) = 0

lstat("/tmp/promo.log", {st_mode=S_IFREG|0644, st_size=14778558, ...}) = 

open("/tmp/promo.log", O_WRONLY|O_CREAT|O_APPEND, 0666) = 27

fstat(27, {st_mode=S_IFREG|0644, st_size=14778558, ...}) = 0

lseek(27, 0, SEEK_CUR)                  = 0

lseek(27, 0, SEEK_CUR)                  = 0

write(27, "20140314065931 : cartitem::calcu"..., 68) = 68

close(27)                               = 0

open("/dev/tty", O_RDWR|O_NOCTTY|O_NONBLOCK) = -1 ENXIO (No such device or address)

writev(2, [{"*** glibc detected *** ", 23}, {"/usr/local/apache-2.2.26/bin/htt"..., 34}, {": ", 2}, {"double free or corruption (!prev"..., 33}, {": 0x", 4}, {"0000000005ebfb20", 16}, {" ***\n", 5}], 7) = 117

open("/usr/local/apache-2.2.26/lib/libgcc_s.so.1", O_RDONLY) = -1 ENOENT (No such file or directory)

open("/usr/lib64/tls/libgcc_s.so.1", O_RDONLY) = -1 ENOENT (No such file or directory)

open("/usr/lib64/libgcc_s.so.1", O_RDONLY) = -1 ENOENT (No such file or directory)

open("/usr/local/apache-2.2.26/lib/libgcc_s.so.1", O_RDONLY) = -1 ENOENT (No such file or 
directory)

open("/etc/ld.so.cache", O_RDONLY)      = 27

fstat(27, {st_mode=S_IFREG|0644, st_size=93743, ...}) = 0

mmap(NULL, 93743, PROT_READ, MAP_PRIVATE, 27, 0) = 0x2ba82af8b000
close(27)                               = 0

open("/lib64/libgcc_s.so.1", O_RDONLY)  = 27

read(27, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P\36`T8\0\0\0"..., 832) = 832

fstat(27, {st_mode=S_IFREG|0755, st_size=58400, ...}) = 0

open("/dev/tty", O_RDWR|O_NOCTTY|O_NONBLOCK) = -1 ENXIO (No such device or address)

writev(2, [{"*** glibc detected *** ", 23}, {"/usr/local/apache-2.2.26/bin/htt"..., 34}, {": ", 2}, {"malloc(): memory corruption", 27}, {": 0x", 4}, {"0000000005fc1e70", 16}, {" ***\n", 5}], 7) = 111

futex(0x2ba822f1dfc0, FUTEX_WAIT_PRIVATE, 9, NULL) = -1 EINTR (Interrupted system call)
--- SIGTERM (Terminated) @ 0 (0) ---

futex(0x2ba822f1b9e0, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EINTR (Interrupted system call)
--- SIGCONT (Continued) @ 0 (0) ---

futex(0x2ba822f1b9e0, FUTEX_WAIT_PRIVATE, 2, NULL
  • Here are some ideas: Add -s99999 to your strace command so you can see the rest of the strings. And show how you've set up /tmp/promo.log in the apache config. And let us know if you have any intereseting apache modules loaded. And compare those addresses (5ebfb20 and 5fc1e70) with the contents of /proc/PID/maps to see if they belong to a module. – Wumpus Q. Wumbley Mar 15 '14 at 15:48
  • /tmp/promo.log is being written to by a php app (sort of in-line debugging), but it seems to be opening, writing and closing properly. Thanks! – madhatta Mar 15 '14 at 16:22
  • Both addresses are in [heap] just after httpd according to the memory map – madhatta Mar 15 '14 at 16:31
  • If promo.log is always being written to just before the memory corruption error, that's a possible clue that whatever comes after the write to the log is the cause. Another thing you might try is changing the glibc MALLOC_CHECK_ mode (gnu.org/software/libc/manual/html_node/…). If you set it to 3 you'll get a backtrace and core dump instead of just a warning message. – Wumpus Q. Wumbley Mar 15 '14 at 16:58
  • Yes, I'm trying to debug it but it's difficult on a production system. Thinking about adding a profiler to the code to output some logging. Thanks so much! – madhatta Mar 16 '14 at 19:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.