0

Since a few days, I see this kind of messages in my syslog:

Sep 23 14:28:42 server kernel: [138926.637593] php5-fpm[6455]: segfault at 7f9ade735018 ip 00007f9ae4026772 sp 00007ffd69b4fad0 error 4 in libmysqlclient.so.18.0.0[7f9ae3ff9000+2f1000]
Sep 23 14:28:44 server kernel: [138928.314016] php5-fpm[22742]: segfault at 7f9ade3db018 ip 00007f9ae4026772 sp 00007ffd69b4fad0 error 4 in libmysqlclient.so.18.0.0[7f9ae3ff9000+2f1000]
Sep 23 14:32:11 server kernel: [139135.318287] php5-fpm[16887]: segfault at 7f9ade4b3018 ip 00007f9ae4026772 sp 00007ffd69b4fad0 error 4 in libmysqlclient.so.18.0.0[7f9ae3ff9000+2f1000]
Sep 23 14:32:49 server kernel: [139173.050377] php5-fpm[668]: segfault at 7f9ade61a018 ip 00007f9ae4026772 sp 00007ffd69b4fad0 error 4 in libmysqlclient.so.18.0.0[7f9ae3ff9000+2f1000]
Sep 23 14:33:19 server kernel: [139203.396935] php5-fpm[26277]: segfault at 7f9ade6c0018 ip 00007f9ae4026772 sp 00007ffd69b4fad0 error 4 in libmysqlclient.so.18.0.0[7f9ae3ff9000+2f1000]
Sep 23 14:35:06 server kernel: [139310.048740] php5-fpm[27017]: segfault at 7f9ade46c018 ip 00007f9ae4026772 sp 00007ffd69b4fad0 error 4 in libmysqlclient.so.18.0.0[7f9ae3ff9000+2f1000]
Sep 23 14:35:19 server kernel: [139323.494188] php5-fpm[31263]: segfault at 7f9ade5e2018 ip 00007f9ae4026772 sp 00007ffd69b4fad0 error 4 in libmysqlclient.so.18.0.0[7f9ae3ff9000+2f1000]
Sep 23 14:36:10 server kernel: [139374.904308] php5-fpm[26422]: segfault at 7f9ade6cf018 ip 00007f9ae4026772 sp 00007ffd69b4fad0 error 4 in libmysqlclient.so.18.0.0[7f9ae3ff9000+2f1000]
Sep 23 14:37:25 server kernel: [139449.360384] php5-fpm[20806]: segfault at 7f9ade644018 ip 00007f9ae4026772 sp 00007ffd69b4fad0 error 4 in libmysqlclient.so.18.0.0[7f9ae3ff9000+2f1000] 

I'm using debian 8 and MariaDB. In the beginning it was only once every 2 or 3 hours, but now it's several times per hours. After some research, I understand as it should be an memory problem, but I didn't find any solution to solve it.

This is what I see in mysqltuner:

-------- Storage Engine Statistics -------------------------------------------
[--] Status: +ARCHIVE +Aria +BLACKHOLE +CSV +FEDERATED +InnoDB +MRG_MyISAM 
[--] Data in InnoDB tables: 2G (Tables: 79)
[--] Data in MyISAM tables: 96M (Tables: 146)
[--] Data in PERFORMANCE_SCHEMA tables: 0B (Tables: 52)
[!!] Total fragmented tables: 34

-------- Security Recommendations  -------------------------------------------
[OK] All database users have passwords assigned

-------- Performance Metrics -------------------------------------------------
[--] Up for: 1d 16h 44m 38s (73M q [502.853 qps], 196K conn, TX: 572B, RX: 14B)
[--] Reads / Writes: 97% / 3%
[--] Total buffers: 17.3G global + 56.2M per thread (500 max threads)
[!!] Maximum possible memory usage: 44.8G (142% of installed RAM)
[OK] Slow queries: 0% (2K/73M)
[OK] Highest usage of available connections: 28% (141/500)
[OK] Key buffer size / total MyISAM indexes: 1.0G/32.6M
[OK] Key buffer hit rate: 100.0% (132M cached / 53K reads)
[OK] Query cache efficiency: 44.9% (50M cached / 113M selects)
[!!] Query cache prunes per day: 260596
[OK] Sorts requiring temporary tables: 0% (2K temp sorts / 2M sorts)
[OK] Temporary tables created on disk: 21% (6K on disk / 28K total)
[OK] Thread cache hit rate: 99% (141 created / 196K connections)
[OK] Table cache hit rate: 72% (500 open / 692 opened)
[OK] Open file limit used: 17% (429/2K)
[OK] Table locks acquired immediately: 99% (25M immediate / 25M locks)
[OK] InnoDB buffer pool / data size: 16.0G/2.4G
[!!] InnoDB log waits: 30

So the maximum memory to use is too high, but I ajusted my innodb buffer pool size to 16Go, for 32Go RAM that should be ok, I don't know what to do for optimize this.

The thing is, my memory general usage in the server is always under 89% (plus for caching). MySQL is actually using 50,6% of RAM. I don't know if there is link between all of this, but I prefere to put it here. Otherwise, everything seems to be ok in MySQL side...

Finally this the principal variables in my.cnf I adjusted which could have an effect on this:

max_connections = 100
max_heap_table_size = 64M
read_buffer_size = 4M
read_rnd_buffer_size = 32M
sort_buffer_size = 8M
query_cache_size = 256M
query_cache_limit = 4M
query_cache_type = 1
query_cache_strip_comments =1
thread_stack = 192K
transaction_isolation = READ-COMMITTED
tmp_table_size = 64M
nnodb_additional_mem_pool_size = 16M
innodb_buffer_pool_size = 16G
thread_cache_size   = 4M
max_connections     = 500
join_buffer_size    = 12M
interactive_timeout = 30
wait_timeout        = 30
open_files_limit    = 800
innodb_file_per_table
key_buffer_size     = 1G
table_open_cache    = 500
innodb_log_file_size    = 256M

Two days ago the server crashed for no reason in syslog execpt the segfault. Can segfault crash the system ? Any Idea for the segfault reason ? A few ways to understand the origin of the problem ?

  • You didn't specified the distribution. I'd suggest to enable core dumps so next time you get more info, what really happened. In case you got out of memory, there would be something about OOM killer (Out Of Memory) in logs (grep -i 'killed process'). – Jaroslav Kucera Sep 23 '17 at 16:59
  • 1
    "After some research, I understand as it should be an memory problem." There's no evidence for that. These log entries point only to a problem with the MySQL client software and php, not the server. Check your web server logs for a page throwing errors to narrow it down. – Michael - sqlbot Sep 23 '17 at 19:59
  • My distribution is Debian GNU/Linux 8 (jessie). Can you explain what is core dump and why should I enable it ? As explain in another topic I enabled core dump with this command:ulimit -c unlimited. I don't if I should do another thing ... – poka Sep 23 '17 at 20:53
  • So you thing it's not a server side problem but a client side ? That's mean the problem could be in the query in my website ? I don't see particular errors in link with that. – poka Sep 23 '17 at 21:05
  • For information, I solved this problem by resolving another problem, in htis thread: serverfault.com/questions/875192/… – poka Nov 13 '17 at 15:11
0

Ok so theses segfaults corresponds to errors in my client website. For theses segfaults:

Sep 23 21:39:07 server kernel: [164772.739464] php5-fpm[10375]: segfault at 7f9ade4be018 ip 00007f9ae4026772 sp 00007ffd69b4fad0 error 4 in libmysqlclient.so.18.0.0[7f9ae3ff9000+2f1000]
Sep 23 22:08:47 server kernel: [166554.093566] php5-fpm[12582]: segfault at 7f9ade5d9018 ip 00007f9ae4026772 sp 00007ffd69b4fad0 error 4 in libmysqlclient.so.18.0.0[7f9ae3ff9000+2f1000]
Sep 23 22:09:17 server kernel: [166584.252833] php5-fpm[31132]: segfault at 7f9ade669018 ip 00007f9ae4026772 sp 00007ffd69b4fad0 error 4 in libmysqlclient.so.18.0.0[7f9ae3ff9000+2f1000]
Sep 23 22:10:30 server kernel: [166656.875041] php5-fpm[24992]: segfault at 7f9ade6e9018 ip 00007f9ae4026772 sp 00007ffd69b4fad0 error 4 in libmysqlclient.so.18.0.0[7f9ae3ff9000+2f1000]
Sep 23 22:39:19 server kernel: [168387.935728] php5-fpm[27380]: segfault at 7f9ade654018 ip 00007f9ae4026772 sp 00007ffd69b4fad0 error 4 in libmysqlclient.so.18.0.0[7f9ae3ff9000+2f1000]
Sep 23 22:48:55 server kernel: [168963.750727] php5-fpm[6309]: segfault at 7f9ade371018 ip 00007f9ae4026772 sp 00007ffd69b4fad0 error 4 in libmysqlclient.so.18.0.0[7f9ae3ff9000+2f1000]
Sep 23 22:49:33 server kernel: [169001.961155] php5-fpm[15416]: segfault at 7f9ade40f018 ip 00007f9ae4026772 sp 00007ffd69b4fad0 error 4 in libmysqlclient.so.18.0.0[7f9ae3ff9000+2f1000]
Sep 23 23:04:22 server kernel: [169892.025852] php5-fpm[5763]: segfault at 7f9ade422018 ip 00007f9ae4026772 sp 00007ffd69b4fad0 error 4 in libmysqlclient.so.18.0.0[7f9ae3ff9000+2f1000]
Sep 23 23:04:50 server kernel: [169919.986130] php5-fpm[16016]: segfault at 7f9ade5cf018 ip 00007f9ae4026772 sp 00007ffd69b4fad0 error 4 in libmysqlclient.so.18.0.0[7f9ae3ff9000+2f1000]
Sep 23 23:09:11 server kernel: [170181.423023] php5-fpm[5129]: segfault at 7f9ade6cc018 ip 00007f9ae4026772 sp 00007ffd69b4fad0 error 4 in libmysqlclient.so.18.0.0[7f9ae3ff9000+2f1000]

I got theses errors in my nginx log error, exactly at the same time so not a coincidence:

2017/09/23 21:39:07 [error] 808#0: *7346034 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 1.1.1.1, server: example.com, request: "GET /page/a-page HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "example.com", referrer: "https://example.com/page/another-page"
2017/09/23 22:08:47 [error] 808#0: *7443193 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 1.1.1.1, server: example.com, request: "GET /engine/search?q=query", upstream: "fastcgi://127.0.0.1:9000", host: "example.com"
2017/09/23 22:09:17 [error] 808#0: *7450102 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 1.1.1.1, server: example.com, request: "GET / HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "example.com"
2017/09/23 22:10:30 [error] 808#0: *7450989 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 1.1.1.1, server: example.com, request: "GET /page/a-page HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "example.com", referrer: "https://example.com/page/another-page"
2017/09/23 22:39:19 [error] 808#0: *7553713 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 21.1.1.1, server: example.com, request: "GET /page/a-page HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "example.com", referrer: "https://example.com/engine/search?q=query"
2017/09/23 22:48:55 [error] 7800#0: *7578116 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 1.1.1.1, server: example.com, request: "GET /engine/search?q=query HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "example.com"
2017/09/23 22:49:33 [error] 7800#0: *7582962 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 1.1.1.1, server: example.com, request: "GET /page/a-page HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "example.com"
2017/09/23 23:04:22 [error] 7800#0: *7639594 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 1.1.1.1, server: example.com, request: "GET /page/a-page HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "example.com"
2017/09/23 23:04:50 [error] 7800#0: *7636486 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 1.1.1.1, server: example.com, request: "GET /page/a-page HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "example.com", referrer: "https://example.com/engine/search?q=query"
2017/09/23 23:09:11 [error] 7800#0: *7654327 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 1.1.1.1, server: example.com, request: "GET /engine/search?q=query3 HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "example.com"

So this is my nginx virtualhost for this website:

server {

   listen  80;
   listen  443;
   server_name example.com;

   if ($http_x_forwarded_proto = "http") {
      return 301 https://$server_name$request_uri;
   }

   ssl  on;
   ssl_certificate /etc/nginx/cloudflare_cert/fullchain.pem;
   ssl_certificate_key /etc/nginx/cloudflare_cert/privkey.pem;

   include /etc/nginx/conf.d/cloudflare;

   root   /var/www/website;
   index  index.php;

   location / {
      # main rewrite rule
      try_files $uri $uri/ /index.php?/$request_uri;

      proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "upgrade";
      proxy_set_header Host $http_host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forward-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forward-Proto http;
      proxy_set_header X-Nginx-Proxy true;

   }

   # php parsing
   location ~ \.php$ {
        include fastcgi_params;
        fastcgi_pass  127.0.0.1:9000;
        fastcgi_index index.php;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
   }

   error_log  /var/log/nginx/example_error.log;
   access_log  /var/log/nginx/example_access.log;

}

server {
   listen  80;
   listen  443;
   server_name www.example.com;
   return 301 https://example.com$request_uri;
}

So it's a client side problem. Now I have to understand what recv() failed (104: Connection reset by peer) while reading response header from upstream means and how to solve it.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.