2

Currently troubleshooting a MySQL server spending a lot of time in futex syscalls. All databases are InnoDB and seeing high load on my 16 core machine, and slightly more than half of CPU time is in system.

Running strace on a process shows it spending a lot of time in futex

root@db1:~# strace -c -p 5026
Process 5026 attached
^CProcess 5026 detached
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 99.95    9.583778        3651      2625       897 futex
  0.02    0.001987          86        23           poll
  0.01    0.001349         337         4           pwrite
  0.01    0.000702         351         2           access
  0.01    0.000561           5       121           sendto
  0.00    0.000410           3       125        23 recvfrom
  0.00    0.000082          41         2           setsockopt
  0.00    0.000071          36         2           getpeername
  0.00    0.000000           0        74           read
  0.00    0.000000           0        18           write
  0.00    0.000000           0         4           open
  0.00    0.000000           0         6           close
  0.00    0.000000           0        14         1 lstat
  0.00    0.000000           0        45           lseek
  0.00    0.000000           0         7           pread
  0.00    0.000000           0         5           sched_yield
  0.00    0.000000           0         2           shutdown
  0.00    0.000000           0         2           unlink
  0.00    0.000000           0         3         3 readlink
------ ----------- ----------- --------- --------- ----------------
100.00    9.588940                  3084       924 total

The part of my config relevant to InnoDB is as follows, innodb_thread_concurrency is set to 16 which matches the CPU count so I'm surprised it's waiting for futex as much as it is:

innodb_buffer_pool_size = 26000M
innodb_log_file_size = 1250M
innodb_flush_log_at_trx_commit = 2
innodb_flush_method = O_DIRECT
innodb_flush_neighbors = 0
innodb_log_files_in_group = 3
innodb_log_buffer_size = 8M
innodb_thread_concurrency = 16
innodb_file_io_threads = 4
innodb_lock_wait_timeout = 180
innodb_file_per_table
innodb_max_dirty_pages_pct = 90

What other sort of thing could cause long waits for futex? Is it possible there are issues with our host which could be causing this?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.