Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a dedicated mysql database server which has been having some performance problems recently, under normal load the server will be running fine, then suddenly out of the blue the performance will fall off a cliff. The server isn't using the swap file and there is 12GB of RAM in the server, more than enough for its needs.

After contacting my hosting comapnies support they have discovered that there is a failed 2GB DIMM in the server and have scheduled to replace it tomorow morning.

My question is could a failed DIMM result in the performance problems I am seeing or is this just coincidence?

My worry is that they will replace the ram tomorrow but the problems will persist and I will still be lost of explanations so I am just trying to think ahead.

The reason I ask is that there is plenty of RAM in the server, more than required and simply missing 2GB should be a problem, so if this failed DIMM is causing these performance problems then the OS must be trying to access the failed DIMM and slowing down as a result. Does that sound like a credible explanation?

This is what DELLs omreport program says about the RAM, notice one dimm is "Critical"

Memory Information

Health : Critical

Memory Operating Mode

Fail Over State                     : Inactive
Memory Operating Mode Configuration : Optimizer

Attributes of Memory Array(s)
Attributes     : Location
Memory Array 1 : System Board or Motherboard

Attributes     : Use
Memory Array 1 : System Memory

Attributes     : Installed Capacity
Memory Array 1 : 12288  MB

Attributes     : Maximum Capacity
Memory Array 1 : 196608  MB

Attributes     : Slots Available
Memory Array 1 : 18

Attributes     : Slots Used
Memory Array 1 : 6

Attributes     : ECC Type
Memory Array 1 : Multibit ECC

Total of Memory Array(s)
Attributes : Total Installed Capacity
Value      : 12288  MB

Attributes : Total Installed Capacity Available to the OS
Value      : 12004  MB

Attributes : Total Maximum Capacity
Value      : 196608  MB

Details of Memory Array 1
Index          : 0
Status         : Ok
Connector Name : DIMM_A1 
Type           : DDR3-Registered
Size           : 2048  MB

Index          : 1
Status         : Ok
Connector Name : DIMM_A2 
Type           : DDR3-Registered
Size           : 2048  MB

Index          : 2
Status         : Ok
Connector Name : DIMM_A3 
Type           : DDR3-Registered
Size           : 2048  MB

Index          : 3
Status         : Critical
Connector Name : DIMM_B1 
Type           : DDR3-Registered
Size           : 2048  MB

Index          : 4
Status         : Ok
Connector Name : DIMM_B2 
Type           : DDR3-Registered
Size           : 2048  MB

Index          : 5
Status         : Ok
Connector Name : DIMM_B3 
Type           : DDR3-Registered
Size           : 2048  MB

the command free -m shows this, the server seems to be using more than 10GB of ram which would suggest it is trying to use the DIMM

             total       used       free     shared    buffers     cached
Mem:         12004      10766       1238          0        384       4809
-/+ buffers/cache:       5572       6432
Swap:         2047          0       2047

iostat output while problem is occuring

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          52.82    0.00   11.01    0.00    0.00   36.17

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              47.00         0.00       576.00          0        576
sda1              0.00         0.00         0.00          0          0
sda2              1.00         0.00        32.00          0         32
sda3              0.00         0.00         0.00          0          0
sda4              0.00         0.00         0.00          0          0
sda5             46.00         0.00       544.00          0        544

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          53.12    0.00    7.81    0.00    0.00   39.06

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              49.00         0.00       592.00          0        592
sda1              0.00         0.00         0.00          0          0
sda2              0.00         0.00         0.00          0          0
sda3              0.00         0.00         0.00          0          0
sda4              0.00         0.00         0.00          0          0
sda5             49.00         0.00       592.00          0        592

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          56.09    0.00    7.43    0.37    0.00   36.10

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             232.00         0.00     64520.00          0      64520
sda1              0.00         0.00         0.00          0          0
sda2            159.00         0.00     63728.00          0      63728
sda3              0.00         0.00         0.00          0          0
sda4              0.00         0.00         0.00          0          0
sda5             73.00         0.00       792.00          0        792

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          52.18    0.00    9.24    0.06    0.00   38.51

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              49.00         0.00       600.00          0        600
sda1              0.00         0.00         0.00          0          0
sda2              0.00         0.00         0.00          0          0
sda3              0.00         0.00         0.00          0          0
sda4              0.00         0.00         0.00          0          0
sda5             49.00         0.00       600.00          0        600

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          54.82    0.00    8.64    0.00    0.00   36.55

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             100.00         0.00      2168.00          0       2168
sda1              0.00         0.00         0.00          0          0
sda2              0.00         0.00         0.00          0          0
sda3              0.00         0.00         0.00          0          0
sda4              0.00         0.00         0.00          0          0
sda5            100.00         0.00      2168.00          0       2168

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          54.78    0.00    6.75    0.00    0.00   38.48

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              84.00         0.00       896.00          0        896
sda1              0.00         0.00         0.00          0          0
sda2              0.00         0.00         0.00          0          0
sda3              0.00         0.00         0.00          0          0
sda4              0.00         0.00         0.00          0          0
sda5             84.00         0.00       896.00          0        896

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          54.34    0.00    7.31    0.00    0.00   38.35

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              81.00         0.00       840.00          0        840
sda1              0.00         0.00         0.00          0          0
sda2              0.00         0.00         0.00          0          0
sda3              0.00         0.00         0.00          0          0
sda4              0.00         0.00         0.00          0          0
sda5             81.00         0.00       840.00          0        840

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          55.18    0.00    5.81    0.44    0.00   38.58

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             317.00         0.00    105632.00          0     105632
sda1              0.00         0.00         0.00          0          0
sda2            224.00         0.00    104672.00          0     104672
sda3              0.00         0.00         0.00          0          0
sda4              0.00         0.00         0.00          0          0
sda5             93.00         0.00       960.00          0        960

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          55.38    0.00    7.63    0.00    0.00   36.98

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              74.00         0.00       800.00          0        800
sda1              0.00         0.00         0.00          0          0
sda2              0.00         0.00         0.00          0          0
sda3              0.00         0.00         0.00          0          0
sda4              0.00         0.00         0.00          0          0
sda5             74.00         0.00       800.00          0        800

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          56.43    0.00    7.80    0.00    0.00   35.77

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              72.00         0.00       784.00          0        784
sda1              0.00         0.00         0.00          0          0
sda2              0.00         0.00         0.00          0          0
sda3              0.00         0.00         0.00          0          0
sda4              0.00         0.00         0.00          0          0
sda5             72.00         0.00       784.00          0        784

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          54.87    0.00    6.49    0.00    0.00   38.64

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              80.20         0.00       855.45          0        864
sda1              0.00         0.00         0.00          0          0
sda2              0.00         0.00         0.00          0          0
sda3              0.00         0.00         0.00          0          0
sda4              0.00         0.00         0.00          0          0
sda5             80.20         0.00       855.45          0        864

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          57.22    0.00    5.69    0.00    0.00   37.09

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              33.00         0.00       432.00          0        432
sda1              0.00         0.00         0.00          0          0
sda2              0.00         0.00         0.00          0          0
sda3              0.00         0.00         0.00          0          0
sda4              0.00         0.00         0.00          0          0
sda5             33.00         0.00       432.00          0        432

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          56.03    0.00    7.93    0.00    0.00   36.04

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              41.00         0.00       560.00          0        560
sda1              0.00         0.00         0.00          0          0
sda2              2.00         0.00        88.00          0         88
sda3              0.00         0.00         0.00          0          0
sda4              0.00         0.00         0.00          0          0
sda5             39.00         0.00       472.00          0        472

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          55.78    0.00    5.13    0.00    0.00   39.09

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              29.00         0.00       392.00          0        392
sda1              0.00         0.00         0.00          0          0
sda2              0.00         0.00         0.00          0          0
sda3              0.00         0.00         0.00          0          0
sda4              0.00         0.00         0.00          0          0
sda5             29.00         0.00       392.00          0        392

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          53.68    0.00    8.30    0.06    0.00   37.95

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              78.00         0.00      4280.00          0       4280
sda1              0.00         0.00         0.00          0          0
sda2              0.00         0.00         0.00          0          0
sda3              0.00         0.00         0.00          0          0
sda4              0.00         0.00         0.00          0          0
sda5             78.00         0.00      4280.00          0       4280
share|improve this question
    
Do you have any data about what is going on when the performance is poor? Does the CPU go to 100%, do you see a lot of I/O? –  Zoredache Mar 14 '11 at 21:15
    
CPU usage and IO seem to remain fairly stable throughout but queries that normally run in .3 of a second run in 5 seconds or more. Its almost like the tables are blocking them rather than the server being too busy to process them. Most db tables are innodb by the way so its not like the tables are being locked. –  murdoch Mar 14 '11 at 21:47
    
Actually i'm not even sure I have attached some output from iostat, I have to be honest i'm not sure if its good or bad, its not my area. –  murdoch Mar 14 '11 at 21:52
    
For the record, the problem is now fixed and turned out to be as a result of a bottleneck in innodb thread concurrency. We upgraded to mysql 5.1 and installed the innodb plugin and the issue is now resolved. –  murdoch Mar 16 '11 at 12:51

2 Answers 2

up vote 0 down vote accepted

When dell identifies memory as critical, all that means is that there are more ECC errors than normal. You can probably see them in /var/log/mcelog. I think this is a red herring.

under normal load the server will be running fine, then suddenly out of the blue the performance will fall off a cliff

I think you need to look at what queries are running when the slowdown occurs. For real-time monitoring of them, try innotop.

share|improve this answer
    
It looks like you are right that ram replacement has had no effect, i have been monitoring innotop like you suggested and seen most queries in the 'Sending data' state when the issue occurs. After a lot of reading there are suggestions that it could be innodb concurrency issue so we are going to see if we can upgrade mysql tonight as we are on relatively old version 5.0.77 and there has been lots of concurrency improvements in more recent versions. –  murdoch Mar 15 '11 at 10:30

Usually if you have bad RAM, either nothing happens (because the system ignores the bad ram), or the system goes along fine for a while, then core dumps.

It's possible that what you're seeing is a fail over after a core dump...that would explain a temporary slowdown.

If it's not crashing, the loss of 2gb of RAM would either cause a persistent problem, or no problem. Do you have any intermittent database jobs that might be slowing it down?

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.