2

I have a RAID 5 array off which a database reads and writes small amounts of data.

In normal circumstances, the database operations including commiting runs in acceptable time. However, when a large amount of data is loaded into the RAID array (but not into the database), the commit (but not the other operations) become unacceptably slow.

My current assumption is that commit takes too long because the parity information needs to be calculated and/or written to disk.

Is there a way if I can verify my assumption, i. e. is there a tool that displays the amount of CPU that was needed to calculate the parity information and if this is the bottleneck.

Additionally, I am considering moving the array to RAID 10 or RAID 1. But before I'd do that, is there a somewhat scientific approach that lets me estimate if it is worth the hassle?

I am running Debian 12 with a software RAID (/dev/md/).

1
  • The CPU load is the least of your problems. The IO capacity of the spinning rust is and R5 write amplification worsens the problem. Better use SSDs and RAID 1 or 10.
    – Zac67
    Commented Jun 6 at 9:16

1 Answer 1

4

Open several terminals and keep an eye on top or htop. Look specifically for spikes in CPU usage during database commits.

top

or

htop

Use iostat to see how much time the CPU spends in I/O operations:

iostat -x 1

The sar command can be used to collect, report, or save system activity information, which includes CPU usage.

sar -u 1 5

Gives you an overview of CPU usage, including the system and I/O wait times

Depending on the RAID controller or software RAID you are using, there might be specific tools or logs that can give insights into the RAID operations. For mdadm check /proc/mdstat for RAID status. Use iostat -xmd 1 for detailed RAID I/O statistics.

Also, profiling the database commit process. Use strace to trace system calls and signals. This can help identify where the process spends most of its time.

strace -c -p <database_pid>

That will summarize system call usage and might highlight the time spent in I/O operations.

Use perf to profile the system and identify bottlenecks.

perf top

or

perf record -a -g

followed by

perf report

I'd guess you've already done all that.

RAID 10 is likely to significantly improve performance, but I would have it properly documented with empirical results and have it reviewed, before issuing some RFC. The following will take a little effort, but it's definitely worth it.

Use benchmarking tools to measure performance before and after changes. Tools like fio (Flexible I/O Tester) can be used to simulate database loads and measure performance.

IMPORTANT:
Create a separate, isolated test environment to set up and benchmark a RAID 10 array without affecting your current production system.

Execute the fio benchmark on your existing RAID 5 array:

fio --name=raid5-test --rw=randwrite --bs=4k --size=1G --numjobs=4 --time_based --runtime=60 --group_reporting --filename=/path/to/raid5/storage

Create Test RAID 10 Array:

mdadm --create --verbose /dev/md-test --level=10 --raid-devices=4 /dev/sdx /dev/sdy /dev/sdz /dev/sdw

Use spare or virtual disks for this test setup (/dev/sdx, /dev/sdy, etc.).

Format and Mount Test RAID 10 Array:

mkfs.ext4 /dev/md-test
mkdir /mnt/raid10-test
mount /dev/md-test /mnt/raid10-test

Run fio Benchmark on RAID 10:

fio --name=raid10-test --rw=randwrite --bs=4k --size=1G --numjobs=4 --time_based --runtime=60 --group_reporting --filename=/mnt/raid10-test/testfile

Compare the fio results from the RAID 5 and RAID 10 benchmarks to assess performance improvements.

Evaluate the Results

CPU Usage and I/O Wait:
Analyze the iostat and strace outputs to identify if CPU usage for parity calculation is a significant bottleneck.

Benchmark Results:
Compare the throughput, latency, and IOPS from the fio tests on RAID 5 and RAID 10.

Metrics to compare:

IOPS: Higher is better. Compare the IOPS for write operations.
Bandwidth (BW): Higher is better. Compare the throughput in MiB/s.
Latency: Lower is better. Compare the average, min, and max latencies.
Standard Deviation (stdev): Lower is better. Indicates consistency in performance.

Then decide on Moving to RAID 10 Based on your benchmarks and CPU usage analysis, you can make an informed decision on whether switching to RAID 10 will resolve your performance issues.

By creating a test RAID 10 array in an isolated environment, you can benchmark its performance and compare it to your current RAID 5 setup without disrupting your production system. This way allows you to scientifically evaluate if moving to RAID 10 is worth the hassle and if it will address your performance concerns effectively.

Good luck!

3
  • In the current form the question does not confirm that the database server is running Linux nor that the RAID solution is software based rather than a hardware RAID controller … And much of your approach would only work in a Linux software RAID scenario
    – HBruijn
    Commented Jun 4 at 21:13
  • Debian 12 is Linux. I did not expect the user to be using anything else in 2024, I did put a lot of dedication to provide a good quality answer. - But you're right about mdadm is a Linux command.
    – Max Haase
    Commented Jun 5 at 7:13
  • 🤦🏻‍♂️ missed that detail
    – HBruijn
    Commented Jun 5 at 20:24

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .