11

We have 8 Cisco servers with 12 spinning disks for data and 2 SSDs for OS. The 2 SSDs are in linux software raid 1. The SSDs all have their wear indicator in single digits and some of those that have reached a value of 1 have failed. I'm in the process of swapping them all out from spares (a long and tiresome process) but I've noticed the wear indicator is dropping 1 or 2% per week (I didn't take exact measurements). There is a single application running on these servers and the vendor has given me some vague ideas, but I really need to find the directories that it is writing to. That way I can really highlight the problem and push the vendor for a fix. I've searched a bit but haven't been able to find too much. iotop for example show full disk throughput including the 12 spinning disks. OS is Redhat 7.9

In answer to some of the questions:

  • disks are "480GB 2.5 inch Enterprise Value 6Gb SATA SSD"
  • product ID is "UCS-SD480GBKS4-EB"
  • disks were supplied standard with the servers in 2018
  • The wearing out appears to have accelerated recently (I am now logging the wear so will have a better answer on that in a few days)
  • I have replaced most disks with identical disks purchased maybe a couple of years later.
  • iotop is showing a constant 8MB/s write.
  • the system is running hadoop across 8 servers. The hadoop file system is on spinning disks so shouldn't touch the SSDs
  • I have reduced the disk IO considerably on suggestion of the vendor although it does still seem high (8MB/s)
10
  • 2
    Mostly duplicate of serverfault.com/q/169676/28549
    – Ben Voigt
    Commented May 10 at 21:07
  • 2
    iostat -t nvme0n1p6 --human 1 will let you keep an eye on how much data is being read/written to the SSD drive from one second to the next. (Replace nvme0n1p6 with the name of your SSD partition of course) Commented May 12 at 3:56
  • Are you sure the kernel knows these are SSD's and use TRIM and all that? Commented May 12 at 5:03
  • @MikeKulls okay cool -that means they're covered by smartnet (if you had it and have kept paying) AND that its not a $50 domestic grade SSD.
    – Criggie
    Commented May 13 at 7:03
  • 1
    @Criggie we did get warranty on 1 SSD but we went to RMA a second one and the servers had gone EOL. I've changed them all out from spares which was a monumental job, 1 hour per disk, can only do 1 at a time, working in a freezing cold exchange :-)
    – MikeKulls
    Commented May 16 at 3:34

4 Answers 4

5

You can use ProcMon for Linux to trace file system calls.

https://github.com/Sysinternals/ProcMon-for-Linux

1
  • This looks like what I'm after however it doesn't seem to target RHEL. I ran into some issues compiling it and have run out of time to resolve them.
    – MikeKulls
    Commented May 13 at 6:50
13

It’s hard to be certain without more details on the age of the systems, the exact model and age of the SSDs, and a handful of other factors.

Assuming good quality SSDs, 1-2% on the wear indicator in a week means you’re writing a couple of terabytes (minimum) of data to them in a week. That’s a huge amount of data for an OS volume. Top culprits I would look at are, in order:

  • Cheap SSDs. Put simply, it sounds a lot like you don’t have particularly good quality SSDs in this system, which would invalidate the assumption that 1-2% usable life expectancy translates to multiple TB of data. I suggest doing some research on the exact model of SSDs you’re using to confirm what their actual rated lifetime write endurance is and that there are no documented firmware issues. Good ones from the past five years or so should be rated for at least 100 times their listed capacity (so at least 100 TB on a 1 TB SSD), but ideally more than that (as a point of comparison, current high-end consumer 1 TB SSDs are typically rated for about 300 TB of writes these days).
  • Block device caching. If you have bcache, dm-cache, ZFS L2ARC, or some other block device caching setup that is using space on the SSD’s, that’s probably the culprit, try turning it off and see what happens (well, other than probably a nasty hit to performance).
  • Logging. Most of your logs are probably on your OS volume. If you’ve got verbose logging turned on, and your application is very busy, this could easily run into the the terabyte range in a week. But it could also be something else, like logs from SELinux, or process accounting, or the auditing daemon.
  • Non-block caching. Essentially, stuff under /var/cache or other locations where caches might be stored (such as ~/.cache in user home directories). This shouldn’t be hitting the required numbers unless it’s a very active terminal server, but it’s worth checking.
  • Swapping. Probably not a major contributor, because hitting the numbers required would translate to swapping frequently enough to cause other performance issues on the system.
3
  • 3
    also atime updates coult be a thing.
    – Jasen
    Commented May 11 at 7:25
  • 4
    Logging with disk flushes after every log line is often a culprit
    – abligh
    Commented May 11 at 7:32
  • Not on enterprise drives or aynthing with supercats that can flush to dram and has the supercaps to handle the write to the cell in case of a failure - this is standard in any non low end SSD for years now.
    – TomTom
    Commented May 11 at 11:34
11

Check swapping - that is a typical indicator. Check whether you run any temp files for whatever software - that may be another one. Both need you to check and given that temp files are software dependent - no real help possible. Build server directories were where i observed that last time - technically a temp structure as every run downloads the repository (ok, updates it), then initializes the source tree and builds - that is a LOT of writes. End user SSD are not made for this. Really depends on the software - no generic answer possible.

Otherwise consider whether using low end SSD is suitable to start with - this sounds like more drop than should be possible

2
  • 3
    FWIW I've been running a build server for five years and it only made a 4% dent in the wear indicator, despite the SSDs being the bottleneck for most of the time. Commented May 11 at 9:09
  • Depends on SSD and whether you do a lot and clean or not builds. The ones I did had to rebuild the source tree and were basically busy nonstop. We used up Samsung SSD's in like 6 months. Now wend with enterprise level drives that handle 5 writes per day and they do not bulk at all over years.
    – TomTom
    Commented May 11 at 11:33
2

You can approach this problem top-down.

That means first set up a monitoring such as netdata that continuously writes all the relevant IO metrics into a database for all servers.

Using that data you can check for swap activity and what amount of writes volume your SSDs are seeing and how it changes over time.

That way you can cross-check whether the change of the wear-indicator is actually plausible. I mean bugs in SSDs firmware that influence SMART reporting aren't unheard of.


For identifying directories and files that are written to at a high rate you can run filetop from the bcc-tools package, e.g.:

# /usr/share/bcc/tools/filetop
23:56:12 loadavg: 1.32 0.83 0.60 4/1273 563644

TID     COMM             READS  WRITES R_Kb    W_Kb    T FILE
563614  yes              0      36757  0       294056  R foo.bar
[..]
3
  • We used a Dockerized netdata on a server and it wrote about 1 TB of data in 1 or 2 months, so beware than netdata can accelerate the wearing. Commented May 13 at 15:17
  • 1
    @A.L-saynotoAI ideally, netdata on the server (or servers) under investigation is configured to just collect all the relevant metrics and send them over the network to a separate host that runs a netdata master instance that writes everything to a timeseries database there and serves the web-UI. Alternatively, you can implement such monitoring using telegraf/grafana, prometheus/grafana or similar. The nice thing about netdata is that its main collector is quite efficient and it provides a well curated set of useful panels in its UI, out of the box. Commented May 13 at 21:58
  • @A.L-saynotoAI FWIW, as of 2024, 1 TB per month likely only consumes a relatively small amount of the SSD's endurance rating budget. For example, the Micron 7500 datacenter SSDs are rated for 1 or 3 drive-writes-per-day (DWPD), under a 5 year warranty. Thus, using a - say - 3.84 TB/1 DWPD SSD the 1 TB/month write load consumes 1/115th or so of the monthly endurance budget. Commented May 13 at 22:32

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .