2

I am looking to begin a tape backup regimen and am looking to keep data flowing to the tape drive in a sufficient manner (120+MBs target sustained) but cannot figure out how to do so without a dedicated source drive/array that idles when not writing tapes. The documentation for our specific drive mentions no minimum throughput required.

Enviroment

  • Linux Debian writing to tape using mt & tar backing up RAR archives with recovery record, each ~1GB-300GB in size
  • LTO-4 Tapes on Quantum TC-42BN tape drive via SAS over external SFF Cable
  • Server is used for file backups only, no network services or fileserving.
  • MD RAID arrays with data intermittently being read/written in spurts throughout the day/night.

If the source array has significant reads/writes (from scheduled backups) during a tape write, throughput to the tape would drop dramatically even if temporarily. So some questions centered around source array/tape write throughput:

  1. I am assuming a sustained drop in throughput to below 10-20MB/s (or less) on the source during a tape write would be a problem?
  2. Do I need to have a source guaranteed to have no backups scheduled to it? Essentially 2 arrays minimum; one for backups and one for archives and tape writing?
  3. Is there a QOS for drives/arrays that could prioritize the tape writing over all else?
  4. LTO-4 tape drives throttle, so is there a common lower throughput limit to maintain for LTO-4 or does it vary widely per drive? Again, documentation mentions max designed speed and "variable speed transfers", but no mention of how variable.
  5. Am I missing something in this source-throughput equation, or have unfounded worries?

Update:

I decided to tax things minimally with a single I/O stream via a 600GB archive job reading from the array at about 30MB/s sustained while a tar was being written to the tape from a 4 drive RAID 6 with consumer SATA. The tape definitely slowed to a crawl via listening to the drive but did NOT seem to run out of data or shoe shine. This tells me to NOT expect things to keep up during a full scheduled backup for our hardware configuration but it can handle a less taxing I/O job wile writing to tape.

As of note, the LOT4 tapes must do 56 end-to-end passes so effectively it writes in ~14GB chunks before it stops for some seconds to slow down and then "go" the other direction. I think this helped keep the drive "fed" with data under lower throughput as I have read ahead and async writes set in the stinit.def.

Another note is a read of "dd if=/dev/st0 of=/dev/null" only produced a result of 107MB/s. This, I would assume, is the real-world max effective throughput of this the drive and NOT 120 MB/s. The drive is currently on a dedicated SAS PCIe HBA with no other PCIe cards installed

In the meantime, I setup a 1TB RAID0 as a Disk2Tape buffer and had to add another disk to server to make this feasible.

I would still love to find away to do some sort of QOS for the tape drive and set writing to tape top priority so we can simplify our arrays and reduce parasitic hardaware costs, but in the mean time, I'm not seeing a way to NOT get around having a dedicated disk2tape buffer if I want to ensure continuous writes no matter what scheduled jobs hit the array.

  • What is your recovery point objective time, and how large is this backup in total? Making up numbers, if the goal is 12 hours / twice a day backups, an 18 hour backup time is no good. – John Mahowald Feb 16 '18 at 14:11
  • Probably just monthly to tape, about 750 GB right now. Once a year a few TB. The data in between those time frames will stored elsewhere. Mostly worried about tape shoe-shining and idle hardware. – Damon Feb 16 '18 at 22:34
1

The manual I've found lists variable speeds from 30.5 to 120 MB/s in ~7 MB/s increments.

Additionally, LTO drives use reasonably sized buffers to equalize the data stream and provide an indicator for speed adjustment, so unless the read speed varies wildly or is very low, backhitching should be minimal.

With data on a somewhat decent array and large files, 120 MB/s shouldn't even be much of a problem (unless the filesystem is highly fragmented). Our tape buffer uses two (slow) 4 TB drives in RAID 0 which can sustain appr. 270 MB/s but we don't write to the buffer while tapes are written.

  • The linked manual is circa 2014, ours is circa 2006. I wish they spec'd out the variable rates in our manual but the other manual's numbers are promising. How do you make sure to not write to the buffer array while tapes are written? Manual scheduling? Well timed tape writes? Do you use your RAID 0 for anything other than the tape write buffer? – Damon Feb 16 '18 at 22:31
  • I guess the older LTO-4 isn't much different from the more recent model. We've got phase 1 (backup VMs to buffer) and phase 2 (write buffer to tape) backup windows (phase 2 jobs also wait for all phase 1 jobs to finish, using flag files). The RAID 0 is only used as backup buffer (disk2disk2tape scheme) - it wasn't planned that way but we needed extra speed + space with no extra cost and can live with the lack of reliability. – Zac67 Feb 17 '18 at 9:41
  • The premise of this questions is really how to go to a single OBR6 down the road for simplicity and to keep parasitic costs (idle drives/power namely) down. Based on your answer, that may not really be a option if I want to guarantee throughput when writing to tapes. Sounds like I need to end up with a tape-buffer subsystem in our backup server probably utilizing RAID0 on some extra disks. I just hate parasitic cost no matter the price hence looking for an alternative. – Damon Feb 17 '18 at 17:32
  • Powerwise and for concurrent use, a sufficiently large SSD would be perfect - if budget and overall througput permit. – Zac67 Feb 17 '18 at 17:41
1

The mbuffer is a small and handy tool which can help you to maintain sustained data flow to the tape drive. It’s available on most linux distributions.

mbuffer - buffers I/O operations and displays the throughput rate. It is multi-threaded, supports network connections, and offers more options than the standard buffer.


Example usage with multithreaded compression on-the-fly:

tar cvf - /backupdir | lbzip2 | mbuffer -m 4G -L -P 80 > /dev/st0

  1. start adding files to the tar file archive
  2. (optional) compress it with lbzip2 to use all CPU cores
  3. begin filling memory buffer
  4. once filled to 80%, start sending data to the tape drive

mbuffer parameters explained:

  • -m 4 4GB memory buffer size. If necessary or available, use a larger buffer.
  • -L locked in memory (optional)
  • -P 80 start writing to the tape after 80% of the buffer has filled. There's no need to put 100 as it'll take some time for a tape drive to start writing, and it'll probably fill to 100% by then.

In this example, once the buffer fills up to 80% of capacity, it'll start sending data to the tape and mbuffer will continue to receive archive stream.

If the archiving process is slow and mbuffer has not received the data fast enough to keep up with the tape drive, it will stop sending data to the tape drive once it reaches 0%. Once the memory buffer is filled up to 80%, it'll start sending data to the tape drive, and recording will continue at full speed.

This way the "shoe-shining" of the tape is reduced to a minimum and tape drive will always get the data at the maximum speed it requires to sustain the stream.

You can also use mbuffer in reverse direction to read backup data from a tape drive and store the stream to some slower media or send it through the network.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.