How can I set up temporary failover for a “write-only” smb share?

Question

I am tasked to set up a storage server which will be used to "receive" data from an Illumina machine running very expensive DNA sequencing operations with limited interruptibility. The sequencer itself is attached to a Windows PC provided by the vendor to run the control software which expects a path where it can dump the data (about 12-16TB over a period of a couple of weeks). The storage server provides a share for this purpose using CentOS/samba. My question is: is there a way that I can cause the share mounted on the Windows PC to "re-map" or at least cache to local storage in the event of the storage server or network going down? And of course, I'd want to be notified that this has happened. Keep in mind that we can't just mirror everything locally because there isn't enough space. Basically I want the sequencer software to be able to keep writing to the share as long as there is some local space to keep the data until it can be sync'd. If this is possible I'd of course add an extra drive or two to increase the time we have to fix the problem.

I am mostly a Unix guy so pointers to "obvious" solutions in the Windows world are appreciated. I don't want to install all kinds of crazy third-party stuff on the sequencer's Windows machine, but solid, tested solutions are probably okay.

EDIT: should add, I don't mind if there is a creative solution that doesn't use smb at all.

TLDR: what's important is that a) the Windows client software sees a path where it can write data files b) one way or another those files end up on the storage server c) if the network connection or storage server fails, there will be some window of time in which the software won't be blocked and can continue writing data, using locally attached storage as a cache.

Answer 1

It's an odd problem. This is the type of thing you'd expect in a centralized storage environment because of all the valuable work being done by servers centrally, however I've never heard of someone doing it with single-attached storage.

You didn't mention whether the storage server was provided or your own- in any case, the only cheap way I know to have sort-of high availability with files is to use DFS and have some alternate path prepared and ready to switch to. The problem is that you'll still experience write fails until you remap DFS to the new share.

This is a simple problem to fix if you have all the money in the world- simply install a clustered storage device that supports geographic displacement of nodes, and have a multi-path driver running. Netapp using a metro-cluster is an example of this, and if you're looking at disks, SVC from IBM with split IO pair and maybe even VDisk mirroring.

edit: @mfinni got me thinking about scripts. You say your control app wants a "path" to a large network share- does it need to know it's large? What if you instructed it to write to the local drive? Could it be configured to write files of a specific, small enough, size that you could keep a script running that would destage them over to the NAS as they are closed by the control application?

Also, will it write to tape? A tape drive would be perfect for these types of write only applications- each cartridge holds 1.6TB of uncompressed data.

Answer 2

Not really. You could set up a failover cluster for the NAS share which would reduce the likelihood of failure. That doesn't remove the possibility of data loss however, it just reduces the points of failure.

The proper way to do this, if data loss isn't allowable, is one of two things:

1. Push back on the software vendor to not allow data to be lost when the network storage disappears.
2. Have the software write locally, and then you write a copy script that will not copy if the network storage disappears.

Answer 3

What kind of output does a DNA sequencer have?? im guessing its a super large database split into 2gb files? You could write a small batch file that checks if there are more then 2 files and copies the one with the oldest modified date and once its completed successfully it deletes it. (assuming it only outputs files and does not need old files) if the copy fails for some reason (e.g. the NAS goes down) it simply does not delete the local files and sends an email.

IMHO a stable NAS has as much chance of going down as the computer connected to the DNA sequencer, yeah the network could go down but if the NAS has a spare Ethernet port you could just plug them directly using a crossover (or not, MDIX and such...)