1

I have Server A that works on a portion of a job and generates a file as output. Server B works on the second part of the job but it needs the file that Serve A generated.

What is the best way to get the file from Server A to Server B? This will happen many times and there might possibly be multiple transfers at the same time.

(These files are almost always <50KB, but can be as large 15MB)

I know I could use rsync or scp but I'm worried about efficiency with these transfers happening so often. Is that a valid worry?

I also looked into the option of NFS, but I need the ability to easily specify different servers and it doesn't seem smart to mount a new nfs every time I need to define a new server.

It's not pretty but for now what I've been doing is just PUT'ing the files over http to a script that writes them to the filesystem. The idea was to rewrite this as a simple client/server and cut the web server out of it. But I thought there must be an existing tool that does something similar.

2 Answers 2

1

There's sort of two questions here. The first one is the obvious one, where you want to share files between servers. You should be able to use NFS, or some sort of clustered filesystem like gluster to do that. Yes, you will have to change configurations as you add in servers.

The sort of second question is how you would push this configuration change to all the servers in question as you add in servers. That's more the domain of configuration management systems, like puppet or chef. With something like that, managing the, say, gluster configuration between the various Server As and the Server Bs can be done centrally, will be more version controlled, etc.

Alternatively, you can also used outside storage for the scratch files, like Server A pushing these out to Amazon S3, and Server B pulling from that common location. It's hard to say if this is a viable option without knowing more about your situation.

1

The best way is entirely subjective.

To me, the best way is, whatever reliably and verifiably gets the file from ServerA to ServerB, using the tools with which I am most familiar / can support most easily.

As such, I would send across the file (in your case, probably with rsync) and an associated hash file (MD5, SHA1, etc), then work it into your ServerA job to do that automatically. Then I would work it into your ServerB job to verify the data file using the hash file and continue the process.

I would likely also want to ensure that ServerB does not start working on a partially-transferred file, so I would probably resort to copying to a 'staging' directory on ServerB, then moving to a 'ready' directory — choosing only to poll or inotify the 'ready' directory.

Once that is in place, your immediate work is done and you can continue with the major milestones of your project and can come back to speeding up the transport later.

The most I might do in the early stages is structure the directories on ServerA, so I can tell what is being generated on ServerA and what is being copied to ServerB; probably with a 'pending' directory to which ServerA writes, a 'copying' directory to which ServerA moves the finished file and from which the hashing/rsync processes pick up the file, and an 'archive' directory to which ServerA moves the file when it is done copying to ServerB. That way I can get a rough idea of the latency/queue-length by checking the number of files in the 'copying' folder.

If you do find that you have to improve transfer times, you will probably find that optimising your network stack will be the best way to do so. Fatter pipes between the servers will be in order (e.g. upgrading 100Mb/s to 1Gb/s or even 10Gb/s). You might be tempted to try bonding multiple network interfaces but, if you do, make sure that your bonding algorithm is not picking the same interface every time based on source and destination IP addresses (or some other criteria that will not change — even source-IP+port to destination-IP+port will not offer increased throughput unless you can open multiple simultaneous connections from different source ports and parallelise the copy process).

If you still find the transport to be a prohibitive bottleneck, look to eliminate that in the upgrade path. Try re-factoring so that the jobs on ServerA and ServerB can both eventually be performed by the newer, beefier ServerC. If it is that important to management that these files are processed quickly, it will be an easy-enough sell come project review time.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .