I am running an AWS EC2 system, with a shared NFS partition between the instances (5x m3.medium). I am uploading a constantly increasing binary file at a rate ~10x below my max bandwidth (uploading 250kBs on 2.5MBs pipe), using rsync --append. Simultaneously, I am analyzing the file on the NFS partition, and distributing these calculations using MPI.

While rsync is not running, the analysis using MPI works fine, taking ~5 seconds to run over the binary file. While rsync is running, MPI communication between the instances reaches a crawl, taking a few hours to complete the same task. The analysis script takes the most time at MPI_File_read_at() commands, reading the binary file on the NFS shared partition. When I stop the rsync and rerun the analysis, the speed is normal.

I have been working on this issue for a few days, and have not made much progress. I am not sure if the problem is using MPI on an rsyncing file, or the file being updated over NFS, or what. I've limited the bandwidth for rsync significantly (rsync --bwlimit=X), and the slowdown still happens.

I do not have much experience with any of these tools, but just trying to get a system that works at the moment. If anyone knows what would be causing this lag, or other incompatibilities between these methods I am using, I would appreciate the advice.

  • Have you looked at CloudWatch to see what it's saying? Particularly around EBS credits and CPU usage. Where is the NFS partition running, on an instance? Why are you using NFS? Have you considered AWS Elastic File System? Could it be some kind of blocking - does it actually finish? Could it be contention? What are you actually trying to achieve? – Tim Dec 12 '16 at 20:13
  • Thanks for the response. CloudWatch was a bit helpful. I am running the NFS partition on an instance I've denoted as "master". While running rsync AND the mpi analysis, the CPU usage on the master is ~100%, while 20% on each of the other node instances. I am actually using Starcluster, an academic toolkit for AWS instances. By default, this uses NFS, and I do not believe it can change. Although I don't really know any other way to have all instances read a constantly-changing binary file uploading from a local machine. Is the solution a better master instance? – HoosierPhysics Dec 12 '16 at 21:17
  • To answer the other points, the analysis does finish, just at an extremely slow rate. And I am trying to achieve both updating the data file using rsync, and analyzing that file between the EC2 instances, simultaneously without the slowdown on the analysis side. – HoosierPhysics Dec 12 '16 at 21:27
  • 1
    You haven't given anyone enough information to solve you problem, unless they're very familiar with that package, which is unlikely. You should probably edit your question to post information / graphics / etc from CloudWatch, or at least clearly describe the output. You need to demonstrate that you're not hitting limits - disk, CPU, EBS credits, etc. – Tim Dec 13 '16 at 0:43
  • As I mentioned when you posted this same question on Stack Overflow, you need to consider how much data rsync needs to read (from the write destination) before it can safely modify an existing file. The bandwidth limit almost certainly only applies to the network usage between source and target, while the reading I speak of would be done on the target only. Watch the output of iostat -x 1. – Michael - sqlbot Dec 13 '16 at 3:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.