I am running an AWS EC2 system, with a shared NFS partition between the instances (5x m3.medium). I am uploading a constantly increasing binary file at a rate ~10x below my max bandwidth (uploading 250kBs on 2.5MBs pipe), using
rsync --append. Simultaneously, I am analyzing the file on the NFS partition, and distributing these calculations using MPI.
rsync is not running, the analysis using MPI works fine, taking ~5 seconds to run over the binary file. While
rsync is running, MPI communication between the instances reaches a crawl, taking a few hours to complete the same task. The analysis script takes the most time at
MPI_File_read_at() commands, reading the binary file on the NFS shared partition. When I stop the
rsync and rerun the analysis, the speed is normal.
I have been working on this issue for a few days, and have not made much progress. I am not sure if the problem is using MPI on an
rsyncing file, or the file being updated over NFS, or what. I've limited the bandwidth for rsync significantly (
rsync --bwlimit=X), and the slowdown still happens.
I do not have much experience with any of these tools, but just trying to get a system that works at the moment. If anyone knows what would be causing this lag, or other incompatibilities between these methods I am using, I would appreciate the advice.