3

As part of our development workflow, we would like to have our build server publish our compiled binaries and other build artifacts to some LAN-based, replicated file system. Ideally, we'd just have our build server drop the build artifacts into a folder according to the build version and the contents of that folder would be replicated to each developer workstations locally. Once published, each build is static and does not change. We don't have strong consistency requirements, we only need the files to be available on each developer workstation within a minute or so. All machines are Linux-based.

The easiest solution is probably a NAS with a rsync+cron, but that feels clunky. Is there another distributed/replicated file system that supports the above requirements? We looked at doing Amazon S3 + s3fs, but that was painfully slow.

In many ways this feels like traditional mirroring from an authoritative source in a webserver scenario, e.g. www1, www2, etc.

Improve this question

3 Answers 3

Reset to default
2

If you only need one-way replication, you really don't want a distributed filesystem. That introduces a lot of complexity for no gain. I'd go for the rsync approach. Have the last step in your build script be pushing out the artifacts if the build was successful. If you'd rather not make this be part of your build scripts, you could use cron to do it at certain intervals or lsyncd to do it when files are created or modified.

Improve this answer
2
  • How would lsyncd handle sporadic network failures? For example, if one of the clients was unreachable when the change occurred? Would that particular client miss out on the update? That's one of the reasons a distributed FS appeals to us because all of that is handled automatically.
    – Jonathan Oliver
    May 5, 2011 at 1:19
  • 1
    Because of our network topology, we ultimately decided to go with a polling-based rsync solution.
    – Jonathan Oliver
    May 11, 2011 at 0:53
2

I think AFS (or OpenAFS) might be a good solution for you. Its been implemented in the linux kernel since 2.6.10 and uses Kerberos for authentication.

Improve this answer
2

I would also suggest looking into approaching this with a distributed version control system such as git. You could use commit hooks to keep everybody up to date, and the independence you would gain by not doing this at the file system level would save you a lot of reliability headaches down the road. You might even find other advantages of having the stuff versioned along the way.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .