16

I'm thinking of a situation where I would have something that creates a copy of a directory, tweaks a few files, and then does some processing on the result. This wold be done fairly often, maybe a few dozen times a day. (The exact use case is testing patch submissions; dupe the code, patch it, build/test/report/etc.)

What I'm looking for could be done by creating a new directory structure and populating it with hard links from the origonal. However this only works if all the tools you use delete and recreate files rather than edit them in place.

Is there a way to have the file system do copy-on-write for a file?


Note: I'm aware that many FSs use COW at a block level (all updates are done via writes to new blocks) but this is not what I want.

5
  • It doesn't appear maintained, but perhaps boklm.eu/copyfs
    – Zoredache
    Apr 7, 2010 at 3:07
  • So you want to do copy-on-write via the filesystem, but you don't want a filesystem that uses copy-on-write? I'm confused. Apr 7, 2010 at 5:02
  • 1
    @Kamil Kisiel: I want file-level COW not block-level COW. File-level COW results in cp and ln doing almost exactly the same thing. Block-level COW is a trick to minimize the interval that the FS meta data is incorrect: en.wikipedia.org/wiki/ZFS#Copy-on-write_transactional_model
    – BCS
    Apr 7, 2010 at 15:01
  • What kind of system is this expected to be used on? Apr 8, 2010 at 4:56
  • It's speculative enough that unless you know of a way to do it for some setup (in that case just post it) I wouldn't want people burning time figuring out how to do it (unless they are doing it for fun :)
    – BCS
    Apr 8, 2010 at 15:19

5 Answers 5

8

If you have your choice of platform for your fileserver, I'd go with a recent OpenSolaris build and use the deduplication feature of ZFS. This way copies of files would take up no additional space, and even common segments between files would not be replicated.

You can read all about ZFS deduplication at this post on Jeff Bonwick's Blog.

4
  • 1
    ZFS is amazing; it will change the way you think about storage
    – tegbains
    Apr 7, 2010 at 5:39
  • Neat, unless there's a down side to having it on for the whole drive.
    – BCS
    Apr 7, 2010 at 14:54
  • I added a link to a blog post that has more technical details. You can see if it's what you need there. Apr 8, 2010 at 3:57
  • New url for post of Jeff Bonwick: blogs.oracle.com/bonwick/zfs-deduplication-v2
    – Mathieu
    Jun 27, 2018 at 9:06
1

I find this question while searching a better solution instead of my current, which is like: cp -al SOURCEDIRNAME copy.SOURCEDIRNAME or

find SOURCEDIRNAME -type d -exec mkdir -p copy.{} ';'
find SOURCEDIRNAME -type f -exec ln {} copy.{} ';'

At this point i have a copy of the structure with a bunch of hardlinks. This works relative fast, and don't need one time more disk space, cause we do not rly copy all files. Now i can do some ''stuff'' (like patch or build) on the copy directory. If they do not modify the original files, only rename them and/or create new files. After the work, i have some results and can remove the copy directory and start over.

If you use more complicated tools and dont trust them (or you know they try to overwrite existing files) you may need cowdancer.

1

Try use one of UniouFS, AUFS or Overlay Filesystem (depends on which supported in Your distro)

For example OverlayFS used by docker as one of drivers to merge layers

Overlay Filesystem

Overlaying mainly involves directories. If a given name appears in both upper and lower filesystems and refers to a non-directory in either, then the lower object is hidden - the name refers only to the upper object.

Where both upper and lower objects are directories, a merged directory is formed.

At mount time, the two directories given as mount options “lowerdir” and “upperdir” are combined into a merged directory:

mount -t overlay overlay -olowerdir=/lower,upperdir=/upper,workdir=/work /merged

The “workdir” needs to be an empty directory on the same filesystem as upperdir.

0

Most linux virtualization solutions offer file-level COW among virtualized OSes (copied from the WP):

  • YES:
    • Docker, Linux-VServer, lmctfy, LXC, Singularity, Virtuozzo, Sandboxie, systemd-nspawn, Solaris Containers(ZFS), FreeBSD jail(ZFS), OpenVZ
  • NO:
    • chroot, sysjail, WPARs, iCore Virtual Accounts, Turbo
1
  • While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes Feb 8, 2019 at 20:01
-2

May be SVN or other versioning tool (git, cvs)?

1
  • That's even worse because it creates two copies of each file and sucks them down over a socket.
    – BCS
    Apr 7, 2010 at 14:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.