3

This question is regarding deployment of web applications.

Intro (you can skip)

I'm using django, and the way my hosting provider have set up their django support ends up causing the webapp to be scattered in at least three locations:

  • Actual application code in /something/<my_apps>/
  • Site settings/urls/templates in /something/<my_site>/
  • css, javascript, and other "media" in /something_else/media/

So when I deploy/upgrade the website, I need to update several directories at once.

Actual question:

Is there a way to make an atomic file copying? I'm not "expert" at linux systems by any means, so please forgive my ignorance.

The copy operation involved several directory trees, either two or three, basically:

copy _tree1 to tree1
copy _tree2 to tree2

By atomic, I mean:

  • It's either fully copied, or not copied at all. It should never be in a state of some-copied-but-some-failed.
  • It's done in the shortest amount of time possible. Ideally there should be no point in time where the system sees the copy as it's happening in progress, it either sees the old version of the files or the new version; at no point in time shall it see the old version of file A but the new version of file B. If that's not entirely possible, then it should take no more than a few milli seconds.

My idea is to have something like double buffering: I prepare everything in a staging area, for instance, _tree_x, and then copy move it to tree_x with should be an atomic operation that simply changes pointers on the disk.

I think that a single such copy move operation is atomic in linux, (is it not?), but I need several such operations to be atomic as well; I want them to be treated as if they were a single move operation.

2
  • I would think that copying a file would be atomic, but not a directory. Jul 29, 2009 at 13:23
  • Copying a file is not atomic. Renaming a directory is (but not copying it). Feb 26, 2014 at 22:56

6 Answers 6

6

I think you are on the right track with a staging area. I'm not aware of any atomic commands, but if you stage your files and then use a script to remove the first directory and move (not copy) the second, and do it for all three directories, it should be really quick.

Alternatively, you may want to use symlinks. That way you could have roughly:

/version/22/<my-apps>
/version/22/<my-site>
/version/22/<my-media>

and deploy a /version/23 directory with the same subdirectories. Then, where the actual file would go (and again, for speed, you'll need a script), you can use a symbolic link so that when anyone goes to the latest page, they get whatever version is current (and it all happens transparently and they have no idea). The benefit of this is that your older work is still around until you decide to delete it. [Although, granted, a versioning control system is a better thing to use for keeping older work.]

You'd have to check that 1) you can run scripts (and in such a way that web users can not!), and 2) that you can use symlinks (as some web servers are configured to not follow them.)

2
  • Ah, symlinks! of course! Sound like the way to go.
    – hasen
    Jul 29, 2009 at 13:30
  • Not atomic in the strict sense, but satisfies your second "ASAP" condition. I've used similar tricks in high volume environments with little trouble. Feb 26, 2014 at 22:30
2

Perhaps I'm not thinking this out entirely, but why not do your copy operation to a new directory? When it's done "mv" the old directory to another name and "mv" the new directory into the desired name.

That's not technically atomic, since there's a period when the old directory will be moved and the new directory won't be in place yet, but it might be good enough.

2
  • This is also my idea (edited original question a bit to clarify), but I want several such move operations to go together as one operation.
    – hasen
    Jul 29, 2009 at 13:22
  • 3
    You cannot do multiple move operations as one atomic operation. Jul 29, 2009 at 13:26
0

For large websites, doing a site update can be handled by having multiple servers handling requests. You can then take one server off line, update it and then put it back on line , repeating for other servers in cluster.

For a single hosted site, perhaps it makes sense to shutdown the website by putting up a site closed page into index.html in the root folder, and then doing the changes.

If you really do need to keep the website up and running for as much of the time as possible, may i suggest the following:

  1. Atomic copies do not exist, however, renaming a single folder does happen atomically. By putting the renames into a script and running the script you can have a series of renames happen very quickly one after the other. You will need twice as much disk space as the site to have both the before and after folders on the server.

  2. This does not solve your problems - just reduces the exposure. The before and after versions may need different database data fields, so the SQL query will also need to run. A person could be doing a page load at the same time as your update occurs. The start of the web page load could load pages from before the change, and the last parts of the page load could use files from after the copy.

0

First, the best way to do this would be to change your httpd configuration to point to the new directories and then restart httpd. I assume that's not possible.

I have an idea that assumes that the data in your three directories isn't changing all the time, as it requires non-atomic moves of those three original directories to copies of those directories. I'm not 100% sure that this would work, but you could test it. It's easier for me to write this as a script than explain it in English. Let me know if I need to explain anything further.

Assume the three nominal paths are: /pathA/dir1, /pathA/dir2, /pathB/dir3

mkdir /pathC
ln -s pathC /linkI
cd /pathA
tar pcf - dir1 | (cd /pathC; tar pxf -)
tar pcf - dir2 | (cd /pathC; tar pxf -)
cd /pathB
tar pcf - dir3 | (cd /pathC; tar pxf -)
cd /pathA
mv dir1 dir1.orig && ln -s /linkI/dir1 dir1
mv dir2 dir2.orig && ln -s /linkI/dir2 dir2
cd /pathB
mv dir3 dir3.orig && ln -s /linkI/dir3 dir3
mkdir /pathD
cd /pathD
mkdir dir1
mkdir dir2
mkdir dir3
(cd dir1 && )
(cd dir2 && )
(cd dir3 && )
cd /
ln -sf pathD /linkI

(Edit: Hm, somehow I missed Clinton Blackmore's response above, which is basically identical to what I'm suggesting. So nevermind.)

1
  • hmm, tbh I don't quite understand the script. care to explain what's tar pcf? and piping to a ( parentheses )?
    – hasen
    Jul 30, 2009 at 1:22
0

For those reading this old question, it is possible, but it requires many symbolic links.

My answer is based on the one by Clinton Blackmore (the currently accepted answer).

Multiple directories (or multiple files, for that matter) cannot be changed atomically. So we cannot use directories directly. Single files can be updated atomically using the rename() system call (replacing the old file with the new one). Symbolic links can be updated the same way, using mv -T to let mv not dereference the destination.

So, if we have these directories:

/srv/dirA/
/srv/dirB/
/srv/dirC/

We can make them all be symbolic links to three other directories:

/srv/dirA -> /version/current/dirA
/srv/dirB -> /version/current/dirB
/srv/dirC -> /version/current/dirC

/version/current, in turn, is just a symbolic link to the directory with the current version:

/version/current -> /version/22/

The whole web application can then be updated with two simple commands, of which the last one 'replaces' all three directories at once (it doesn't actually replace the directories, it only replaces where those directories point at):

$ ln -s /version/23/ /version/next
$ mv -T /version/next /version/current

I have not actually tested this, but it should work. The -T flag may be a non-standard flag. As an alternative, python -c "import os; os.rename('/version/next', '/version/current')" may be used instead (also untested).

I don't know what the performance impact will be, but I doubt it will be significant if you're already running Django. I think it may matters if you're serving huge amounts of static files with the lowest latency possible (like CDNs), and even then it will probably only be a minor performance impact. In short, you shouldn't need to worry about performance.

Note, that there are a few gotchas: Django is a server and won't be restarted at exactly the same time. For this to be truly atomic, Django would have to be set up in a way that skips the notion of a 'current version' altogether. Instead, you would start Django from the current version for production use. For updating, the next version would be started, then the server would be restarted (I assume most webservers have some way of restarting without going offline), and the whole process should be atomic. But I'm no expert in this area.

Another gotcha (as mentioned by Ptolemy) is that on a busy servers, there will be some people who see pages partially from one version and partially from the other, due to caching and the fact that multiple resources are requested at different times during pageload (there can be a few seconds between resource loads). I think of the two, caching will be the most important one but also the easiest to work around. I doubt this will be much of an issue in practice, though.

0

Have you tried using atomic-rsync? It uses a combination of rsync and mv commands as suggested in other answers.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .