Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I run multiple instances per machine of a large (~10GB) closed-source service under linux. Is there a solution to somehow have common files between the installations not take much space? (nearly all of the installs are common to all the installations). I thought about making a big hardlink tree, but each install runs its own auto-updater, so things could end up inconsistent.

share|improve this question
    
That would be really hard, if you can't get the auto-updater out of the picture. Upvoted, because I find it an interesting problem. –  Kvisle Oct 16 '11 at 20:43

3 Answers 3

up vote 1 down vote accepted

Sounds like you need a filesystem with deduplication support.

If there are no native filesystems with dedup support, you could consider hosting your data via NFS and placing it on say, ZFS, so each new copy doesn't take up as much additional space.

share|improve this answer
    
I'm not sure it's just a deduplication issue; he has several instances operating independently and may make alterations to the filesystem as if they were independent. Without the applications being aware of each other, you might risk corruption problems. –  Bart Silverstrim Oct 17 '11 at 23:20
    
"solution to somehow have common files between the installations not take much space?" -- that's exactly what deduplicating filesystems are for. If 4000 files take up 10GB space and 90% of the contents are identical, the diskspace is used only once. Dedupliating filesystems can often work on either file-level, block-level or byte-level. –  Alex Holst Oct 18 '11 at 17:00
    
Yes, deduplication works well for things like mail servers where one 2 meg attachment is sent to 10 users; deduping will keep it storing one 2 meg file and 9 pointers to that file rather than 20 meg in 10 files. Or for backups solutions. But the OP has several servers designed to act like they are independent across multiple servers; depending on how it's designed he may run into issues like the kind you see in sharing an application from a network share not meant to be shared among running instances. –  Bart Silverstrim Oct 18 '11 at 17:51
    
If the application isn't designed for multiple access you may run into issues with the filesystem playing magic tricks under the application. It's essentially asking for multiple co-instances of snapshots to be running from a central storage for a particular set of files while keeping others running live. In either case, he's playing with getting it to do what it's not designed to do (the closed-source application) and risks having problems if he does it. The best thing to do is work with the company/devs and find a solution through that so it's not broken in the process (or corrupted.) –  Bart Silverstrim Oct 18 '11 at 17:54
    
In the setup I'm talking about, there's no way the application will ever know what's going on. Here's an example of a guy using network exported ZFS store to dedup NTFS volumes: edplese.com/blog/2009/11/24/zfs-deduplication-with-ntfs –  Alex Holst Oct 19 '11 at 20:18

This would almost entirely depend on how your software is architected. Assuming you have access to the source code and programmers behind the program, you could talk to the devs and see if there's a way to share common data. But if each program is using its own database or something like that, you're out of luck.

Otherwise you could store the data on some kind of NAS or SAN and keep the common information in a common location.

If the program isn't made for it, though, I'd definitely not risk it. You'll corrupt things and create more problems for yourself.

share|improve this answer

I have an application that is I think might be similar.

I run ~30 instances of a particular app each running on it's own VM under vsphere. Each server shares the same vmdk file for the application data store (~12 GB) which is configured as independant/non-persistant (which causes data only be persistant across reboots, and not shutdowns). symlinks are in place pointing the application to the data files on the non-persistant volume. Application state and the data we need to be persistant is written to another separate volume.

When an update is released to the data set and assuming we can't shutdown the VMs (or don't have a maintenance window), we simply push the update to the non-persistant store as the changes are non-persistant and vm-specific (at least until the next shutdown). And during the next available change window, we can then finally update the backing vmdk.

We've done this live a couple times in dev with hot-removing/hot-adding disks with the VM never going down, but, that's probably not something we could do in production.

Obviously, this only works if you have some way of virtualizing your application, and some way of controlling how your application updates.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.