Tag Info

Hot answers tagged

35

Since instant updates are also acceptable, you could use lsyncd. It watches directories (inotify) and will rsync changes to slaves. At startup it will do a full rsync, so that will take some time, but after that only changes are transmitted. Recursive watching of directories is possible, if a slave server is down the sync will be retried until it comes back. ...


29

Dropbox uses a binary diff algorythm to break down all files into blocks, and only upload blocks that it doesn't already have in the cloud. All of this is done locally on your computer. Dropbox doesn't just use your files that you have already uploaded, it aggregates everyones files into one database of blocks, and checks each local block hash against that ...


28

If your array is md0 then echo "idle" > /sys/block/md0/md/sync_action 'idle' will stop an active resync/recovery etc. There is no guarantee that another resync/recovery may not be automatically started again, though some event will be needed to trigger this. http://www.mjmwired.net/kernel/Documentation/md.txt#477


24

technical debt For the reasons below, it is much simpler to address this problem early on to avoid the accumulation of technical debt. Even if you find yourself already in this situation, it's probably better to deal with it in the near future than let it continue building. networked filesystems This question seems to be focused on the narrow scope of ...


14

once you reach a certain size (and it is always sooner than you think) you will realize that changing your passwords or disabling accounts for someone on all the hosts is a PITA. That's why people use systems with LDAP databases (or NIS but don't do that, not safe nowadays) like openldap or nowadays the excellent freeipa. You maintain all accounts/groups ...


14

I wanted to slow down or pause the resync process to save some I/O to backup some stuff on another computer. This thread helped me but I found another solution. On my Debian Lenny : echo "idle" > /sys/block/md0/md/sync_action works but the resync process is immediately restarted. checkarray -x --all : works, but same result: the resync process is ...


11

And this is why there will always be robocopy. In fact, robocopy is so commonly used that it has been included by default in all versions of Windows since Vista. I've used robocopy since it first appeared in the first Windows NT Resource Kit, and it has never failed me, even with a huge number of files.


10

You really do want a Multicast File Transfer Program: UFTP, with decent documentation and proxt-style extensions for NAT/router traversal too.


9

I uh... uhm... wha... no, please don't do this. Active Directory needs to use ntds.dit and that's that. NTDS.dit, the Active Directory database, is way more than just a repository of user accounts. Using some sort of custom Franken-database in place of NTDS.dit is nuts, and would in no way be supported by Microsoft at all. You can do things like ...


9

if you run a NTP daemon you will not have this problem, it will speed up/slow down your clock so it gets synchronized over time rather then jumping.


8

Unison is close in that it will do bi-directional copying and runs on windows and Linux and is under the GPL. Unison is based on rsync so it can transfer pretty efficiently. It doesn't implement any of the scheduling or bandwidth limiting features you want. Those other features could be implemented by combining unison with additional tools and/or scripts ...


8

I'm using OCFS2 with DRBD. A DRBD resource /etc/drbd.d/r0.res: resource r0 { syncer { rate 1000M; } net { allow-two-primaries; after-sb-0pri discard-zero-changes; after-sb-1pri discard-secondary; after-sb-2pri disconnect; } startup { become-primary-on both; } on s1 { device /dev/drbd1; ...


8

Consider using a distributed filesystem, such as GlusterFS. Being designed with replication and parallelism in mind, GlusterFS may scale up to 10 servers much more smoothly than ad-hoc solutions involving inotify and rsync. For this particular use-case, one could build a 10-server GlusterFS volume of 10 replicas (i.e. 1 replica/brick per server), so that ...


8

1) There is no such thing a PDC or BDC anymore. Please stop confusing your terminology. 2) Don't use your DCs for other services. (Like filesharing, for example.) This causes a lot of easily avoidable headaches. 3) Technet guide to setting up DFS on Server 2003. Might want to set up a proper file server or two on a modern Windows Server OS and then use ...


7

I doubt rsync would work for this in the normal way, because scanning a million files and comparing it to the remote system 10 times would take to long. I would try to implement a system with something like inotify that keeps a list of modified files and pushes them to the remote servers (if these changes don't get logged in another way anyway). You can ...


7

For what it's worth, Dropbox claims to create hashes on every 4MB of each file. That way, if you change a contiguous 2MB of a 100MB file, it will likely only need to upload 4MB (or 8MB if you cross into a second 4MB block) to re-sync the file. The hashes we use are only for the 4MB file chunks Source: http://forums.dropbox.com/topic.php?id=5502


7

rdiff-backup Have you ever taken a look at rdiff-backup? We've used it to synchronize over 4 million files from one Ubuntu file server to another without any issues. One very nice feature of rdiff-backup is that is keeps both a mirror copy and a differential copy (you can configure set how long). This comes in handy when you're backing up things like ...


7

We are using SyncBack, which has lots of good features in all versions: free, SE and PRO version: synchronization or backup over network shares or FTP, ZIP, filters, HTML and email logs, scheduling etc. We use both commercial and free versions on ours servers for web application deployment/synchronization, log file transfers and backup-to-disk jobs.


6

rsync is pretty standard unison is good if you want to transfer data in both directions


6

Have you looked at KeePass at all? I find it works beautifully for me. It supports syncing with a password file that's hosted online, plus there are many other add-ins for it.


6

Given that the files are under version control, how about creating a branch in svn for the "production ready" version of your files. Then instead of doing an svn export on your master server, do a checkout on ALL of the servers, and use "svn update" to update them all from the "Production Ready" branch. The idea of having a production ready branch is that ...


6

You may also want to look into something like Unison. It's designed to do exactly what you want and will let you know if a file has been updated on both drives, whereas the rsync method will just clobber the file on the the USB disk.


6

Rsync if you are using Linux/Unix. For a Windows system you can use SyncToy from the PowerToys kit, or use Robocopy (from the Resource Kit for 2000-2003, incl. in 2008+ though) if you want to script it. I believe there are a few versions of rsync for windows. Since you want a multi-platform sync tool, rsync may be your best bet. One comment I would have ...


6

How about using DFS? It'll replicate files between servers (seeing as that's what it was designed for).


6

If you just want to keep a remote copy of the file then using something based on rsync will probably work for you. Rsync only copies the parts of the file that have changed so it can reduce the amount of data you have to to copy. For windows you could use deltacopy or syncrify.


6

It doesn't take much bandwidth at all to do this. Even site to site VPN over DSL connections would be fine so long as you're not making massive changes to AD all the time, which most people don't. The only thing that would take awhile would be the initial seeding of the AD database, and you can even use a backup from an existing database to seed it upon ...


6

rsync by default does not copy files based on their modification time and size. If you are not using -t in the command line, rsync will not preserve the modification times and the size + time check will of course fail. There is also a good chance that the timestamp precision of your differing filesystems is causing the problem - ext4 supports microseconds ...


5

There are two popular approaches You either setup central authentication (ldap, nis). You setup a configuration management system (puppet,chef,cfengine, shell script) to automatically create accounts, and configure the environment on all your managed systems. Central authentication systems are good when all the systems are all under the control of the ...


5

Drive "resync" occurs if a drive swap happens and the RAID is being rebuilt by the system. In RAID 1, this means copying the entire drive. You can use the OS as normal, but it's will noticeably slow and during sync you do not have redundancy as the RAID is in a degraded state.



Only top voted, non community-wiki answers of a minimum length are eligible