Tag Info

Hot answers tagged

36

Since instant updates are also acceptable, you could use lsyncd. It watches directories (inotify) and will rsync changes to slaves. At startup it will do a full rsync, so that will take some time, but after that only changes are transmitted. Recursive watching of directories is possible, if a slave server is down the sync will be retried until it comes back. ...


30

If your array is md0 then echo "idle" > /sys/block/md0/md/sync_action 'idle' will stop an active resync/recovery etc. There is no guarantee that another resync/recovery may not be automatically started again, though some event will be needed to trigger this. http://www.mjmwired.net/kernel/Documentation/md.txt#477


29

Dropbox uses a binary diff algorythm to break down all files into blocks, and only upload blocks that it doesn't already have in the cloud. All of this is done locally on your computer. Dropbox doesn't just use your files that you have already uploaded, it aggregates everyones files into one database of blocks, and checks each local block hash against that ...


25

technical debt For the reasons below, it is much simpler to address this problem early on to avoid the accumulation of technical debt. Even if you find yourself already in this situation, it's probably better to deal with it in the near future than let it continue building. networked filesystems This question seems to be focused on the narrow scope of ...


21

Using -t (preserve timestamps) and --size-only will only compare files on size. If the size matches, rsync will not copy the file but since -t is specified, it will update the timestamp on the destination file without recopying it. Make sure to not use -u (update) as this will skip files that already exist and completely skip updating the timestamp. I had ...


16

I wanted to slow down or pause the resync process to save some I/O to backup some stuff on another computer. This thread helped me but I found another solution. On my Debian Lenny : echo "idle" > /sys/block/md0/md/sync_action works but the resync process is immediately restarted. checkarray -x --all : works, but same result: the resync process is ...


15

Connecting your ntpd to NTP servers outside your LAN to time sync can lead to the inconsistencies you are seeing, because every connection will have to go thru several routers, each one with unpredictable latencies depending on traffic. If each server connects by itself, the time between all the servers will drift a little. To avoid the inconsistency, the ...


14

once you reach a certain size (and it is always sooner than you think) you will realize that changing your passwords or disabling accounts for someone on all the hosts is a PITA. That's why people use systems with LDAP databases (or NIS but don't do that, not safe nowadays) like openldap or nowadays the excellent freeipa. You maintain all accounts/groups ...


11

And this is why there will always be robocopy. In fact, robocopy is so commonly used that it has been included by default in all versions of Windows since Vista. I've used robocopy since it first appeared in the first Windows NT Resource Kit, and it has never failed me, even with a huge number of files.


10

The reason people would run sync; sync before a halt is because the halt command wouldn't shutdown the system cleanly on older linuxes. The correct way to do this on SYSVr4 systems has always be to tell init to move to a different run level. BSD and SunOS 4 aren't SYSVr4 operating systems which is why they differ. Solaris (SunOS 5) is SYSVr4 and Linux picks ...


10

Consider using a distributed filesystem, such as GlusterFS. Being designed with replication and parallelism in mind, GlusterFS may scale up to 10 servers much more smoothly than ad-hoc solutions involving inotify and rsync. For this particular use-case, one could build a 10-server GlusterFS volume of 10 replicas (i.e. 1 replica/brick per server), so that ...


10

You really do want a Multicast File Transfer Program: UFTP, with decent documentation and proxt-style extensions for NAT/router traversal too.


9

if you run a NTP daemon you will not have this problem, it will speed up/slow down your clock so it gets synchronized over time rather then jumping.


9

I uh... uhm... wha... no, please don't do this. Active Directory needs to use ntds.dit and that's that. NTDS.dit, the Active Directory database, is way more than just a repository of user accounts. Using some sort of custom Franken-database in place of NTDS.dit is nuts, and would in no way be supported by Microsoft at all. You can do things like ...


8

Unison is close in that it will do bi-directional copying and runs on windows and Linux and is under the GPL. Unison is based on rsync so it can transfer pretty efficiently. It doesn't implement any of the scheduling or bandwidth limiting features you want. Those other features could be implemented by combining unison with additional tools and/or scripts ...


8

I'm using OCFS2 with DRBD. A DRBD resource /etc/drbd.d/r0.res: resource r0 { syncer { rate 1000M; } net { allow-two-primaries; after-sb-0pri discard-zero-changes; after-sb-1pri discard-secondary; after-sb-2pri disconnect; } startup { become-primary-on both; } on s1 { device /dev/drbd1; ...


8

I doubt rsync would work for this in the normal way, because scanning a million files and comparing it to the remote system 10 times would take to long. I would try to implement a system with something like inotify that keeps a list of modified files and pushes them to the remote servers (if these changes don't get logged in another way anyway). You can ...


8

1) There is no such thing a PDC or BDC anymore. Please stop confusing your terminology. 2) Don't use your DCs for other services. (Like filesharing, for example.) This causes a lot of easily avoidable headaches. 3) Technet guide to setting up DFS on Server 2003. Might want to set up a proper file server or two on a modern Windows Server OS and then use ...


7

We are using SyncBack, which has lots of good features in all versions: free, SE and PRO version: synchronization or backup over network shares or FTP, ZIP, filters, HTML and email logs, scheduling etc. We use both commercial and free versions on ours servers for web application deployment/synchronization, log file transfers and backup-to-disk jobs.


7

For what it's worth, Dropbox claims to create hashes on every 4MB of each file. That way, if you change a contiguous 2MB of a 100MB file, it will likely only need to upload 4MB (or 8MB if you cross into a second 4MB block) to re-sync the file. The hashes we use are only for the 4MB file chunks Source: http://forums.dropbox.com/topic.php?id=5502


7

rdiff-backup Have you ever taken a look at rdiff-backup? We've used it to synchronize over 4 million files from one Ubuntu file server to another without any issues. One very nice feature of rdiff-backup is that is keeps both a mirror copy and a differential copy (you can configure set how long). This comes in handy when you're backing up things like ...


7

If you can trust the filesystem last-modified timestamps, you can speed things up by combining Rsync with the UNIX/Linux 'find' utility. 'find' can assemble a list of all files that show last-modified times within the past day, and then pipe ONLY that shortened list of files/directories to Rsync. This is much faster than having Rsync compare the metadata of ...


6

rsync is pretty standard unison is good if you want to transfer data in both directions


6

Have you looked at KeePass at all? I find it works beautifully for me. It supports syncing with a password file that's hosted online, plus there are many other add-ins for it.


6

Given that the files are under version control, how about creating a branch in svn for the "production ready" version of your files. Then instead of doing an svn export on your master server, do a checkout on ALL of the servers, and use "svn update" to update them all from the "Production Ready" branch. The idea of having a production ready branch is that ...


6

You may also want to look into something like Unison. It's designed to do exactly what you want and will let you know if a file has been updated on both drives, whereas the rsync method will just clobber the file on the the USB disk.


6

Rsync if you are using Linux/Unix. For a Windows system you can use SyncToy from the PowerToys kit, or use Robocopy (from the Resource Kit for 2000-2003, incl. in 2008+ though) if you want to script it. I believe there are a few versions of rsync for windows. Since you want a multi-platform sync tool, rsync may be your best bet. One comment I would have ...


6

How about using DFS? It'll replicate files between servers (seeing as that's what it was designed for).


6

It doesn't take much bandwidth at all to do this. Even site to site VPN over DSL connections would be fine so long as you're not making massive changes to AD all the time, which most people don't. The only thing that would take awhile would be the initial seeding of the AD database, and you can even use a backup from an existing database to seed it upon ...



Only top voted, non community-wiki answers of a minimum length are eligible