Tag Info

Hot answers tagged

37

Since instant updates are also acceptable, you could use lsyncd. It watches directories (inotify) and will rsync changes to slaves. At startup it will do a full rsync, so that will take some time, but after that only changes are transmitted. Recursive watching of directories is possible, if a slave server is down the sync will be retried until it comes back. ...


31

If your array is md0 then echo "idle" > /sys/block/md0/md/sync_action 'idle' will stop an active resync/recovery etc. There is no guarantee that another resync/recovery may not be automatically started again, though some event will be needed to trigger this. http://www.mjmwired.net/kernel/Documentation/md.txt#477


30

Dropbox uses a binary diff algorithm to break down all files into blocks, and only upload blocks that it doesn't already have in the cloud. All of this is done locally on your computer. Dropbox doesn't just use your files that you have already uploaded, it aggregates everyone's files into one database of blocks, and checks each local block hash against ...


25

technical debt For the reasons below, it is much simpler to address this problem early on to avoid the accumulation of technical debt. Even if you find yourself already in this situation, it's probably better to deal with it in the near future than let it continue building. networked filesystems This question seems to be focused on the narrow scope of ...


24

Using -t (preserve timestamps) and --size-only will only compare files on size. If the size matches, rsync will not copy the file but since -t is specified, it will update the timestamp on the destination file without recopying it. Make sure to not use -u (update) as this will skip files that already exist and completely skip updating the timestamp. I had ...


16

I wanted to slow down or pause the resync process to save some I/O to backup some stuff on another computer. This thread helped me but I found another solution. On my Debian Lenny : echo "idle" > /sys/block/md0/md/sync_action works but the resync process is immediately restarted. checkarray -x --all : works, but same result: the resync process is ...


15

Connecting your ntpd to NTP servers outside your LAN to time sync can lead to the inconsistencies you are seeing, because every connection will have to go thru several routers, each one with unpredictable latencies depending on traffic. If each server connects by itself, the time between all the servers will drift a little. To avoid the inconsistency, the ...


14

once you reach a certain size (and it is always sooner than you think) you will realize that changing your passwords or disabling accounts for someone on all the hosts is a PITA. That's why people use systems with LDAP databases (or NIS but don't do that, not safe nowadays) like openldap or nowadays the excellent freeipa. You maintain all accounts/groups ...


11

And this is why there will always be robocopy. In fact, robocopy is so commonly used that it has been included by default in all versions of Windows since Vista. I've used robocopy since it first appeared in the first Windows NT Resource Kit, and it has never failed me, even with a huge number of files.


10

The reason people would run sync; sync before a halt is because the halt command wouldn't shutdown the system cleanly on older linuxes. The correct way to do this on SYSVr4 systems has always be to tell init to move to a different run level. BSD and SunOS 4 aren't SYSVr4 operating systems which is why they differ. Solaris (SunOS 5) is SYSVr4 and Linux picks ...


10

Consider using a distributed filesystem, such as GlusterFS. Being designed with replication and parallelism in mind, GlusterFS may scale up to 10 servers much more smoothly than ad-hoc solutions involving inotify and rsync. For this particular use-case, one could build a 10-server GlusterFS volume of 10 replicas (i.e. 1 replica/brick per server), so that ...


10

You really do want a Multicast File Transfer Program: UFTP, with decent documentation and proxt-style extensions for NAT/router traversal too.


9

You misunderstand rsync. This command: rsync -avv left/ right/ will not sync anything in right to left. It will, as @atbg says, only sync left to right. Rsync is not a bi-directional syncer. It syncs the dest with the source. Man page for reference: http://linux.die.net/man/1/rsync


9

I uh... uhm... wha... no, please don't do this. Active Directory needs to use ntds.dit and that's that. NTDS.dit, the Active Directory database, is way more than just a repository of user accounts. Using some sort of custom Franken-database in place of NTDS.dit is nuts, and would in no way be supported by Microsoft at all. You can do things like ...


9

if you run a NTP daemon you will not have this problem, it will speed up/slow down your clock so it gets synchronized over time rather then jumping.


8

For what it's worth, Dropbox claims to create hashes on every 4MB of each file. That way, if you change a contiguous 2MB of a 100MB file, it will likely only need to upload 4MB (or 8MB if you cross into a second 4MB block) to re-sync the file. The hashes we use are only for the 4MB file chunks Source: http://forums.dropbox.com/topic.php?id=5502


8

Unison is close in that it will do bi-directional copying and runs on windows and Linux and is under the GPL. Unison is based on rsync so it can transfer pretty efficiently. It doesn't implement any of the scheduling or bandwidth limiting features you want. Those other features could be implemented by combining unison with additional tools and/or scripts ...


8

I'm using OCFS2 with DRBD. A DRBD resource /etc/drbd.d/r0.res: resource r0 { syncer { rate 1000M; } net { allow-two-primaries; after-sb-0pri discard-zero-changes; after-sb-1pri discard-secondary; after-sb-2pri disconnect; } startup { become-primary-on both; } on s1 { device /dev/drbd1; ...


8

1) There is no such thing a PDC or BDC anymore. Please stop confusing your terminology. 2) Don't use your DCs for other services. (Like filesharing, for example.) This causes a lot of easily avoidable headaches. 3) Technet guide to setting up DFS on Server 2003. Might want to set up a proper file server or two on a modern Windows Server OS and then use ...


8

I doubt rsync would work for this in the normal way, because scanning a million files and comparing it to the remote system 10 times would take to long. I would try to implement a system with something like inotify that keeps a list of modified files and pushes them to the remote servers (if these changes don't get logged in another way anyway). You can ...


7

Of course minutes later I discover an easy method. rsync has a --prune-empty-dirs option.


7

rdiff-backup Have you ever taken a look at rdiff-backup? We've used it to synchronize over 4 million files from one Ubuntu file server to another without any issues. One very nice feature of rdiff-backup is that is keeps both a mirror copy and a differential copy (you can configure set how long). This comes in handy when you're backing up things like ...


7

We are using SyncBack, which has lots of good features in all versions: free, SE and PRO version: synchronization or backup over network shares or FTP, ZIP, filters, HTML and email logs, scheduling etc. We use both commercial and free versions on ours servers for web application deployment/synchronization, log file transfers and backup-to-disk jobs.


7

If you can trust the filesystem last-modified timestamps, you can speed things up by combining Rsync with the UNIX/Linux 'find' utility. 'find' can assemble a list of all files that show last-modified times within the past day, and then pipe ONLY that shortened list of files/directories to Rsync. This is much faster than having Rsync compare the metadata of ...


7

lsyncd seems to be the perfect solution. it combines inotify (kernel builtin function witch watches for file changes in a directory trees) and rsync (cross platform file-syncing-tool). lsyncd -rsyncssh /home remotehost.org backup-home/ Quote from github: Lsyncd watches a local directory trees event monitor interface (inotify or fsevents). It ...


6

Try unison, it was specifically designed to solve this problem by keeping the change lists (building file list), locally to each server, speeding up the time to calculate the delta, and the reduce amount that is sent across the wire afterwards.


6

It doesn't take much bandwidth at all to do this. Even site to site VPN over DSL connections would be fine so long as you're not making massive changes to AD all the time, which most people don't. The only thing that would take awhile would be the initial seeding of the AD database, and you can even use a backup from an existing database to seed it upon ...


6

If you just want to keep a remote copy of the file then using something based on rsync will probably work for you. Rsync only copies the parts of the file that have changed so it can reduce the amount of data you have to to copy. For windows you could use deltacopy or syncrify.


6

rsync is pretty standard unison is good if you want to transfer data in both directions


6

Have you looked at KeePass at all? I find it works beautifully for me. It supports syncing with a password file that's hosted online, plus there are many other add-ins for it.



Only top voted, non community-wiki answers of a minimum length are eligible