76

Using -t (preserve timestamps) and --size-only will only compare files on size. If the size matches, rsync will not copy the file but since -t is specified, it will update the timestamp on the destination file without recopying it. Make sure to not use -u (update) as this will skip files that already exist and completely skip updating the timestamp. I had ...


39

Since instant updates are also acceptable, you could use lsyncd. It watches directories (inotify) and will rsync changes to slaves. At startup it will do a full rsync, so that will take some time, but after that only changes are transmitted. Recursive watching of directories is possible, if a slave server is down the sync will be retried until it comes back. ...


33

Some unrelated points: 80K is a lot of files. 80,000 files in one directory? No operating system or app handles that situation very well by default. You just happen to notice this problem with rsync. Check your rsync version Modern rsync handles large directories a lot better than in the past. Be sure you are using the latest version. Even old rsync ...


31

technical debt For the reasons below, it is much simpler to address this problem early on to avoid the accumulation of technical debt. Even if you find yourself already in this situation, it's probably better to deal with it in the near future than let it continue building. networked filesystems This question seems to be focused on the narrow scope of ...


30

lsyncd seems to be the perfect solution. it combines inotify (kernel builtin function witch watches for file changes in a directory trees) and rsync (cross platform file-syncing-tool). lsyncd -rsyncssh /home remotehost.org backup-home/ Quote from github: Lsyncd watches a local directory trees event monitor interface (inotify or fsevents). It aggregates ...


30

Of course minutes later I discover an easy method. rsync has a --prune-empty-dirs option.


23

The simple answer is "set the date manually", which you need to do, but to prevent this occurring again, there is more that you should do. Ensure that the system timezone configuration is in a sane state. Unless there is a very strong reason not to do so (such as software compatibility issues), server clocks should always run on UTC time. If you decide ...


22

To exclude all files/folders: cd to your dropbox folder (usually cd ~/Dropbox) then type ~/bin/dropbox.py exclude add * This will exclude everything in your dropbox folder from syncing. (Be careful! This will remove all the files that you synced) Then, if you want to start syncing the folder "dir", type ~/bin/dropbox.py exclude remove dir Taken from http:/...


20

Connecting your ntpd to NTP servers outside your LAN to time sync can lead to the inconsistencies you are seeing, because every connection will have to go thru several routers, each one with unpredictable latencies depending on traffic. If each server connects by itself, the time between all the servers will drift a little. To avoid the inconsistency, the ...


18

once you reach a certain size (and it is always sooner than you think) you will realize that changing your passwords or disabling accounts for someone on all the hosts is a PITA. That's why people use systems with LDAP databases (or NIS but don't do that, not safe nowadays) like openldap or nowadays the excellent freeipa. You maintain all accounts/groups ...


14

With standard options, rsync will copy the new file assigning it a semi-random name, then it will rename the new file with the original name. In this process, no writes are directed at the original file, preserving its hadlinks. On the other side, using the non-default--inplace option will overwrite the original file and its hardlinks. Anyway, I strongly ...


13

You can cancel an array resync in progress using the following sequence of commands (as root): echo frozen > /sys/block/md0/md/sync_action echo none > /sys/block/md0/md/resync_start echo idle > /sys/block/md0/md/sync_action Note that this may leave your array in an inconsistent state. Don't do this unless you're sure the array is in good shape, ...


13

The cause of your ListObjects error is that you assigned permission to access the contents of your bucket (arn:aws:s3:::bucket/*) but you did not give permissions to the bucket itself (arn:aws:s3:::bucket). The ListObjects command requires access to the bucket. To test this, I did the following: Used two AWS accounts: Account A, Account B Created bucket-a ...


12

You really do want a Multicast File Transfer Program: UFTP, with decent documentation and proxt-style extensions for NAT/router traversal too.


11

Consider using a distributed filesystem, such as GlusterFS. Being designed with replication and parallelism in mind, GlusterFS may scale up to 10 servers much more smoothly than ad-hoc solutions involving inotify and rsync. For this particular use-case, one could build a 10-server GlusterFS volume of 10 replicas (i.e. 1 replica/brick per server), so that ...


10

There are two popular approaches You either setup central authentication (ldap, nis). You setup a configuration management system (puppet,chef,cfengine, shell script) to automatically create accounts, and configure the environment on all your managed systems. Central authentication systems are good when all the systems are all under the control of the same ...


9

I uh... uhm... wha... no, please don't do this. Active Directory needs to use ntds.dit and that's that. NTDS.dit, the Active Directory database, is way more than just a repository of user accounts. Using some sort of custom Franken-database in place of NTDS.dit is nuts, and would in no way be supported by Microsoft at all. You can do things like ...


9

Apparently this is by design: If the local clock time of the client is less than three minutes ahead of the time on the server, W32Time will quarter or halve the clock frequency for long enough to bring the clocks into sync. If the client is less that 15 seconds ahead, it will halve the frequency; otherwise, it will quarter the frequency. The ...


8

I doubt rsync would work for this in the normal way, because scanning a million files and comparing it to the remote system 10 times would take to long. I would try to implement a system with something like inotify that keeps a list of modified files and pushes them to the remote servers (if these changes don't get logged in another way anyway). You can ...


8

1) There is no such thing a PDC or BDC anymore. Please stop confusing your terminology. 2) Don't use your DCs for other services. (Like filesharing, for example.) This causes a lot of easily avoidable headaches. 3) Technet guide to setting up DFS on Server 2003. Might want to set up a proper file server or two on a modern Windows Server OS and then use ...


7

I can only speak to why you would issue sync multiple times. The command schedules the flush to disk but returns before the actual flush completes. Any subsequent sync command will block until any outstanding flush is in progress before scheduling another flush and exiting. Therefore, sync; sync ensures a synchronous flush. You do not need to do it more than ...


7

Answer to Question 1 MySQL Replication suffers from two major problems MySQL Replication is Asynchronous. This may introduce replication delay. This manifests itself with communication problems between a Master and the Slave via the Slave I/O Thread. This may logically and numerically be seen in Seconds_Behind_Master. Data Drift. This is a intermittent ...


7

As it was mentioned already, DRBD isn't greatest solution especially if you have a requirements for further scale. I believe that the best solution on Linux would be Ceph Cluster - https://en.wikipedia.org/wiki/Ceph_(software) with its native cluster aware file system CephFS. For the configuration of an HA storage on Windows, take into account StarWind VSAN ...


6

Using --size-only will cause rsync to skip comparing file timestamps (and therefore file contents) if the sizes match. Combining this with --times will clone the timestamps across to the target tree.


6

rsync by default does not copy files based on their modification time and size. If you are not using -t in the command line, rsync will not preserve the modification times and the size + time check will of course fail. There is also a good chance that the timestamp precision of your differing filesystems is causing the problem - ext4 supports microseconds ...


6

Drive "resync" occurs if a drive swap happens and the RAID is being rebuilt by the system. In RAID 1, this means copying the entire drive. You can use the OS as normal, but it's will noticeably slow and during sync you do not have redundancy as the RAID is in a degraded state.


6

vdirsyncer https://github.com/untitaker/vdirsyncer by https://stackoverflow.com/users/1544347/markus-unterwaditzer can do this via CardDAV, two-way as well. I'd advise doing it via multiple runs against a local "vdir" rather than directly, and you can even implement git VCS for data security.


6

RAID is definitely not your answer. RAID is generally for drives being used in an array in the same physical location. If you want to sync file systems across continents, then you need to use a NAS product that has some form of live mirroring capability. Most enterprise-class SAN/NAS products have this feature. Otherwise, something like lsyncd or even a ...


5

Google just released a new password sync product called "Google Apps Password Sync (GAPS)" http://support.google.com/a/bin/answer.py?hl=en&answer=2611859&topic=2611858&ctx=topic


5

Those of you telling us all that "sync;sync;sync" has no purpose are revealing your age. Back in the good ol' days, before Unix was something for teenagers, we used to have to use TAPE for our streaming/backup needs. Often-times, we'd mount a tape-based filesystem to stream backups to, and so on. This one long thin band of magnetic plastic tape was all ...


Only top voted, non community-wiki answers of a minimum length are eligible