44
votes
Accepted
Faster rsync of huge directory which was not changed
Some unrelated points:
80K is a lot of files.
80,000 files in one directory? No operating system or app handles that situation very well by default. You just happen to notice this problem with rsync....
- 7,767
27
votes
Accepted
How can I *include* folders through Dropbox's selective sync feature in Linux, from the command line?
To exclude all files/folders:
cd to your dropbox folder (usually cd ~/Dropbox)
then type ~/bin/dropbox.py exclude add * This will exclude everything in your dropbox folder from syncing. (Be careful! ...
- 430
15
votes
Accepted
Does rsync change file content (dealing with hardlinks)?
With standard options, rsync will copy the new file assigning it a semi-random name, then it will rename the new file with the original name. In this process, no writes are directed at the original ...
- 45.5k
14
votes
Accepted
Access Denied when syncing between s3 buckets on different AWS accounts
The cause of your ListObjects error is that you assigned permission to access the contents of your bucket (arn:aws:s3:::bucket/*) but you did not give permissions to the bucket itself (arn:aws:s3:::...
- 841
8
votes
How to interrupt software raid resync?
Possible solution for this, took a bit to get into the details.
My system: CentOS 6.5
mdadm v3.3.2
Constant checks every week, wanted to pause one of them, RAID is clean, check was called via the /...
- 81
7
votes
Sync Servers with DRBD
As it was mentioned already, DRBD isn't greatest solution especially if you have a requirements for further scale.
I believe that the best solution on Linux would be Ceph Cluster - https://en....
- 8,928
6
votes
How to sync two CalDAV servers
vdirsyncer https://github.com/untitaker/vdirsyncer
by https://stackoverflow.com/users/1544347/markus-unterwaditzer
can do this via CardDAV, two-way as well.
I'd advise doing it via multiple runs ...
- 61
5
votes
Faster rsync of huge directory which was not changed
You can also try lsyncd, which will rsync only when changes are detected on the filesystem and just the changed subdirectories. I've been using it for directories with up to two million files on a ...
- 191
5
votes
Google Cloud Storage Fuse vs GlusterFS, pros, cons and costs
I have used GCS Fuse as a shared directory to a web server before, and I found that it's latency was unacceptably high and that it's transfer speed was a slow when serving files through nginx.
I don'...
- 153
5
votes
SYNC a folder on Ubuntu server with amazon S3 bucket automatically
You can also try Minio client aka mc, with its open source and S3 compatible API. You can use mc mirrorcommand to archive the directories.
Installing the minio client:
$ wget https://dl.minio.io/...
- 401
5
votes
Live file syncronization across multiple Linux servers with millions of files and directories
I'd seriously recommend using something like ZFS for the filesystem.
Built-in tools like the ZFS snapshot and ZFS send/receive allow you to take block-level snaps of the filesystem and ship it to a ...
- 196k
5
votes
Accepted
unison-fsmonitor doesn't detect any changes
The executable unison-fsmonitor is not meant to be run on its own, but is instead just a tool that Unison uses to watch your filesystem. To have Unison watch your filesystem for changes just run ...
- 257
5
votes
Accepted
Problems with Azure AD connect custom install
Azure AD connect custom installer is a tough enemy to fight with. And as you truly mentioned the errors is not clear and from time to time I have a filing that MS just don't want to waste a code on ...
- 4,877
5
votes
Sync Servers with DRBD
Yes, you can absolutely do that! See:
https://docs.linbit.com/doc/users-guide-83/s-three-nodes/
The only problem you need to have cluster-aware file system or do active-passive-passive mode with ...
- 12.7k
4
votes
Syncing user accounts across multiple servers
For smaller/simpler deployments, a project called Internet Account Replication (IAR) could help. It has a simple server-client architecture and synchronises the credentials using SSH.
- 2,334
4
votes
How to upload a directory recursively to an FTP server by just using ftp or lftp?
cd {local_dir}
lftp {server}
cd {remote_dir}
mput {local_dir}/*
This worked for me, many other attempts were failing. Once in lftp, more info available via:
help mput
- 141
4
votes
Accepted
Rsync over sshfs skips filenames starting with underscore
It turns out the culprit was in the sshfs flags. The noappledouble flag I was using to get rid of .DS_Store files was actually interfering with rsync's work.
From the sshfs Mount Options docs:
...
- 191
4
votes
Accepted
If two scheduled tasks are missed, in which order will they run next?
The order is not specified, it's an implementation detail. So you can't rely on the execution order, because it can be random, or change with Windows updates.
You have two options here:
Disable Run ...
- 6,510
3
votes
Alfresco Group Sync with Active Directory. Looking for a real groupID
Unfortunately Alfresco's repository is not designed to support namechanges for users and groups (authorities). You may change the displayname by using the Alfresco API but this is not what you are ...
- 311
3
votes
What can I really do with percona toolkit's pt-table-sync?
Answer to Question 1
pt-table-sync (togerther with pt-table-checksum) can be used to correct replication errors like data corruption, someone directly modifying data on the slave, server crashes, ...
- 1,518
3
votes
Rsync : copying over timestamps only
Use the source (e.g. /path/to/source) directory as reference for the touch command. Just cd to your target directory and do
find -type f -exec touch -r /path/to/source/{} {} \;
You cannot copy sub-...
- 31
3
votes
S3 upload delayed
While the first answer suggesting a blind timeout will likely work, might I suggest looking into S3 notification events. If you plan to scale out to more than two servers, using something like SNS or ...
- 39
3
votes
Large file synchronisation
Getting the initial copy is the hard part, perhaps an external drive and sneakernet is best. Then you should consider rsync for keeping it up-to-date. Rsync may even be a good option for the initial ...
- 115k
3
votes
Faster rsync of huge directory which was not changed
I think that 80k files is nothing extraordinary today.
My explanation for the problem lies in the way how rsync works: See here.
They say: While it is being built, each entry is transmitted to the ...
- 31
3
votes
Sync two Non-Enterprise SQL Servers
Always On Availability Groups allows active (queryable) secondaries, and may do what you want. It's available in standard edition in SQL Server 2016. It does, however, require Windows server ...
- 18.6k
3
votes
SYNC a folder on Ubuntu server with amazon S3 bucket automatically
In my point of view there is no perfect solution here, but you could try these 2 workarounds:
Mount on your Servers a Bucket as a File Storage System, with S3FS or like https://github.com/s3fs-fuse/...
- 111
3
votes
Live file syncronization across multiple Linux servers with millions of files and directories
If you cannot change the filesystem on the production server, I would put the files on another server and mount them with NFS. I would use Linux and ZFS if man-hours are inexpensive, maybe some kind ...
- 3,527
3
votes
Is it possible to change an MIT Kerberos password programmatically?
When a user changes their Active Directory password, I would like to sync it back into MIT Kerberos.
In its current form, this is too vague of a statement for it to really be answered in one shot.
...
- 55.3k
3
votes
Accepted
chrony does not log anything on my ec2 instance
I just verified chrony logging by doing the following:
created a default Ubuntu 18.04 LTS EC2 instance - AMI id: ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20180912 (ami-07a3bd4944eb120a0)...
- 4,017
Only top scored, non community-wiki answers of a minimum length are eligible