Tag Info

Hot answers tagged

7

If you can trust the filesystem last-modified timestamps, you can speed things up by combining Rsync with the UNIX/Linux 'find' utility. 'find' can assemble a list of all files that show last-modified times within the past day, and then pipe ONLY that shortened list of files/directories to Rsync. This is much faster than having Rsync compare the metadata of ...


6

Try unison, it was specifically designed to solve this problem by keeping the change lists (building file list), locally to each server, speeding up the time to calculate the delta, and the reduce amount that is sent across the wire afterwards.


3

We use a wrapper around Unison that, on failure, creates a file in /tmp based on the pathname of the directory to sync. Then, there is a service check in Nagios that checks for the presence of these files, and sends up an alert if found. This way there is the verbose Unison log for more information, a quick way to know if something went wrong, and ...


3

Many of your problems are handled by configuration management tools like Puppet, Chef or cfengine. It works differently than you envision it, but it's the goal that matters, right? With e.g. Puppet, you install the basic OS (might be from an image), with a Puppet client and then Puppet handles the installation/configuraton of the software. To a degree, ...


2

I solved the problem, and it was the HOME=/ in crontab, it should be the path where your .unison exists So my crontab content now is SHELL=/bin/bash PATH=/sbin:/bin:/usr/sbin:/usr/bin MAILTO=root HOME=/home/user1/ # For details see man 4 crontabs # Example of job definition: # .---------------- minute (0 - 59) # | .------------- hour (0 - 23) # | | ...


2

Why are you running unison as root? Why not use sudo -u bitnami unison instead? Then the file you mentioned will automatically belong to bitnami, and you have the added benefit of not exposing your server to accidental file deletions or other mishaps with root privileges.


2

I wouldn't use Unison in this case, since its main purpose is the two way sync. Plain old rsync will do the job here. This will synchronise files from the source to the destination without deleting files on the destination (although it will overwrite files on the destination to be the same as on the source). rsync --recursive --times --verbose source/ ...


2

There is no global config file, but a shortcut: Use include file to source other config files. Now you can save the above lines to common and add include common at the beginning of each of your config files. There is a more sophisticated setup in the official manual.


2

One idea would be to use the stdout generated by Unison while it is running. There is some junk in stdout that Unison uses to create a dynamic effect in the terminal while it is "looking for changes." This junk can be removed fairly easily by deleting every line containing a Carriage Return (CR) character (in vim this would be something like ...


2

GlusterFS? You would have both web servers as gluster servers (say, exporting /srv/export), and also acting as gluster clients, mounting the same exported volume (say, moutning /srv/share). File operations would be on /srv/share and would propagate to all boxes in the cluster.


2

two easy answers, (and the same you've already mentioned): Unison if you want something that 'just works'. Has a nice GUI and you can easily define a task there and call from command line, a script, cron, whatever. I use it to synchronize both my laptop and my wife's with our respective user directories on the desktop. It's really that easy. rsync if ...


2

Unison is a 2-ways synchronization tool. If you only want to synchronize in one-way, take a look at --force, --prefer option or use rsync for alternative.


2

How about DRBD with its dual-primary mode and a cluster file system such as OCFS2 or GFS? Might work surprisingly well especially if your directory tree is not containing a huge amount (say, millions or more) of frequently changing small files.


2

http://oss.linbit.com/csync2/ is designed for this sort of thing, I'd give that a try.


2

You should seriously consider DRBD. DRBD is a RAID1 over TCP-IP software. It will replicate a block device over any link in real time. Every time you modify a HD block, it will replicate it. It is FS-agnostic therefore you can put any FS you want on top of it. It will also work hand-in-hand with heartbeat which will allow you to start the hot spare as soon ...


1

I critically need the sync to be immediate as the user needs to display the thumbnail straight away on a web page So you've no control over server affinity in your load balancing? Time to get a new load balancer. I looked at unison and lsync, but there doesn't seem to be a rpm available from the centOS5.5 repo for either. Previously I did this by ...


1

I would use a backup program like Bacula to do this. It keeps a database of your backups so it don't need to compare file sizes and time stamps against already backed up files like rsync. If you want to use rsync, you could use find to find files changed after your last backup, write this to a file and read the file names to copy from this file with ...


1

Whats wrong with the normal rsync output? /tmp # ls -l ra total 8 -rw-r--r-- 1 phemmer unix 5 2011/02/09-16:42:49 adsf -rw-r--r-- 1 phemmer unix 29 2011/02/09-16:49:41 date /tmp # date > ra/date /tmp # rsync -Haxv ra rb building file list ... done ra/date sent 180 bytes received 42 bytes 444.00 bytes/sec total size is 34 speedup is 0.15 Just ...


1

Taking the -z out of the rsync command which is no compression made the "receiving file list" go so much faster and we had to transfer about 500 GB. Before it took a day with the -z switch.


1

If you're using the -z switch on rsync, try running without it. For some reason I've seen this speed up even the initial enumeration of files.


1

You should be able to setup ssh login with keys as you have stated. Make sure your sshd_config allows root logins PermitRootLogin without-password Then copy the generated public-key to /root/.ssh/authorized_keys. You should be able to login now, just tell unison to use the ssh key you generated.


1

you can try using this -> " . like ip:"path1 with whitespace" /path2


1

Why not to point Apache to your shared folder on VM in the first place? This is as simple as to add/change one file for your virtualhost (I believe you've done this anyway). This will let you not to wait 5-10 seconds while your server document root is being updated. However what you actually ask is a periodically running some command, so in the end you ...


1

I think GlusterFS can solve your problem. It is a distributed filesystem that operates in userspace, so you don't have to change your current filesystem. With a gluster replicated volume you have a single virtual FS, which you can mount on your servers using FUSE module or NFS. This virtual FS is immediately synced between the nodes on your cluster.


1

Looks like you have to place the unison options after defining the root directories: Formatted like this: link to unison manual (RTFM!) unison root1 root2 [options] So my code should be: ${_unison} -batch "${p}" "ssh://${r}/${p}" -times -owner -group Once the options are placed here, the script executes without any errors.


1

You can use tools rsync or unison, but you will not get a realtime sync. There should be some delay until new/updated files get synced even if you run such a tool every minute. You can implement it if you think it is acceptable to have a small delay. If you want to get changes synced in realtime, you need to use a tool like NFS/glusterfs, or DRBD. The main ...


1

This asymmetric architectures doubles your chance of the service failing - and it's not very good for performance. i use rsync ( as explained above ). I didn't see an explanation of how you use rsync. In order to sync the application code Deployment of application code should be a managed process. How you manage replication of content uploaded ...


1

OK, I've found the culprit: the unison does ignore the fastcheck option for xls and mpp files and always performs the full comparison for them. It does so, because Excel is in habit of modifying the xls files without changing the last modified date. Unfortunately for us, xls makes files about 20% of the whole volume of the documents. Editing the ...


1

I found my answer just after posting, but I would like to keep it here for other people looking into the same problem: In those cases, you should use "-prefer" instead of "-force" : Including the preference -prefer root causes Unison always to resolve conflicts in favor of root, rather than asking for guidance from the user. (The syntax of root ...


1

"Connection reset by peer" means that the remote end--the SSH server, in this case--closed the TCP stream abnormally. One way that can happen is if the remote process crashes. "ssh_exchange_identification" means that the server and client were exchanging banner strings which identify the SSH software on each end of the connection. The client was waiting for ...



Only top voted, non community-wiki answers of a minimum length are eligible