47

For a server I am hosting a website on I want to backup the data and settings to an S3 bucket. I found out that you can't directly use rsync to backup to an S3 bucket. Is there another way to achieve the following rsync command to backup the data to an S3 bucket? 

rsync -­av ­/Data /s3bucket

I also want to backup the mysql database on that server to the S3 bucket. What is the best way to achieve this?

Last question, if I managed to backup everything to the S3. What is the best way restore the server if it's crashed or in worst case completely wiped? Do I have to note the server settings myself and reconfigure the server or is there a way to also backup this?

Improve this question

3 Answers 3

Reset to default
56

To communicate to s3 you need to have 2 things

  1. IAM user credentials who has read-write access to s3 bucket.
  2. A client like aws-cli for bash, boto library for Python etc.

once you have both, you can transfer any file from your machine to s3 and from s3 to your machine. Below is the example for aws-cli.

To sync all files in a folder

aws s3 sync source_folder  s3://your_bucket_name/destination_folder/

To copy one file to s3

aws s3 cp source_file s3://your_bucket_name/destination_folder/

Just replace source & destination in the command to download any file from s3.

for more info follow aws docs

Improve this answer
3
  • 2
    Note: you will probably want to run "aws configure" first, in order to store your credentials, so that you don't have to enter them with each sync.
    – Tim Burch
    Apr 28, 2018 at 1:29
  • 3
    It's worth pointing out that rsync computes checksums, client side and server side, and then compares them to accurately decide if a file has changed. Whereas aws s3 sync does not - it uses file sizes and timestamps. Someone please correct me if I'm wrong ?
    – Lqueryvg
    Mar 8, 2019 at 14:38
  • I've also noticed that aws s3 sync transfers everything even if the file hasn't changed...where as rsync only transfers what changed.
    – chovy
    May 18, 2021 at 19:40
31

You can try rclone.

Setup (docs): 

rclone config

Sync (docs): 

rclone sync /my/local/folder s3service:bucket-name
Improve this answer
4
  • 2
    Sounds nice, but can you elaborate what the advantage is over using aws s3 sync ?
    – Kutzi
    Aug 2, 2019 at 9:30
  • It fulfills OP's requirement of using rsync. I haven't used s3 sync, or rclone with s3, so I don't know any more.
    – ki9
    Aug 4, 2019 at 2:01
  • 3
    @Kutzi the main advantages I see (when you want to use s3) are in the case you want to sync between different s3(-like) providers (awscli doesn't allow that). Furthermore, if you use a s3-like service not from AWS (minio, backblaze, digital ocean spaces, etc...), you need to specify the endpoint in the command, instead of declaring it the configuration file (there is an open issue for that).
    – Lucas Basquerotto
    Sep 2, 2020 at 18:53
  • 4
    Also rclone preserves timestamps, while aws s3 sync does not.
    – Markus
    Sep 25, 2020 at 14:18
3

aws-cli is great for syncing to S3 if you are not interested in permission.

Keep in mind that it does not keep permisión information nor other metadata that the files can have.

It is no well advice in the documentation, and thet you have surprises when restoring.

May be it is not important to sync your collection of music of photos, but it is of crucial importance if you are using it to backups your system.

There is another program s3cmd that keeps this data and can do a similar job. It is slower. But I have not tried it.

I was told that the problem is that you can keep in high costs as in S3 you do not only pay per MB saved, but for requests and rsync makes a lot of them.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.