SYNC a folder on Ubuntu server with amazon S3 bucket automatically

Question

I have an app running on Digital Ocean server and there are two folders which are being updated with user uploaded images.

Is it possible to transfer the new uploads from the server folder to s3 bucket automatically.

I have already configured AWS CLI on ubuntu server.

thanks

Answer 1

You can also try Minio client aka mc, with its open source and S3 compatible API. You can use mc mirrorcommand to archive the directories.

Installing the minio client:

 $ wget https://dl.minio.io/client/mc/release/linux-amd64/mc 
 $ chmod 755 mc
 $ ./mc --help

Configuring mc for Amazon S3:

$ ./mc config host add <ALIAS> <YOUR-S3-ENDPOINT> <YOUR-ACCESS-KEY> <YOUR-SECRET-KEY>

Example

$ ./mc config host add mys3 https://s3.amazonaws.com BKIKJAA5BMMU2RHO6IBB V7f1CwQqAcwo80UEIJEjc5gVQUSSx5ohQ9GSrr12

Copying the local folder to AWS S3:

$ ./mc mb mys3/mys3baucket
$ ./mc mirror mylocaldir/ mys3/mys3baucket

The first command creates a bucket named "mys3baucket". The second command mirrors the local directory to the S3 bucket by name. This can be easily set in cron for a periodic mirror.

mc implements the following commands:

  ls        List files and folders.
  mb        Make a bucket or folder.
  cat       Display contents of a file.
  pipe      Write contents of stdin to one or more targets. When no target is specified, it writes to stdout.
  share     Generate URL for sharing.
  cp        Copy one or more objects to a target.
  mirror    Mirror folders recursively from a single source to many destinations.
  diff      Compute differences between two folders.
  rm        Remove file or bucket [WARNING: Use with care].
  access    Manage bucket access permissions.
  session   Manage saved sessions of cp and mirror operations.
  config    Manage configuration file.
  update    Check for a new software update.
  version   Print version.

Hope it helps.

Disclaimer: I work for Minio

Answer 2

In my point of view there is no perfect solution here, but you could try these 2 workarounds:

• Mount on your Servers a Bucket as a File Storage System, with S3FS or like https://github.com/s3fs-fuse/s3fs-fuse.
• Schedule s3cmd sync to synchronise your folder with the S3 bucket. http://s3tools.org/s3cmd-sync.

This works like a rsync of sorts.

A better implementation for your web application would be to directly upload and retrieve to the S3 bucket. S3 is designed specifically for this use and offers nice tricks like form based validation.