Hot answers tagged

20

This is possible with s3cmd 1.5+ (link): $ mysqldump ... | s3cmd put - s3://bucket/file-name.sql


8

It looks like you have the syntax wrong (although it's possible you just typed it into this thing wrong). From the S3 Tools documentation it looks like you want something like this: s3cmd del s3://BUCKET/OBJECT and s3cmd del -r s3://BUCKET/FOLDER Typically you need to use the -r, or recursive, flag to delete folders. If that doesn't work, check that ...


5

Check your bucket policy by going to bucket, then click on Properties and Edit Bucket Policy. If you have something like this: { "Sid": "Stmt1391783519913", "Effect": "Allow", "Principal": { "AWS": "*" }, "Action": [ "s3:GetObject", ], "Resource": "arn:aws:s3:::bucket/*" ...


4

You can use Minio client aka mc, its Open Source & compatible with AWS S3. Installing Minio client on Linux $ wget https://dl.minio.io/client/mc/release/linux-amd64/mc $ chmod 755 mc $ ./mc --help Adding AWS S3 credentials $ ./mc config host add mys3one https://s3.amazonaws.com BKIKJAA5BMMU2RHO6Izz V7f1CwQqAcwo80UEIJEjc5gVQUSSx5ohQ9GSrr12 $ ./mc ...


3

you simply have to authorize your IAM user to access to the buvket, within the S3 bucket policy, like so: { "Version": "2012-10-17", "Statement": [ { "Sid": "whatever", "Effect": "Allow", "Principal": { "AWS": [ "<ARN OF YOUR IAM USER>" ] ...


3

I've fixed this on github.com/s3tools/s3cmd master branch. It was introduced in the new signature v4 code, but does not need to be a warning(), but instead lowered to info() which won't output on normal runs.


3

To send a mail via mail (and using stdin as body), use the following syntax: echo "test body" | mail -s 'test subject' reciever@example.com For more information take a look at the Manpage man mail.


2

It's probably not running because it's unable to find the binary. Without any error log though, that's just an assumption I'm making. Try adjusting the s3cmd part in your script to use its full path. I'm not sure how you installed s3cmd (either via yum or manually), but you can find out its full path by running which s3cmd.


2

This is a bad way to do backups. You should be separating your OS configuration from your valuable data. None of your permissions will be transferred, which in the Linux world are a necessity if you're planning on restoring backups (which you should be - backups without verified restorations are pointless). Firstly, you can synchronise your valuable ...


2

I'd say it's most likely that either rvmsudo or s3cmd need extra environment variables to run successfully, which aren't set in the context of logrotate, but which are in your interactive shell session. I wouldn't be surprised if rvmsudo isn't available in the restrictive PATH which cron sets.


2

You can use Minio client aka mc, It is open source and compatible with AWS S3. Minio client has resume session command which will upload the object where it was left in last disconnect. I have created a Youtube video showing same Hope it helps. Disclaimer : I work for Minio


2

You are executing a new cmd for every file which is terribly expensive. This is not new: There are some disadvantages to using CMD.exe with FORFILES, a new process will be created and destroyed for every file that FORFILES processes, so if you loop through 1000 files, then 1000 copies of CMD.exe will be opened and closed, this will affect ...


1

Is it possible that you have the Object Lock (WORM) feature enabled? That would block delete operations.


1

Metadata (data about data) is stored separately from the object, and will not corrupt the object. It's similar to the filename and the modified date. You can use object versioning to keep versions to help ensure this. S3 Metadata documentation can be found here. I can't speak to the reliability of s3cmd. Instead of coping each file natively to S3 you should ...


1

I don't believe the S3 API lets you submit multiple files in a single API call, but you could look into concurrency options for the client you are using. A good starting point would be the official AWS Command Line Interface (CLI) which has some S3 configuration values which allow you to adjust concurrency for aws s3 CLI transfer commands including cp, ...


1

You will have to write a script or program to retrieve files from MongoDB and copy them to S3. Most probably files in MongoDB are stored in BSON format, so they require to be converted first. As you have noted that you are using Parse - try these utils to extract images. https://github.com/parse-server-modules/parse-files-utils


1

In addition to the snapshot, you can also use the scp command. Depending on your environment, this may be problematic with large files, it's good to know the options. To transfer the file from the second instance (ec2-2) to first instance (ec2-1), run scp on ec2-1: scp -i /Path-To-Key-File-for-ec2-2/key.pem ec2-user@Elastic-IP-of-ec2-2:/path/filename your/...


1

The problem is that bash for loop is using $IFS as field separator, which defaults to space. So each part of filename separated by a space will come as a different iteration in loop. IFS can be changed to match newline instead of space and allow correct processing of find output. For example: # save and set IFS OIFS=$IFS IFS=$'\n' # do your processing ...


1

Accounts use different credentials. There's no way to sync from one account to another and provide separate credentials. The only way to do that is to allow public access to the second bucket with a temporary account. If that is not an option, then your best bet is to: s3 sync s3://acct1/bucket . --profile acct1 s3 sync . s3://acct2/bucket --profile acct2


1

The up to date answer to this question is to use Google Transfer Service: What is Storage Transfer Service? Storage Transfer Service transfers data from an online data source to a data sink. Your data source can be an Amazon Simple Storage Service (Amazon S3) bucket, an HTTP/HTTPS location, or a Cloud Storage bucket. Your data sink (the ...


1

If you are synchronizing a large amount of data between clouds you might consider setting up a Google Compute Engine account and running gsutil there. Since cross-provider gsutil data transfers flow through the machine where gsutil is running, doing this can make your transfer run significantly faster than running gsutil on your local workstation. Reference:...


1

Does your proxy accept wildcards? You'll want to whitelist *.s3.amazonaws.com at least, and most likely *.s3-*.amazonaws.com if you point at region-specific endpoints.


1

Just use the AWS cli: $ aws s3 cp --recursive /path/to/local/files/* s3://bucket/ Use a set of IAM credentials that has write-only access to the bucket, so that you can be ensured that no files are ever copied back to the source system.


1

It looks like s3cmd can accept input from stdin at least according to the resolution of this bug on 2/6/2014. If your s3cmd is newer than that you should be able to do: tar -czvf - ${BACKUP_DEST} | s3cmd --reduced-redundancy put - s3://MY-S3-BUCKET/`date +\%G-\%m-\%d`_db.tar.gz --no-encrypt Most utilities use - as a filename to indicate writing to stdout ...


1

This appear to now be possible. With s3cmd v1.6.1: curl -v "http://remote-server/file.mp4" | s3cmd [-c .s3cfg-aws] put - [-P] s3://my-bucket/[folder/]filename.mp4


1

You are missing the actual file you want to backup to start. s3cmd put /backup_dir/somefile.sql.gz s3://bucket/sql/ s3cmd takes two basic arguments, the file, and the bucket to backup too. Secondly, I can't take credit for the following, but its basically doing what you want with an intermediate script. Basically, create a bak.sh file with the following, ...


1

After you make a file private in S3, you may still be able to download it your browser has cached the file. As soon as you make the file private, it will return a 403 in any other browser (or private/incognito window), but may still be accessible from the original browser for a certain period of time. If you want to confirm that the file is private, try ...


1

Until recently the default timeout was 10 seconds, so unless you have the newest version of the s3cmd, this is probably the case. But this is fixable. Location your .s3cfg file, which should be in your home directory. Edit the .s3cfg file to add the line: socket_timeout = 300 This should fix the issue.


1

Of course any upload in the terrabyte range is going to take some time, how slow is your "slow" uplink? A 1 terrabyte upload over a 10Mbit uplink will take 1024*1024 = 1048576 seconds = 290 hours = 12 full days. I would take a look at Amazon's bulk upload by shipping a NAS appliance Take a look at http://aws.amazon.com/importexport/


1

http://docs.aws.amazon.com/AWSAndroidSDK/latest/javadoc/com/amazonaws/services/s3/model/AccessControlList.html Every bucket and object in Amazon S3 has an owner, the user that created the bucket or object. The owner of a bucket or object cannot be changed. However, if the object is overwritten by another user (deleted and rewritten), the new object will ...


Only top voted, non community-wiki answers of a minimum length are eligible