Questions tagged [s3cmd]

s3cmd is a simple command-line interface to storing and retrieving data to/from Amazon's S3 cloud storage service.

Filter by
Sorted by
Tagged with
0 votes
0 answers
83 views

s3cmd multipart chunk-size memory management

I was wondering how to predict how much memory s3cmd is going to use to upload a given file. I'm concerned that for large files it could use too much system memory. And if I increase the chunk size, ...
9 votes
3 answers
23k views

s3cmd: Deleting files or folder does not work

s3cmd s3://MYBUCKET/FOLDER does not work. (folder is empty) s3cmd s3://MYBUCKET/FILE does not work. Output of both commands is "File FILE sucessfully deleted", yet the file remains on the bucket. ...
1 vote
1 answer
2k views

s3cmd error on sync: S3 error: 400 (InvalidRequest)

I'm trying to configure s3cmd to backup some directories on my server to my AWS S3 bucket. My problem is that when I do the sync command: s3cmd sync -r -p --no-delete-removed /path/to/dir s3://...
2 votes
1 answer
2k views

How backup files with logrotate and Amazon S3?

I would like to create a new standard of two things: how long would be generated and be rotated the logs generated by my applications ? how to transfer the logs to Amazon S3, as a backup server ?. ...
2 votes
0 answers
29 views

Swift Stack Proxy Server Reboot

I'm fairly new to Object Storage and have been thrown into support for an old product we no longer have a support contract for and therefore can't seek help from the vendor. I have a 13-node cluster, ...
2 votes
2 answers
3k views

How to upload a large file using aws commandline when connection may be unreliable?

I've been having trouble uploading a large (800MB) file to s3, using the aws commandline tool. The first attempt completed (after many hours) but was not visible, and I was advised (here) that it has ...
2 votes
2 answers
818 views

Logrotate script not uploading to s3

I am trying to use logrotate to upload log files from my rails and nginx commands once a day. When I run the logrotate command manually the files successfully upload, but it never does it on its own. ...
0 votes
1 answer
902 views

cannot Delete objects on s3 ceph bucket

I have a ceph cluster and running a few s3 bucket on it, 'gitlab-s3-api' user has full permission on everything on ( users=;buckets=;metadata=;usage=;zone=* ) but very oddly it cannot delete any file ...
9 votes
5 answers
12k views

how to pipe a mysql dump to s3cmd

I want to transfer a mysql dump, compressed, to s3. I tried: mysqldump -u root -ppassword -all-databases | gzip -9 | s3cmd put s3://bucket/sql/databases.sql.gz but then I get: ERROR: Not enough ...
1 vote
1 answer
761 views

Gitlab not backing up to s3. Where can I find logs?

I am currently trying to migrate our old sameersbn/gitlab:12.1.6 installation to a new Gitlab instance set up with the helm chart. The easiest way seems to enable the s3 backup capability on the old ...
0 votes
1 answer
747 views

Where is the AWS S3 metadata stored written by s3cmd? [closed]

I was looking for a AWS S3 solution for backing up a Linux file system so that it would preserve the timestamp and permissions. It appears that s3cmd allows this by writing this information to the ...
4 votes
2 answers
5k views

Amazon S3 - private file is still downloadable for everyone?

I have some secret files which I want to restrict anonymous user from viewing/downloading. I have tried to make it private by running something like: s3cmd setacl --acl-private s3://bucket/some/path/*...
0 votes
1 answer
246 views

Transfer image files from MongoDB to S3

I have a NodeJS app that stores all of its data, even images, to MongoDB. Now I want to use an S3 to store my app's files (images, 43gb) and so, I need to transfer there all files that I have in my ...
0 votes
1 answer
4k views

How to upload multiple files from directory to S3?

I have a directory on an Ubuntu, with 340K images, 45GB of total size! Is there an efficient way to transfer them all to an S3 of DigitalOcean? I thought of using s3cmd put or s3cmd sync but I'm ...
1 vote
0 answers
74 views

Spectra S3 Java CLI (DS3) difficulty verifying checksum

We have a SpectraLogic BlackPearl+T950 which uses S3 style commands to put/get/sync etc. To access the Spectra flavour of S3 commands we use a Java CLI client to perform these operations. Supposedly ...
0 votes
1 answer
539 views

run s3cmd on Synology NAS

I'm trying to use s3cmd on a Synology NAS with DSM 6.0, and I'm getting this error. I've installed the python-magic libs but still. [root@OFFICE_01 arc]# s3cmd --version !!!!!!!!!!!!!!!!!!!!!!!!!!!!!...
2 votes
2 answers
4k views

How best way to rsync from aws s3 to gcloud?

I must execute a rsync ( not only copy or move ) from all my buckets from s3 to google cloud. I need create the same infraestructure of s3 to google. I have like 2Tera in s3, and all days I receive ...
1 vote
1 answer
2k views

How to copy more than 300GB files one AWS EC2 server to another EC2

I need to copy more than 300 GB files (most them are images) one EC2 instance to another EC2 instance, what will be the best method for doing this? EBS Snapshot will be not work in my situation ...
0 votes
1 answer
1k views

s3cmd restore recursive not working right

I am running the following command to restore the data from Amazon S3 to the web server. s3cmd restore --recursive s3://my-bucket/Weekly-Sunday/dbexports/ and getting this error: Restore is not ...
1 vote
1 answer
151 views

Ways to extract files information for large data

I am trying to recursively extract some file information in my file server. I was able to run the commands below at my own laptop. When I run them on my file server, which is going thru 4TB of data, ...
1 vote
1 answer
3k views

Setting cache for already uploaded amazon s3 files

In amazon s3 for our site through s3cmd tool we uploaded around 60GB of files including mp3, jpg to a bucket. But now when we checked site slowness files are not caching and further checking we came ...
0 votes
1 answer
110 views

can I attach IAM role to instance after launching it?

Getting error : root@omp_affiliate_backend ~]# s3cmd get s3://admarvel-noc/all-users/nrpe-plugins.tar.gz . s3://admarvel-noc/all-users/nrpe-plugins.tar.gz -> ./nrpe-plugins.tar.gz [1 of 1] ERROR:...
0 votes
1 answer
2k views

How to use s3cmd put when there are spaces in the filename?

for file in $(find . -type f -mmin -240) do filename=$(echo "$file" | cut -d'.' --complement -s -f1) s3cmd put $file s3://${bucketname}/$filename done I am trying to ...
4 votes
0 answers
1k views

S3 restoration using s3api get-object is not working in aws china region

I have set up a daily backup script in my aws china instance which uploads my required files to be backed-up to s3 bucket. I have a restoration script which uses s3api to restore the objects to the ...
2 votes
3 answers
5k views

How to transfer data between s3 buckets of the different aws account using command line

How to transfer data between s3 buckets of the different aws account using s3cmd ? command will be like this: s3cmd rsync s3://acc1_bucket/folder/ s3://acc2_bucket/folder --recursive But then how ...
3 votes
0 answers
524 views

Solution for file data backup, versioning and replication

We have a growing set of data files (.wav files, image files etc) which are data, i.e not part of the application code - uploaded and modified by users. The number of files is in the 1000s and the ...
1 vote
1 answer
341 views

Getting error : Malformed URL error in ec2

I am getting the error Malformed URL ec2-describe-instances instance_id xxx Malformed URL: 'ec2.ap-southeast-1.amazonaws.com' However, I am able to get access to the bucket through command line: ...
0 votes
1 answer
113 views

How can I control a gradual file upload with cron or similar? [closed]

I have several terabytes of data I need to back up to S3/Glacier over a slow link. I want to upload from, say 9pm to 9am and resume from where it left off. Is there free software that would do that ...
1 vote
1 answer
484 views

s3 service using s3cmd tool through URL based firewalls

I am behind a URL based application firewall and currently using s3cmd tool to access amazon's s3 service. Doing a bit of searching has asked me to whitelist s3.amazonaws.com but I see that s3cmd uses ...
4 votes
1 answer
4k views

Sync files from local filesystem with S3 - one way

I have a process that generates log files to the local filesystem, and files that are more than 2 weeks old are automatically purged. I want to use the aws command line tools to sync my folder with ...
1 vote
1 answer
765 views

Redirect warning when running s3cmd sync

When running s3cmd I get a redirect warning as follows: WARNING: Redirected to: xyz.s3-eu-west-1.amazonaws.com The script that produces the warning is this: s3cmd sync /var/backups/db/ s3://xyz/...
2 votes
1 answer
8k views

mail: Cannot parse address

This is the command I use to tar my backup files, write to stdout and then upload everything to Amazon S3 with an email notification. tar -czvf - --exclude-caches /var/www | s3cmd --reduced-...
2 votes
1 answer
4k views

How to copy all s3 buckets to another AWS account?

I'm copying all EC2 instances to another AWS account (creating the AMIs, sharing them with that account and running the instances from AMIs there). I also need to move my S3 buckets since the ...
2 votes
1 answer
959 views

s3cmd: on-the-fly backup

To do backups I created a script that makes an archive of all the folders I need to backup, sends it to S3 (through s3cmd) and then deletes it after upload has been completed. I'm looking for a way ...
0 votes
1 answer
913 views

How continue sync with S3CMD after broken connection?

I'm using s3cmd to backup some logs into Amazon S3 buckets. Which is a great tool. But I notice that, very often, my sync's are broken, they just cancel and try to upload again, until they can upload ...
-1 votes
1 answer
1k views

Python error when s3cmd is executed within bash script

I'm creating a bash script of which will run on a cron, it's purpose is to backup a directory to Amazon S3 using the s3cmd command. I get the following error when I run the script: /usr/bin/env: ...
0 votes
0 answers
3k views

s3cmd 403 Access Denied error on put but not ls

I can access s3 using s3cmd on an ec2 instance to do s3cmd ls s3://myc bucketname but not using 'put' to add a file. It gives a 403 access denied error. I am using IAM and have group setup to access ...
1 vote
1 answer
307 views

Saving httpd logfiles to S3 on shutdown using s3cmd

I have an auto-scaling group on Amazon Web Services (AWS). I want to preserve the logfiles on an instance (Amazon Linux AMI) when the group scales down. My current idea is to use the s3cmd tool in a ...
5 votes
1 answer
3k views

Linux server sync to an Amazon S3 bucket

I am looking for a stable solution to replace a classic server backup to another server using rsync. I have to sync a whole filesystem (more than 1Tb) to Amazon S3. Where am I? Solution 1: I mapped ...
4 votes
1 answer
8k views

s3cmd with --delete-removed

I'm currently writing a script to sync files in s3 buckets with s3cmd. I check the document and it says: s3cmd sync LOCAL_DIR s3://BUCKET[/PREFIX] or s3://BUCKET[/PREFIX] LOCAL_DIR also I find a ...
4 votes
1 answer
4k views

s3cmd put only files that do not exist in destination bucket

I've been using s3cmd to sync local files on our server with the S3 bucket. There are over 200,000 files on the local server. Can't seem to find any documentation on the site that explains how to ...
1 vote
1 answer
2k views

One Way Sync of a Bucket With Local Directory

I have a local directory that I would like to synchronize with an S3 bucket. I have two specific requirements: If local file is the same as the remote, do not re-transfer it to the bucket. If some ...
1 vote
3 answers
536 views

Backing up data (including mysqldumps) to S3

We have a web app on a number of servers and we want to add an additional layer of redundancy by backing up the key data to S3. The key data is the MySQL database and a folder containing dynamically ...