Questions tagged [s3cmd]

s3cmd is a simple command-line interface to storing and retrieving data to/from Amazon's S3 cloud storage service.

Filter by
Sorted by
Tagged with
0
votes
1answer
644 views

s3cmd not preserving the Linux file permissions

I'm using s3cmd version 2.0.2 on CentOS. I'm using the --preserve command doing a put to AWS S3, and adding the --preserve command when doing the get command. What was a 0644 file permissions when ...
0
votes
0answers
17 views

Gitlab not backing up to s3. Where can I find logs?

I am currently trying to migrate our old sameersbn/gitlab:12.1.6 installation to a new Gitlab instance set up with the helm chart. The easiest way seems to enable the s3 backup capability on the old ...
0
votes
1answer
64 views

Where is the AWS S3 metadata stored written by s3cmd?

I was looking for a AWS S3 solution for backing up a Linux file system so that it would preserve the timestamp and permissions. It appears that s3cmd allows this by writing this information to the ...
0
votes
1answer
68 views

How to upload multiple files from directory to S3?

I have a directory on an Ubuntu, with 340K images, 45GB of total size! Is there an efficient way to transfer them all to an S3 of DigitalOcean? I thought of using s3cmd put or s3cmd sync but I'm ...
0
votes
1answer
56 views

Transfer image files from MongoDB to S3

I have a NodeJS app that stores all of its data, even images, to MongoDB. Now I want to use an S3 to store my app's files (images, 43gb) and so, I need to transfer there all files that I have in my ...
1
vote
0answers
16 views

Spectra S3 Java CLI (DS3) difficulty verifying checksum

We have a SpectraLogic BlackPearl+T950 which uses S3 style commands to put/get/sync etc. To access the Spectra flavour of S3 commands we use a Java CLI client to perform these operations. Supposedly ...
0
votes
0answers
15 views

Excluding the '#recycle' directory from s3cmd upload

I'm using s3cmd on a Synology NAS. I built an exclusion file including #recycle/* and the --exclude-from=/path/to/exc/file option but it doesn't work. I already tried this: "#recycle/*" "*/#...
0
votes
1answer
132 views

run s3cmd on Synology NAS

I'm trying to use s3cmd on a Synology NAS with DSM 6.0, and I'm getting this error. I've installed the python-magic libs but still. [root@OFFICE_01 arc]# s3cmd --version !!!!!!!!!!!!!!!!!!!!!!!!!!!!!...
0
votes
1answer
481 views

How to copy more than 300GB files one AWS EC2 server to another EC2

I need to copy more than 300 GB files (most them are images) one EC2 instance to another EC2 instance, what will be the best method for doing this? EBS Snapshot will be not work in my situation ...
1
vote
1answer
111 views

Ways to extract files information for large data

I am trying to recursively extract some file information in my file server. I was able to run the commands below at my own laptop. When I run them on my file server, which is going thru 4TB of data, ...
0
votes
1answer
104 views

can I attach IAM role to instance after launching it?

Getting error : root@omp_affiliate_backend ~]# s3cmd get s3://admarvel-noc/all-users/nrpe-plugins.tar.gz . s3://admarvel-noc/all-users/nrpe-plugins.tar.gz -> ./nrpe-plugins.tar.gz [1 of 1] ERROR:...
0
votes
1answer
940 views

How to use s3cmd put when there are spaces in the filename?

for file in $(find . -type f -mmin -240) do filename=$(echo "$file" | cut -d'.' --complement -s -f1) s3cmd put $file s3://${bucketname}/$filename done I am trying to ...
3
votes
0answers
525 views

S3 restoration using s3api get-object is not working in aws china region

I have set up a daily backup script in my aws china instance which uploads my required files to be backed-up to s3 bucket. I have a restoration script which uses s3api to restore the objects to the ...
1
vote
1answer
1k views

s3cmd error on sync: S3 error: 400 (InvalidRequest)

I'm trying to configure s3cmd to backup some directories on my server to my AWS S3 bucket. My problem is that when I do the sync command: s3cmd sync -r -p --no-delete-removed /path/to/dir s3://...
0
votes
1answer
734 views

s3cmd restore recursive not working right

I am running the following command to restore the data from Amazon S3 to the web server. s3cmd restore --recursive s3://my-bucket/Weekly-Sunday/dbexports/ and getting this error: Restore is not ...
2
votes
1answer
2k views

How to upload a large file using aws commandline when connection may be unreliable?

I've been having trouble uploading a large (800MB) file to s3, using the aws commandline tool. The first attempt completed (after many hours) but was not visible, and I was advised (here) that it has ...
1
vote
3answers
3k views

How to transfer data between s3 buckets of the different aws account using command line

How to transfer data between s3 buckets of the different aws account using s3cmd ? command will be like this: s3cmd rsync s3://acc1_bucket/folder/ s3://acc2_bucket/folder --recursive But then how ...
1
vote
1answer
257 views

Getting error : Malformed URL error in ec2

I am getting the error Malformed URL ec2-describe-instances instance_id xxx Malformed URL: 'ec2.ap-southeast-1.amazonaws.com' However, I am able to get access to the bucket through command line: ...
1
vote
1answer
556 views

Logrotate script not uploading to s3

I am trying to use logrotate to upload log files from my rails and nginx commands once a day. When I run the logrotate command manually the files successfully upload, but it never does it on its own. ...
2
votes
2answers
2k views

How best way to rsync from aws s3 to gcloud?

I must execute a rsync ( not only copy or move ) from all my buckets from s3 to google cloud. I need create the same infraestructure of s3 to google. I have like 2Tera in s3, and all days I receive ...
1
vote
1answer
400 views

s3 service using s3cmd tool through URL based firewalls

I am behind a URL based application firewall and currently using s3cmd tool to access amazon's s3 service. Doing a bit of searching has asked me to whitelist s3.amazonaws.com but I see that s3cmd uses ...
3
votes
0answers
477 views

Solution for file data backup, versioning and replication

We have a growing set of data files (.wav files, image files etc) which are data, i.e not part of the application code - uploaded and modified by users. The number of files is in the 1000s and the ...
3
votes
1answer
2k views

Sync files from local filesystem with S3 - one way

I have a process that generates log files to the local filesystem, and files that are more than 2 weeks old are automatically purged. I want to use the aws command line tools to sync my folder with ...
1
vote
1answer
654 views

Redirect warning when running s3cmd sync

When running s3cmd I get a redirect warning as follows: WARNING: Redirected to: xyz.s3-eu-west-1.amazonaws.com The script that produces the warning is this: s3cmd sync /var/backups/db/ s3://xyz/...
1
vote
1answer
2k views

Setting cache for already uploaded amazon s3 files

In amazon s3 for our site through s3cmd tool we uploaded around 60GB of files including mp3, jpg to a bucket. But now when we checked site slowness files are not caching and further checking we came ...
1
vote
1answer
4k views

mail: Cannot parse address

This is the command I use to tar my backup files, write to stdout and then upload everything to Amazon S3 with an email notification. tar -czvf - --exclude-caches /var/www | s3cmd --reduced-...
2
votes
1answer
795 views

s3cmd: on-the-fly backup

To do backups I created a script that makes an archive of all the folders I need to backup, sends it to S3 (through s3cmd) and then deletes it after upload has been completed. I'm looking for a way ...
2
votes
2answers
12k views

s3cmd: Deleting files or folder does not work

s3cmd s3://MYBUCKET/FOLDER does not work. (folder is empty) s3cmd s3://MYBUCKET/FILE does not work. Output of both commands is "File FILE sucessfully deleted", yet the file remains on the bucket. ...
6
votes
4answers
8k views

how to pipe a mysql dump to s3cmd

I want to transfer a mysql dump, compressed, to s3. I tried: mysqldump -u root -ppassword -all-databases | gzip -9 | s3cmd put s3://bucket/sql/databases.sql.gz but then I get: ERROR: Not enough ...
-1
votes
1answer
691 views

Python error when s3cmd is executed within bash script

I'm creating a bash script of which will run on a cron, it's purpose is to backup a directory to Amazon S3 using the s3cmd command. I get the following error when I run the script: /usr/bin/env: ...
0
votes
0answers
2k views

s3cmd 403 Access Denied error on put but not ls

I can access s3 using s3cmd on an ec2 instance to do s3cmd ls s3://myc bucketname but not using 'put' to add a file. It gives a 403 access denied error. I am using IAM and have group setup to access ...
4
votes
2answers
2k views

Amazon S3 - private file is still downloadable for everyone?

I have some secret files which I want to restrict anonymous user from viewing/downloading. I have tried to make it private by running something like: s3cmd setacl --acl-private s3://bucket/some/path/*...
1
vote
1answer
293 views

Saving httpd logfiles to S3 on shutdown using s3cmd

I have an auto-scaling group on Amazon Web Services (AWS). I want to preserve the logfiles on an instance (Amazon Linux AMI) when the group scales down. My current idea is to use the s3cmd tool in a ...
0
votes
1answer
103 views

How can I control a gradual file upload with cron or similar? [closed]

I have several terabytes of data I need to back up to S3/Glacier over a slow link. I want to upload from, say 9pm to 9am and resume from where it left off. Is there free software that would do that ...
2
votes
1answer
4k views

How to copy all s3 buckets to another AWS account?

I'm copying all EC2 instances to another AWS account (creating the AMIs, sharing them with that account and running the instances from AMIs there). I also need to move my S3 buckets since the ...
5
votes
1answer
2k views

Linux server sync to an Amazon S3 bucket

I am looking for a stable solution to replace a classic server backup to another server using rsync. I have to sync a whole filesystem (more than 1Tb) to Amazon S3. Where am I? Solution 1: I mapped ...
2
votes
1answer
5k views

s3cmd with --delete-removed

I'm currently writing a script to sync files in s3 buckets with s3cmd. I check the document and it says: s3cmd sync LOCAL_DIR s3://BUCKET[/PREFIX] or s3://BUCKET[/PREFIX] LOCAL_DIR also I find a ...
0
votes
1answer
657 views

How continue sync with S3CMD after broken connection?

I'm using s3cmd to backup some logs into Amazon S3 buckets. Which is a great tool. But I notice that, very often, my sync's are broken, they just cancel and try to upload again, until they can upload ...
2
votes
1answer
2k views

How backup files with logrotate and Amazon S3?

I would like to create a new standard of two things: how long would be generated and be rotated the logs generated by my applications ? how to transfer the logs to Amazon S3, as a backup server ?. ...
3
votes
1answer
3k views

s3cmd put only files that do not exist in destination bucket

I've been using s3cmd to sync local files on our server with the S3 bucket. There are over 200,000 files on the local server. Can't seem to find any documentation on the site that explains how to ...
1
vote
1answer
2k views

One Way Sync of a Bucket With Local Directory

I have a local directory that I would like to synchronize with an S3 bucket. I have two specific requirements: If local file is the same as the remote, do not re-transfer it to the bucket. If some ...
1
vote
3answers
472 views

Backing up data (including mysqldumps) to S3

We have a web app on a number of servers and we want to add an additional layer of redundancy by backing up the key data to S3. The key data is the MySQL database and a folder containing dynamically ...