2

I'm having trouble running aws s3 sync as a cron job (on Ubuntu 16.04). Here's what I am able to do:

  • Manually run the script in the /etc/cron.hourly directory using sudo (so it doesn't seem to be a problem with passing environment values)
  • Manually run the script in a user directory without sudo

  • Add either the script or the command itself to /etc/crontab - using either one of these:

    23 * * * * myusername aws s3 sync /home/myusername/temp/ s3://manning2323

    */1 * * * * myusername /home/dbclinton/myusername/temp/s3script.sh

The problem with /etc/crontab, of course, is that it's liable to be overwritten by updates, so it's not good enough for my purposes.

Now here's what doesn't work:

  • Leaving the script in cron.hourly and waiting for it to run automatically. It does run (as syslog and journalctl report), but the AWS CLI isn't providing me with any output that I can see so I have no way of knowing why the sync isn't happening. How do I know the sync isn't happening? Because new files aren't showing up in my S3 bucket.

  • I get exactly the same result with my own crontab adding this line through crontab -e

    47 18 * * * /home/myusername/temp/s3script.sh >> /home/myusername/cronlog.txt

Any ideas what I'm doing wrong? Thanks,

Improve this question

4 Answers 4

Reset to default
0

As it turns out, because cron is a bit unpredictable in how it sees the PATH, I had to reference the aws binary in my script using its absolute address (/usr/local/bin/aws).

Improve this answer
1
  • I'd suggest you run a script from cron, rather than having the commands directly within cron. That makes testing easier as well. This is how I do s3 sync with cron on Amazon Linux
    – Tim
    Jul 10, 2017 at 0:13
0

Before you enable your cron, you can have your aws credentials configure in your session like:

$ aws configure
AWS Access Key ID [None]: yourAccessKeyID
AWS Secret Access Key [None]: yourSecretAccessKey
Default region name [None]: yourRegion
Default output format [None]: json

And so you can try with:

* * * * * /usr/local/bin/./aws s3 sync local/dir s3://my-bucket/path

Be careful to configuring aws with user correct to cron

Improve this answer
1
  • It is much better to use aws roles to handle your credentials than storing them on the instance itself.
    – holmberd
    Nov 28, 2018 at 18:27
0

Cron is a little demanding and specific this is how i got to work it on two different ubuntu boxes using two different awscli install methods.

On Ubuntu 16.04 installed via PIP

sudo /home/ubuntu/.local/bin/aws s3 sync --region ap-northeast-1 /var/backups/ s3://BUCKET/backups

On Ubuntu 18.04 installed via apt-get Run it with user ubuntu, when running with user root this will not work

sudo /usr/bin/aws s3 sync  --region ap-northeast-1 /var/backups/ s3://BUCKET/backups/

In both situations the --region is not required, but i paste it in for clarity

Improve this answer
1
  • How did you use sudo in a crontab? And WHY?
    – Michael Hampton
    Sep 17, 2020 at 13:47
0

If you install aws console using snap then the absolute/executable address will be like /snap/bin/aws otherwise its looks like /usr/bin/aws . It can be different according to your installation. You can find this path using $ whereis aws .

eg :

snap/bin/aws s3 sync --exclude "folder/path to exclude" pathtosync s3://bucketname/objects
Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .