0

I'm having trouble running aws s3 sync as a cron job (on Ubuntu 16.04). Here's what I am able to do:

  • Manually run the script in the /etc/cron.hourly directory using sudo (so it doesn't seem to be a problem with passing environment values)
  • Manually run the script in a user directory without sudo
  • Add either the script or the command itself to /etc/crontab - using either one of these:

    23 * * * * myusername aws s3 sync /home/myusername/temp/ s3://manning2323

    */1 * * * * myusername /home/dbclinton/myusername/temp/s3script.sh

The problem with /etc/crontab, of course, is that it's liable to be overwritten by updates, so it's not good enough for my purposes.

Now here's what doesn't work:

  • Leaving the script in cron.hourly and waiting for it to run automatically. It does run (as syslog and journalctl report), but the AWS CLI isn't providing me with any output that I can see so I have no way of knowing why the sync isn't happening. How do I know the sync isn't happening? Because new files aren't showing up in my S3 bucket.
  • I get exactly the same result with my own crontab adding this line through crontab -e

    47 18 * * * /home/myusername/temp/s3script.sh >> /home/myusername/cronlog.txt

Any ideas what I'm doing wrong? Thanks,

0

As it turns out, because cron is a bit unpredictable in how it sees the PATH, I had to reference the aws binary in my script using its absolute address (/usr/local/bin/aws).

1
  • I'd suggest you run a script from cron, rather than having the commands directly within cron. That makes testing easier as well. This is how I do s3 sync with cron on Amazon Linux – Tim Jul 10 '17 at 0:13
0

Before you enable your cron, you can have your aws credentials configure in your session like:

$ aws configure
AWS Access Key ID [None]: yourAccessKeyID
AWS Secret Access Key [None]: yourSecretAccessKey
Default region name [None]: yourRegion
Default output format [None]: json

And so you can try with:

* * * * * /usr/local/bin/./aws s3 sync local/dir s3://my-bucket/path

Be careful to configuring aws with user correct to cron

1
  • It is much better to use aws roles to handle your credentials than storing them on the instance itself. – holmberd Nov 28 '18 at 18:27
0

Cron is a little demanding and specific this is how i got to work it on two different ubuntu boxes using two different awscli install methods.

On Ubuntu 16.04 installed via PIP

sudo /home/ubuntu/.local/bin/aws s3 sync --region ap-northeast-1 /var/backups/ s3://BUCKET/backups

On Ubuntu 18.04 installed via apt-get Run it with user ubuntu, when running with user root this will not work

sudo /usr/bin/aws s3 sync  --region ap-northeast-1 /var/backups/ s3://BUCKET/backups/

In both situations the --region is not required, but i paste it in for clarity

1
  • How did you use sudo in a crontab? And WHY? – Michael Hampton Sep 17 '20 at 13:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.