0

I'm having trouble running aws s3 sync as a cron job (on Ubuntu 16.04). Here's what I am able to do:

  • Manually run the script in the /etc/cron.hourly directory using sudo (so it doesn't seem to be a problem with passing environment values)
  • Manually run the script in a user directory without sudo
  • Add either the script or the command itself to /etc/crontab - using either one of these:

    23 * * * * myusername aws s3 sync /home/myusername/temp/ s3://manning2323

    */1 * * * * myusername /home/dbclinton/myusername/temp/s3script.sh

The problem with /etc/crontab, of course, is that it's liable to be overwritten by updates, so it's not good enough for my purposes.

Now here's what doesn't work:

  • Leaving the script in cron.hourly and waiting for it to run automatically. It does run (as syslog and journalctl report), but the AWS CLI isn't providing me with any output that I can see so I have no way of knowing why the sync isn't happening. How do I know the sync isn't happening? Because new files aren't showing up in my S3 bucket.
  • I get exactly the same result with my own crontab adding this line through crontab -e

    47 18 * * * /home/myusername/temp/s3script.sh >> /home/myusername/cronlog.txt

Any ideas what I'm doing wrong? Thanks,

0

As it turns out, because cron is a bit unpredictable in how it sees the PATH, I had to reference the aws binary in my script using its absolute address (/usr/local/bin/aws).

  • I'd suggest you run a script from cron, rather than having the commands directly within cron. That makes testing easier as well. This is how I do s3 sync with cron on Amazon Linux – Tim Jul 10 '17 at 0:13
0

Before you enable your cron, you can have your aws credentials configure in your session like:

$ aws configure
AWS Access Key ID [None]: yourAccessKeyID
AWS Secret Access Key [None]: yourSecretAccessKey
Default region name [None]: yourRegion
Default output format [None]: json

And so you can try with:

* * * * * /usr/local/bin/./aws s3 sync local/dir s3://my-bucket/path

Be careful to configuring aws with user correct to cron

  • It is much better to use aws roles to handle your credentials than storing them on the instance itself. – holmberd Nov 28 '18 at 18:27

Your Answer

By clicking "Post Your Answer", you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.