3

I have this script from Colin Johnson on Github - https://github.com/colinbjohnson/aws-missing-tools/tree/master/ec2-automate-backup

It seems great. I have modified it to send email to myself every time an EBS snapshot is created or deleted. The following works like a charm

ec2-automate-backup.sh -v "vol-myvolumeid" -k 3

However, it does not execute at all as part of my crontab (I didn't receive any emails)

#some command that got commented out

*/5 * * * * ec2-automate-backup.sh -v "vol-fb2fbcdf" -k 3;

* * * * * date >> /root/logs/crontab.log;

*/5 * * * * date >> /root/logs/crontab2.log

Please note that the 2nd and 3rd execute just fines as I can see the date and time in log files.

What could I have missed here?

The full ec2-automate-backup.sh is as follows:

#!/bin/bash -
# Author: Colin Johnson / [email protected]
# Date: 2012-09-24
# Version 0.1
# License Type: GNU GENERAL PUBLIC LICENSE, Version 3
#
#confirms that executables required for succesful script execution are available

prerequisite_check()
{
    for prerequisite in basename ec2-create-snapshot ec2-create-tags ec2-describe-snapshots ec2-delete-snapshot date
    do
        #use of "hash" chosen as it is a shell builtin and will add programs to hash table, possibly speeding execution. Use of type also considered - open to suggestions.
        hash $prerequisite &> /dev/null
        if [[ $? == 1 ]] #has exits with exit status of 70, executable was not found
            then echo "In order to use `basename $0`, the executable \"$prerequisite\" must be installed." 1>&2 | mailx -s "Error happened 0" [email protected] ; exit 70
        fi
    done
}

#get_EBS_List gets a list of available EBS instances depending upon the selection_method of EBS selection that is provided by user input
get_EBS_List()
{
    case $selection_method in
        volumeid)
            if [[ -z $volumeid ]]
                then echo "The selection method \"volumeid\" (which is $app_name's default selection_method of operation or requested by using the -s volumeid parameter) requires a volumeid (-v volumeid) for operation. Correct usage is as follows: \"-v vol-6d6a0527\",\"-s volumeid -v vol-6d6a0527\" or \"-v \"vol-6d6a0527 vol-636a0112\"\" if multiple volumes are to be selected." 1>&2 | mailx -s "Error happened 1" [email protected] ; exit 64
            fi
            ebs_selection_string="$volumeid"
            ;;
        tag) 
            if [[ -z $tag ]]
                then echo "The selected selection_method \"tag\" (-s tag) requires a valid tag (-t key=value) for operation. Correct usage is as follows: \"-s tag -t backup=true\" or \"-s tag -t Name=my_tag.\"" 1>&2 | mailx -s "Error happened 2" [email protected] ; exit 64
            fi
            ebs_selection_string="--filter tag:$tag"
            ;;
        *) echo "If you specify a selection_method (-s selection_method) for selecting EBS volumes you must select either \"volumeid\" (-s volumeid) or \"tag\" (-s tag)." 1>&2 | mailx -s "Error happened 3" [email protected] ; exit 64 ;;
    esac
    #creates a list of all ebs volumes that match the selection string from above
    ebs_backup_list_complete=`ec2-describe-volumes --show-empty-fields --region $region $ebs_selection_string 2>&1`
    #takes the output of the previous command 
    ebs_backup_list_result=`echo $?`
    if [[ $ebs_backup_list_result -gt 0 ]]
        then echo -e "An error occured when running ec2-describe-volumes. The error returned is below:\n$ebs_backup_list_complete" 1>&2 | mailx -s "Error happened 4" [email protected] ; exit 70
    fi
    ebs_backup_list=`echo "$ebs_backup_list_complete" | grep ^VOLUME | cut -f 2`
    #code to right will output list of EBS volumes to be backed up: echo -e "Now outputting ebs_backup_list:\n$ebs_backup_list"
}

create_EBS_Snapshot_Tags()
{
    #snapshot tags holds all tags that need to be applied to a given snapshot - by aggregating tags we ensure that ec2-create-tags is called only onece
    snapshot_tags=""
    #if $name_tag_create is true then append ec2ab_${ebs_selected}_$date_current to the variable $snapshot_tags
    if $name_tag_create
        then
        ec2_snapshot_resource_id=`echo "$ec2_create_snapshot_result" | cut -f 2`
        snapshot_tags="$snapshot_tags --tag Name=ec2ab_${ebs_selected}_$date_current"
    fi
    #if $purge_after_days is true, then append $purge_after_date to the variable $snapshot_tags
    if [[ -n $purge_after_days ]]
        then
        snapshot_tags="$snapshot_tags --tag PurgeAfter=$purge_after_date --tag PurgeAllow=true"
    fi
    #if $snapshot_tags is not zero length then set the tag on the snapshot using ec2-create-tags
    if [[ -n $snapshot_tags ]]
        then echo "Tagging Snapshot $ec2_snapshot_resource_id with the following Tags:"
        ec2-create-tags $ec2_snapshot_resource_id --region $region $snapshot_tags
        #echo "Snapshot tags successfully created" | mailx -s "Snapshot tags successfully created" [email protected]
    fi
}

date_command_get()
{
    #finds full path to date binary
    date_binary_full_path=`which date`
    #command below is used to determine if date binary is gnu, macosx or other
    date_binary_file_result=`file -b $date_binary_full_path`
    case $date_binary_file_result in
        "Mach-O 64-bit executable x86_64") date_binary="macosx" ;;
        "ELF 64-bit LSB executable, x86-64, version 1 (SYSV)"*) date_binary="gnu" ;;
        *) date_binary="unknown" ;;
    esac
    #based on the installed date binary the case statement below will determine the method to use to determine "purge_after_days" in the future
    case $date_binary in
        gnu) date_command="date -d +${purge_after_days}days -u +%Y-%m-%d" ;;
        macosx) date_command="date -v+${purge_after_days}d -u +%Y-%m-%d" ;;
        unknown) date_command="date -d +${purge_after_days}days -u +%Y-%m-%d" ;;
        *) date_command="date -d +${purge_after_days}days -u +%Y-%m-%d" ;;
    esac
}

purge_EBS_Snapshots()
{
    #snapshot_tag_list is a string that contains all snapshots with either the key PurgeAllow or PurgeAfter set
    snapshot_tag_list=`ec2-describe-tags --show-empty-fields --region $region --filter resource-type=snapshot --filter key=PurgeAllow,PurgeAfter`
    #snapshot_purge_allowed is a list of all snapshot_ids with PurgeAllow=true
    snapshot_purge_allowed=`echo "$snapshot_tag_list" | grep .*PurgeAllow'\t'true | cut -f 3`

    for snapshot_id_evaluated in $snapshot_purge_allowed
    do
        #gets the "PurgeAfter" date which is in UTC with YYYY-MM-DD format (or %Y-%m-%d)
        purge_after_date=`echo "$snapshot_tag_list" | grep .*$snapshot_id_evaluated'\t'PurgeAfter.* | cut -f 5`
        #if purge_after_date is not set then we have a problem. Need to alter user.
        if [[ -z $purge_after_date ]]
            #Alerts user to the fact that a Snapshot was found with PurgeAllow=true but with no PurgeAfter date.
            then echo "A Snapshot with the Snapshot ID $snapshot_id_evaluated has the tag \"PurgeAllow=true\" but does not have a \"PurgeAfter=YYYY-MM-DD\" date. $app_name is unable to determine if $snapshot_id_evaluated should be purged." 1>&2 | mailx -s "Error happened 5" [email protected]
        else
            #convert both the date_current and purge_after_date into epoch time to allow for comparison
            date_current_epoch=`date -j -f "%Y-%m-%d" "$date_current" "+%s"`
            purge_after_date_epoch=`date -j -f "%Y-%m-%d" "$purge_after_date" "+%s"`
            #perform compparison - if $purge_after_date_epoch is a lower number than $date_current_epoch than the PurgeAfter date is earlier than the current date - and the snapshot can be safely removed
            if [[ $purge_after_date_epoch < $date_current_epoch ]]
                then
                echo "The snapshot \"$snapshot_id_evaluated\" with the Purge After date of $purge_after_date will be deleted."
                ec2-delete-snapshot --region $region $snapshot_id_evaluated
                echo "Old snapshots successfully deleted for $volumeid" | mailx -s "Old snapshots successfully deleted for $volumeid" [email protected]
            fi
        fi
    done
}

#calls prerequisitecheck function to ensure that all executables required for script execution are available
prerequisite_check

app_name=`basename $0`

#sets defaults
selection_method="volumeid"
region="ap-southeast-1"
#date_binary allows a user to set the "date" binary that is installed on their system and, therefore, the options that will be given to the date binary to perform date calculations
date_binary=""

#sets the "Name" tag set for a snapshot to false - using "Name" requires that ec2-create-tags be called in addition to ec2-create-snapshot
name_tag_create=false
#sets the Purge Snapshot feature to false - this feature will eventually allow the removal of snapshots that have a "PurgeAfter" tag that is earlier than current date
purge_snapshots=false
#handles options processing
while getopts :s:r:v:t:k:pn opt
    do
        case $opt in
            s) selection_method="$OPTARG";;
            r) region="$OPTARG";;
            v) volumeid="$OPTARG";;
            t) tag="$OPTARG";;
            k) purge_after_days="$OPTARG";;
            n) name_tag_create=true;;
            p) purge_snapshots=true;;
            *) echo "Error with Options Input. Cause of failure is most likely that an unsupported parameter was passed or a parameter was passed without a corresponding option." 1>&2 ; exit 64;;
        esac
    done

#sets date variable
date_current=`date -u +%Y-%m-%d`
#sets the PurgeAfter tag to the number of days that a snapshot should be retained
if [[ -n $purge_after_days ]]
    then
    #if the date_binary is not set, call the date_command_get function
    if [[ -z $date_binary ]]
        then date_command_get
    fi
    purge_after_date=`$date_command`
    echo "Snapshots taken by $app_name will be eligible for purging after the following date: $purge_after_date."
fi

#get_EBS_List gets a list of EBS instances for which a snapshot is desired. The list of EBS instances depends upon the selection_method that is provided by user input
get_EBS_List

#the loop below is called once for each volume in $ebs_backup_list - the currently selected EBS volume is passed in as "ebs_selected"
for ebs_selected in $ebs_backup_list
do
    ec2_snapshot_description="ec2ab_${ebs_selected}_$date_current"
    ec2_create_snapshot_result=`ec2-create-snapshot --region $region -d $ec2_snapshot_description $ebs_selected 2>&1`
    if [[ $? != 0 ]]
        then echo -e "An error occured when running ec2-create-snapshot. The error returned is below:\n$ec2_create_snapshot_result" 1>&2 ; exit 70
    else
        ec2_snapshot_resource_id=`echo "$ec2_create_snapshot_result" | cut -f 2`
        echo "Snapshots successfully created for volume $volumeid" | mailx -s "Snapshots successfully created for $volumeid" [email protected]
    fi  
    create_EBS_Snapshot_Tags
done

#if purge_snapshots is true, then run purge_EBS_Snapshots function
if $purge_snapshots
    then echo "Snapshot Purging is Starting Now."
    purge_EBS_Snapshots
fi

cron log

Oct 23 10:24:01 ip-10-130-153-227 CROND[28214]: (root) CMD (root (ec2-automate-backup.sh -v "vol-myvolumeid" -k 3;))
Oct 23 10:24:01 ip-10-130-153-227 CROND[28215]: (root) CMD (date  >> /root/logs/crontab.log;)
Oct 23 10:25:01 ip-10-130-153-227 CROND[28228]: (root) CMD (date  >> /root/logs/crontab.log;)
Oct 23 10:25:01 ip-10-130-153-227 CROND[28229]: (root) CMD (date >> /root/logs/crontab2.log)
Oct 23 10:26:01 ip-10-130-153-227 CROND[28239]: (root) CMD (date  >> /root/logs/crontab.log;)
Oct 23 10:27:01 ip-10-130-153-227 CROND[28247]: (root) CMD (root (ec2-automate-backup.sh -v "vol-myvolumeid" -k 3;))
Oct 23 10:27:01 ip-10-130-153-227 CROND[28248]: (root) CMD (date  >> /root/logs/crontab.log;)
Oct 23 10:28:01 ip-10-130-153-227 CROND[28263]: (root) CMD (date  >> /root/logs/crontab.log;)
Oct 23 10:29:01 ip-10-130-153-227 CROND[28275]: (root) CMD (date  >> /root/logs/crontab.log;)
Oct 23 10:30:01 ip-10-130-153-227 CROND[28292]: (root) CMD (root (ec2-automate-backup.sh -v "vol-myvolumeid" -k 3;))
Oct 23 10:30:01 ip-10-130-153-227 CROND[28293]: (root) CMD (date  >> /root/logs/crontab.log;)
Oct 23 10:30:01 ip-10-130-153-227 CROND[28294]: (root) CMD (date >> /root/logs/crontab2.log)
Oct 23 10:31:01 ip-10-130-153-227 CROND[28312]: (root) CMD (date  >> /root/logs/crontab.log;)
Oct 23 10:32:01 ip-10-130-153-227 CROND[28319]: (root) CMD (date  >> /root/logs/crontab.log;)
Oct 23 10:33:01 ip-10-130-153-227 CROND[28325]: (root) CMD (date  >> /root/logs/crontab.log;)
Oct 23 10:33:01 ip-10-130-153-227 CROND[28324]: (root) CMD (root (ec2-automate-backup.sh -v "vol-myvolumeid" -k 3;))
Oct 23 10:34:01 ip-10-130-153-227 CROND[28345]: (root) CMD (date  >> /root/logs/crontab.log;)
Oct 23 10:35:01 ip-10-130-153-227 CROND[28362]: (root) CMD (date  >> /root/logs/crontab.log;)
Oct 23 10:35:01 ip-10-130-153-227 CROND[28363]: (root) CMD (date >> /root/logs/crontab2.log)

Mails to root

From [email protected]  Tue Oct 23 06:00:01 2012
Return-Path: <[email protected]>
Date: Tue, 23 Oct 2012 06:00:01 GMT
From: [email protected] (Cron Daemon)
To: [email protected]
Subject: Cron <root@ip-10-130-153-227> root ec2-automate-backup.sh -v "vol-myvolumeid" -k 3
Content-Type: text/plain; charset=UTF-8
Auto-Submitted: auto-generated
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
X-Cron-Env: <USER=root>
Status: R

/bin/sh: root: command not found

Update Apparently, undx is right about cron having limited environment (x-cron-env) The backup script is located in /usr/local/bin/ which is not included in PATH of x-cron-env. This opens a new horizon to me as I assume cron simply execute the shell script the same way the script executes itself. Off to making the AWS CLI Tools work with cron now

As Colin Johnson has updated his script and as some of you pointed out, there was actually no need for me to edit such script. I simply needed to use it properly (with proper understanding of CRON and a bit of AWS CLI Tools). Doesn't make sense for amateur like myself to attempt to change the awesome script by guru like Colin either.

The backup script has worked beautifully me ever since. Strongly recommend AWS Missing Tools to everyone.

3
  • Does the job show up in the cron log? Does anything show up in the default mail directory (/var/spool/mail/<USERNAME>)?
    – Ladadadada
    Oct 23, 2012 at 10:28
  • Yes, sir :) let me update my question
    – ericn
    Oct 23, 2012 at 10:33
  • 1
    I've since updated ec2-automate-backup to have a "cron-primer" option to make configuration much easier - simply run ec2-automate-backup with "-c path_to_cron_primer" file to provide only the environment configuration necessary for running under cron. Nov 12, 2012 at 3:33

5 Answers 5

12

On a default UNIX-like system, cron has a minimal environment defined. Usually HOME, SHELL, LOGNAME are defined and PATH is set to /bin.

You have two solutions:

  • enter the full path of your script. ie: /home/me/bin/ec2-automate-backup.sh
  • alter the PATH environment variable.

If you want to receive email from the cron daemon define the MAILTO variable.

[email protected]
PATH=/bin:/home/me/bin
*/5 * * * * ec2-automate-backup.sh -v "vol-fb2fbcdf" -k 3
# OR
*/5 * * * * /path/to/script/ec2-automate-backup.sh -v "vol-fb2fbcdf" -k 3
2
  • As mentionned in your update in Mails to root Your PATH is X-Cron-Env: <PATH=/usr/bin:/bin>. Where is located your script ?
    – undx
    Oct 23, 2012 at 10:47
  • My script is at /usr/local/bin/ec2-automate-backup.sh. You are absolutely right about cron having limited environment defined. Having full path to the script does get my script executed. Thank you very much!
    – ericn
    Oct 23, 2012 at 12:53
1

Your cron script starts with #!/bin/bash -; however, this is ignored by cron.

Cron executes all commands with /bin/sh.

If you wish to execute a bash script with cron, you need to invoke it directly with bash:

*/5 * * *  * /bin/bash /full/path/to/your/script

Your script uses bash-specific features which will fail when run with the standard Bourne sh(1).

There are numerous issues with the script you present; I would strongly advise you to study the BashGuide for expert advice on how to craft efficient and safe bash scripts.

0

executable ? permissions ? Don't forget that since they are executed inside cron, they won't be executed as your normal user (so you may have some permission problems)

what if you change the line to

*/5 * * * * root (ec2-automate-backup.sh -v "vol-fb2fbcdf" -k 3;)
2
  • Hi @Nikolaidis, the file has permission 755. I've tried having the username root there in crontab but no success
    – ericn
    Oct 23, 2012 at 10:32
  • That runs it in a subshell; how is that an improvement ?
    – adaptr
    Oct 23, 2012 at 12:32
0

I had to export more variables, to be the same as in ec2-user environment.

I defined a new script ec2-backup.sh as below:

#!/bin/sh
export EC2_HOME=/opt/aws/apitools/ec2
export JAVA_HOME=/usr/lib/jvm/jre
export AWS_ACCESS_KEY=...
export AWS_SECRET_KEY=...
export PATH=/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/aws/bin:/home/ec2-user/bin
/home/ec2-user/aws-missing-tools-master/ec2-automate-backup/ec2-automate-backup.sh -v 'all volumes ' -n -r eu-west-1 -p -k 2

And then execute this script from my cronjob. Sending the job output to a log file was extremely helpful for debugging:

0 0 * * * /home/ec2-user/ec2-backup.sh >> /home/ec2-user/cronoutput.txt 2>&1
0

We had the same issue. What we did not want, was to create a script to export elements, but instead to source the existing environement variables, as this seemed less prone to errors in case the configuration changed.

I found my solution (well hidden in the last anstwer) here: AWS Help

The env vars for AWS EBS are stored in a file under: /opt/elasticbeanstalk/support/envvars. So sourcing or reading this file did the trick for me. In a root crontab this would look like this:

30 2 * * * root . /opt/elasticbeanstalk/support/envvars && php /var/www/bin/some_command.php

Like this, you:

  • only maintain your env vars in one place (your choice)
  • can use them wherever you need them
  • always use the same env vars

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .