How to post logs to Cloudwatch with CLI?

Question

Im tring to create a bash script that checks the status of the website, Im using this command:

This one to create the logstream

aws logs create-log-stream --log-group-name "WebsiteStatusMessage" --log-stream-name $timestamp

This other one to post the logs

aws logs put-log-events --log-group-name "WebsiteStatusMessage" --log-stream-name "$timestamp" --log-events file://$iam/logsoutput.json

logsoutput.json

 [ 
   { 
     "timestamp": 202006041832, 
     "message": "test event1" 
   } 
 ]

And when I execute the command or the script it shows this message:

{
    "rejectedLogEventsInfo": {
        "tooOldLogEventEndIndex": 1
    }
}

And also this other message on console:

Expecting property name enclosed in double quotes: line 6 column 2 (char 84)

Bash script code

#!/bin/bash

#variables
iam=$(pwd)
timestamp=$(date +"%Y%m%d%H%M")
instance_id="i-######"

#Read line per line and storage it on a array
getArray()
{
array=()
  while IFS= read -r line
  do
       array+=("$line")
   done < "$1"
}
getArray "$iam/sites.txt"


#working
for url in "${array[@]}"
do
  echo "The website is: $url"
  STATUS=$(curl -s -o /dev/null -w "%{http_code}\n" $url)
    if [ "$STATUS" == "200" ] || [ "$STATUS" == "301" ] || [ "$STATUS" == "302" ]; then
        echo "$url is up, returned $STATUS"
    else
        echo "$url is not up, returned $STATUS"
###
# This will send the metric to metrics Cloudwatch
###

        rm $iam/metricsOutput.json
        echo " [ " >> $iam/metricsOutput.json
        echo " { " >> $iam/metricsOutput.json
        echo " \"MetricName\": \"SiteStatus\", " >> $iam/metricsOutput.json        
        echo " \"Timestamp\": \"$timestamp\", " >> $iam/metricsOutput.json
        echo " \"Value\": 1, " >> $iam/metricsOutput.json
        echo " } " >> $iam/metricsOutput.json
        echo " ] " >> $iam/metricsOutput.json


        aws cloudwatch put-metric-data --namespace "Custom-2" --metric-data file://$iam/metricsOutput.json


###
# This sends the message to logstream on Cloudwatch
###
        rm $iam/logsoutput.json
        echo " [ " >> $iam/logsoutput.json
        echo " { " >> $iam/logsoutput.json
        echo " \"timestamp\": $timestamp, " >> $iam/logsoutput.json
        echo " \"message\": \"test event1\" " >> $iam/logsoutput.json
        echo " } " >> $iam/logsoutput.json
        echo " ] " >> $iam/logsoutput.json

        aws logs create-log-stream --log-group-name "WebsiteStatusMessage" --log-stream-name $timestamp
        aws logs put-log-events --log-group-name "WebsiteStatusMessage" --log-stream-name "$timestamp" --log-events file://$iam/logsoutput.json


    fi
done

I tried with different json structures but still nothing, any idea? (The aws cli have full cloudwatch permissions)

Answer 1

Haven’t tested but I think the timestamp should be a unix timestamp (seconds since 1970-01-01 00:00:00) not a date, i.e. $(date +%s) and quite possibly in millisecond precision so append 000 at the end.

Hope that helps :)

Answer 2

1, You should use the Epoch format time. 2. The time should be milliseconds in epoch format, however you are giving the seconds in the sample. https://docs.aws.amazon.com/cli/latest/reference/logs/put-log-events.html

Try sth like this:

EPOCH_TIMESTAMP=$(($(date +%s)*1000))
aws logs put-log-events \
      --log-group-name "WebsiteStatusMessage" \
      --log-stream-name "$EPOCH_TIMESTAMP" \
      --log-events file://$iam/logsoutput.json

Answer 3

You should set a timestamp in unix time in miliseconds. So change this in your script.

timestamp=$(date +%s%3N)

Answer 4

I'd suggest you seperate the two functions of your script. You can make your script handle the outputting of the logs into the .json files, and install the CloudWatch agent on the instance. When you configure the CloudWatch agent, you can tell it to include your custom log folder and it will push everything in a clean fashion to CloudWatch. There are some minor charges for using the agent because of the granularity so definitely check out the pricing for it. It's a much better solution than using a CLI command in a bash script to manually push your logs. If there's a managed service for something, always use that - that's my motto at least.