23

Figured it out thanks to strace. The current instance id is cached in /var/tmp/aws-mon and that incorrect cache was preserved via the AMI process. The solution: # mv /var/tmp/aws-mon /var/tmp/aws-mon.bak


21

The Amazon CloudWatch Monitoring Scripts for Linux are comprised of two Perl scripts, both using one Perl module - a short peek into the source reveals the following AWS API actions being used: CloudWatchClient.pm - DescribeTags mon-get-instance-stats.pl - GetMetricStatistics, ListMetrics mon-put-instance-data.pl - PutMetricData With this information you ...


18

Amazon notes in their documentation on T2 (burstable) instances that while these instances normally have 30 CPU credits initially, there are limits to how this initial credit will be applied. There are limits to how many T2 instances will launch or start with the initial CPU credit, which by default is set to 100 launches or starts of any T2 instance ...


14

A very good observation and we have run into this as well. Here's what I found: Be careful measuring CPU usage from within an EC2 instance. It’s possible to see CPU usage well below 100%—and yet be completely maxed out. Trust me: been there, done that. (CloudWatch CPUUtilization, by the way, is measured from outside the instance and is always correct.) ...


9

Do you have detailed monitoring enabled on your instance? If you create CloudWatch Alarms for period 1 minute without detailed monitoring enabled, it will stuck at Insufficient Data for 4 minutes after showing OK for every 5th minute . This is because with normal monitoring the reporting data comes with the interval of 5 minutes whereas in detailed ...


9

A couple of command tools for a CloudWatch tail: https://github.com/jorgebastida/awslogs https://github.com/liamoehlman/cloudwatch-logs-tail


7

A different approach that can work with AWS is to store the updated site/data/config somewhere like S3. Configure the AMI (or specify appropriate user-data script) so that when a new instance is spun up, it updates itself. You can prevent it from being added to the load balancer by not responding positively to the health check until the updates have ...


7

I have had some perspective on this in the last few months & I believe these items to watch will address all the concerns above: 1) The comment from @Ross on the original posting is the key. T2 instances, no matter what scale and no matter whether they are EC2 or RDS, will stop performing when their CPU credits run out as the peak CPU demands continue. ...


6

Please read http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/QuickStartEC2Instance.html for installation instruction for ubuntu/debian: Get installer with curl https://s3.amazonaws.com//aws-cloudwatch/downloads/latest/awslogs-agent-setup.py -O and run the installer sudo python ./awslogs-agent-setup.py --region us-east-1 After you have ...


5

CloudWatch uses universal time to record metrics. (As it ought to. Consider servers and people located in different time zones, different DST change dates, not to mention the insanity of DST itself. As in aviation, UT is the only reasonable way to cope.) So the CloudWatch system uses UT. This is independent of the systems being monitored. Which is why ...


4

In your email it clearly states that your alarm is set to trigger after 3600 seconds. Threshold: The alarm is in the ALARM state when the metric is LessThanThreshold 2.0 for 3600 seconds. There should be an option to set "EvaluationPeriods". What this does is it tells the alarm how many times to evaluate the specific metric you wish to check. So in your ...


4

CloudWatch on EC2 is enabled with the EC2 API. Since you have already created the instances outside CloudFormation, CloudFormation cannot modify them. You can create Alarms with CloudFormation, but you cannot enable monitoring on the instances. If you have monitoring enabled, you can reference the instances by specifying the "Dimensions" property in the ...


4

Unresponsive != no heartbeats. The kernel is still running. AWS has no way of knowing that you've consumed all of your memory. AWS Cloudwatch monitoring is really just the bare minimum you should do. If you need more detailed monitoring, you'll need to roll your own.


4

i had the same problem. you can see the logs of the script with: cat /var/log/awslogs-agent-setup.log there were some problems with the locals and a export LC_ALL=C solved it for me.


3

I ran into the same problem as you. I've got an auto scaling group that scales up an down with traffic and I push updates all the time even under peak load. My solution is quite a bit more involved since I need a staging area to be able to test my code updates before they hit live servers. At its simplest level here's how I build / update my servers. Build ...


3

Your EC2 instance must be enabled for monitoring at 1-minute intervals. If you have not enabled detailed monitoring on your EC2 instance, then you would be collecting data in 5-minute intervals. 3 consecutive periods of 5-minute intervals would be 15 minutes. I'm not certain, but after reviewing some of my own Cloudwatch alarms and playing with a new one ...


3

You can't do it thru CloudFormation templates. As you suggested, it will have to be handled thru the user-data script.


3

You can use OpsGenie (http://www.opsgenie.com) to send rich alert for CloudWatch. Currenly CloudWatch has a limited set of alerting mechanism including Email and SMS via its SNS mechanism. You can configure CloudWatch to call OpsGenie web services API, get the right people notified rapidly via push notifications to iPhone/Android apps, SMS, voice calls, etc....


3

I was really disappointed with awslogs and cloudwatch-logs-tail so I made my own tool called Saw that efficiently streams CloudWatch logs to the console (and colorizes the JSON output): You can install it on MacOS with: brew tap TylerBrock/saw brew install saw It has a bunch of nice features like the ability to automatically expand (indent) the JSON ...


3

Answering my own question with: There were abandonend multipart uploads that were correctly counted in CloudWatch.


3

I can confirm what Dojo said above. Although the wording in the description of the dimensions block makes it seem as though it is not required and strings/values are arbitrary, there is a specific list of names that can be used for each type of service you want to create an alarm for. This list maps the service type to allowable values for in the dimensions ...


3

The CloudWatch Logs agent awslogs RPM package is only available on Amazon Linux. If you have Ubuntu Server, CentOS, or Red Hat, you'll need to manually install it. You need to have Python installed and apt-get update is recommended before installing the CloudWatch Logs agent. You have two options. Run directly from the internet: curl https://s3.amazonaws....


3

Do you have your ELB set to TCP? If so, your ELB will register every connection made into the surge queue. There is no getting around that I'm afraid. You have to use http or https to get surge queue working properly. -- Update from OP I created a tiny EC2 instance, with a simple TCP server which just replies 'potato' to every request. I put it behind a ...


3

You will need to install the CloudWatch Agent in your instance. Both Linux and Windows is supported. This link will provide details: Collect Metrics from Amazon EC2 Instances and On-Premises Servers


3

It was aborted multipart uploads. S3 keeps every uploaded part of every failed multipart upload indefinitely by default! A process had been failing and retrying multipart uploads without explicitly cleaning up the failed transfers. We remedied this by temporarily enabling versioning, setting a lifecycle rule to remove aborted multipart upload chunks after 1 ...


2

You can implement an EC2 status check. It's done from the EC2 dashboard. Go to instances, select your instance, choose the status checks tab (next to instance description) Click on create status check alarm The default "Status Check Failed (any)" should be good. I always set the interval to greater than one so I don't get bothered for transient issues. It's ...


2

You can use Route 53 and its "Health checks". With this, you can send SNS alerts and also redirect your users to another secondary website or an error screen. I think this is better solution for your problem then Cloudwatch.


2

You can create an Alarm in Cloudwatch and set the alarm to notify you when it goes into "Insufficient Data" state. Most of the already available metrics are from the VM Host, which doesn't have any real idea about what's happening inside your machine. At a start, I'd recommend installing the Amazon tools in your instance and set up a script to report ...


2

Yes you can. Sign up for the RSS feed of the zone and service of your concern such as "Amazon Elastic Compute Cloud (N. Virginia)". Find this on the Amazon Service Status Page. When you select the small orange icon in the RSS column, your browser will prompt you to choose a service to use for viewing that RSS feed. With the Google Chrome browser, I ...


2

There's an Amazon provided IAM policy for CloudWatch. No need to build your own. CloudWatchFullAccess


Only top voted, non community-wiki answers of a minimum length are eligible