0

Environment, system details, and tools:

Has anyone ever seen this? We have a cron job that pushes Cloudwatch metrics from inside an instance by basically doing these steps:

  1. Get instanceId by running "wget -q -O - http://169.254.169.254/latest/meta-data/instance-id"
  2. Collecting some metric or other and building an AWS CLI query using aws cloudwatch put-metric-data ...
  3. Repeat

The weird thing we are seeing is, very infrequently, one of these runs will die after the wget query, with no output. As if the Metadata service just failed to respond.

Example end-of-script (we set bash -e and -x to die and to gather debug output):

++ wget -q -O - http://169.254.169.254/latest/meta-data/instance-id
+ INSTANCE_ID=

The script ends there and exits because presumably wget exited with a non-zero exit status.

This is not reproducible, but it happens on the order of once every 2 weeks.

  • It seems like the first thing you'd want to do is to not suppress the actual error you're hitting, which you're doing by specifying -q. The output from -O - goes to STDOUT but all the other nonsense wget normally spews on a successful request goes to STDERR, so there isn't an actual need to suppress it when you're capturing its output. – Michael - sqlbot May 3 '16 at 21:10
  • ...although you could reduce the spew somewhat by trimming some known nonsense from the STDERR stream with something like INSTANCE_ID=$(wget -O - http://169.254.169.254/latest/meta-data/instance-id 2> >(perl -pe 'undef $_ if /^Length/ || /^Saving\ to/ || /written\ to\ stdout/ || /100%/ || /^$/' >&2)) ... or just use INSTANCE_ID=$(ec2metadata --instance-id), which might also provide a useful error -- not sure, since I've never encountered this problem. – Michael - sqlbot May 3 '16 at 21:17
  • Thanks. My colleague and I just discussed removing -q, so we'll probably do that. Beyond that though, I'm going to wrap this call in a retry loop and call it a day. Thanks – JDS May 3 '16 at 21:26
1

You could be throttled. Specially if the issue is inconsistent. http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html#instancedata-throttling

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.