0

My cluster is on AWS and I've used kops to build by cluster.

I'm trying to send the kubernetes logs to AWS Cloud-watch using fluentd. I was able to stand-up the fluentd pods.

I am seeing the below error from the pod's logs. I have created necessary policies for the pods to assume to send logs to the log-group in cloud watch logs.

2018-10-23 18:16:03 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2018-10-23 18:15:59 +0000 error_class="Aws::Errors::MissingCredentialsError" error="unable to sign
request without credentials set" plugin_id="object:2b07171c9ce4" 
018-10-23 18:16:04 +0000 [warn]: /fluentd/vendor/bundle/ruby/2.4.0/gems/aws-sdk-core-3.22.1/lib/aws-sdk-core/plugins/signature_v4.rb:72:in `sign_request'
  2018-10-23 18:16:04 +0000 [warn]: /fluentd/vendor/bundle/ruby/2.4.0/gems/aws-sdk-core-3.22.1/lib/aws-sdk-core/plugins/signature_v4.rb:112:in `apply_signature'
  2018-10-23 18:16:04 +0000 [warn]: /fluentd/vendor/bundle/ruby/2.4.0/gems/aws-sdk-core-3.22.1/lib/aws-sdk-core/plugins/signature_v4.rb:65:in `call'
  2018-10-23 18:16:04 +0000 [warn]: /fluentd/vendor/bundle/ruby/2.4.0/gems/aws-sdk-core-3.22.1/lib/aws-sdk-core/plugins/helpful_socket_errors.rb:10:in `call'
  2018-10-23 18:16:04 +0000 [warn]: /fluentd/vendor/bundle/ruby/2.4.0/gems/aws-sdk-core-3.22.1/lib/aws-sdk-core/plugins/retry_errors.rb:138:in `call'
  2018-10-23 18:16:04 +0000 [warn]: /fluentd/vendor/bundle/ruby/2.4.0/gems/aws-sdk-core-3.22.1/lib/aws-sdk-core/json/handler.rb:11:in `call'
  2018-10-23 18:16:04 +0000 [warn]: /fluentd/vendor/bundle/ruby/2.4.0/gems/aws-sdk-core-3.22.1/lib/aws-sdk-core/plugins/user_agent.rb:13:in `call'
  2018-10-23 18:16:04 +0000 [warn]: /fluentd/vendor/bundle/ruby/2.4.0/gems/aws-sdk-core-3.22.1/lib/seahorse/client/plugins/endpoint.rb:45:in `call'
  2018-10-23 18:16:04 +0000 [warn]: /fluentd/vendor/bundle/ruby/2.4.0/gems/aws-sdk-core-3.22.1/lib/aws-sdk-core/plugins/param_validator.rb:24:in `call'
  2018-10-23 18:16:04 +0000 [warn]: /fluentd/vendor/bundle/ruby/2.4.0/gems/aws-sdk-core-3.22.1/lib/seahorse/client/plugins/raise_response_errors.rb:14:in `call'
  2018-10-23 18:16:04 +0000 [warn]: /fluentd/vendor/bundle/ruby/2.4.0/gems/aws-sdk-core-3.22.1/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in `call'
  2018-10-23 18:16:04 +0000 [warn]: /fluentd/vendor/bundle/ruby/2.4.0/gems/aws-sdk-core-3.22.1/lib/aws-sdk-core/plugins/idempotency_token.rb:17:in `call'
  2018-10-23 18:16:04 +0000 [warn]: /fluentd/vendor/bundle/ruby/2.4.0/gems/aws-sdk-core-3.22.1/lib/aws-sdk-core/plugins/param_converter.rb:24:in `call'
  2018-10-23 18:16:04 +0000 [warn]: /fluentd/vendor/bundle/ruby/2.4.0/gems/aws-sdk-core-3.22.1/lib/aws-sdk-core/plugins/response_paging.rb:10:in `call'
  2018-10-23 18:16:04 +0000 [warn]: /fluentd/vendor/bundle/ruby/2.4.0/gems/aws-sdk-core-3.22.1/lib/seahorse/client/plugins/response_target.rb:23:in `call'
  2018-10-23 18:16:04 +0000 [warn]: /fluentd/vendor/bundle/ruby/2.4.0/gems/aws-sdk-core-3.22.1/lib/seahorse/client/request.rb:70:in `send_request'
  2018-10-23 18:16:04 +0000 [warn]: /fluentd/vendor/bundle/ruby/2.4.0/gems/aws-sdk-cloudwatchlogs-1.4.0/lib/aws-sdk-cloudwatchlogs/client.rb:736:in `describe_log_groups'
  2018-10-23 18:16:04 +0000 [warn]: /fluentd/vendor/bundle/ruby/2.4.0/gems/fluent-plugin-cloudwatch-logs-0.4.5/lib/fluent/plugin/out_cloudwatch_logs.rb:403:in `log_group_exists?'
  2018-10-23 18:16:04 +0000 [warn]: /fluentd/vendor/bundle/ruby/2.4.0/gems/fluent-plugin-cloudwatch-logs-0.4.5/lib/fluent/plugin/out_cloudwatch_logs.rb:152:in `block in write'
  2018-10-23 18:16:04 +0000 [warn]: /fluentd/vendor/bundle/ruby/2.4.0/gems/fluent-plugin-cloudwatch-logs-0.4.5/lib/fluent/plugin/out_cloudwatch_logs.rb:144:in `each'
  2018-10-23 18:16:04 +0000 [warn]: /fluentd/vendor/bundle/ruby/2.4.0/gems/fluent-plugin-cloudwatch-logs-0.4.5/lib/fluent/plugin/out_cloudwatch_logs.rb:144:in `write'
  2018-10-23 18:16:04 +0000 [warn]: /fluentd/vendor/bundle/ruby/2.4.0/gems/fluentd-0.12.43/lib/fluent/buffer.rb:354:in `write_chunk'
  2018-10-23 18:16:04 +0000 [warn]: /fluentd/vendor/bundle/ruby/2.4.0/gems/fluentd-0.12.43/lib/fluent/buffer.rb:333:in `pop'
  2018-10-23 18:16:04 +0000 [warn]: /fluentd/vendor/bundle/ruby/2.4.0/gems/fluentd-0.12.43/lib/fluent/output.rb:342:in `try_flush'
  2018-10-23 18:16:04 +0000 [warn]: /fluentd/vendor/bundle/ruby/2.4.0/gems/fluentd-0.12.43/lib/fluent/output.rb:149:in `run'
2018-10-23 18:16:07 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2018-10-23 18:16:01 +0000 error_class="Aws::Errors::MissingCredentialsError" error="unable to sign
request without credentials set" plugin_id="object:2b07171c9ce4"
  2018-10-23 18:16:07 +0000 [warn]: suppressed same stacktrace 
0

I installed kube2iam and it resolved the issue.

https://github.com/jtblin/kube2iam

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.