1

We had a major issue with Google Cloud Platform some time between 2019-04-16T11:00:00Z and 2019-04-17T02:30:00Z, when all of our service accounts lost their access keys for an unknown reason, so our Google Compute Engine and Google Cloud Storage API requests failed, disrupting our production services.

ie. When we visited https://console.cloud.google.com/iam-admin/serviceaccounts, the entire column for "Key ID" displayed "No keys" and we had to create new keys for each service account to restore access.

Why or how could this happen and what can we do to prevent it happening again?

Improve this question
4
  • 2
    Have you contacted GCP support?
    – ceejayoz
    Apr 24, 2019 at 20:46
  • 4
    Check your audit logs.
    – Michael Hampton
    Apr 25, 2019 at 0:47
  • @ceejayoz We don't have a premium support package so I don't think we can't contact them directly. Instead I've reported an issue, submitted feedback and posted here and on Google Groups as they recommend here: cloud.google.com/support/docs/community
    – Sim Kennedy
    Apr 25, 2019 at 13:49
  • @MichaelHampton Aha! I didn't know about these. I just checked and can now see the entries for DeleteServiceAccountKey from an old user with admin access. Exactly what we needed! Thank you so much!
    – Sim Kennedy
    Apr 25, 2019 at 13:54

1 Answer 1

Reset to default
2

Thanks to Michael Hampton's comment, we got to the bottom of this issue after looking at the Audit logs (https://console.cloud.google.com/logs/viewer). The Service Account key deletions were executed via API by Terraform (https://www.terraform.io/), which was being used by a dev with Editor access testing Terraform on what they thought were their own projects. Automation can be dangerous..

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .