App Engine Flex service randomly crashes returning 502 error

Question

I work in a company that decided to move backend services to Google App Engine back in April. It's been pretty smooth sailing until today. We have Standard service for handling REST API calls and Flex service for handling WebSocket API. Few hours ago our Flex service started responding 502 to WebSocket handshakes. There were no errors in Flex service's logs, but there were multiple errors that looked like this:

{
  "insertId": "1cu1r4ugayjsq18",
  "jsonPayload": {
    "operation": {
      "zone": "europe-west3-c",
      "type": "operation",
      "name": "operation-1602285169497-5b145165b53ce-38a5e996-d4a261fe",
      "id": "8770045770006876317"
    },
    "error": [
      {
        "code": "ZONE_RESOURCE_POOL_EXHAUSTED",
        "location": "",
        "detail_message": ""
      }
    ],
    "trace_id": "operation-1602285169497-5b145165b53ce-38a5e996-d4a261fe",
    "version": "1.2",
    "event_subtype": "compute.instances.insert",
    "actor": {
      "user": "[email protected]"
    },
    "event_type": "GCE_OPERATION_DONE",
    "resource": {
      "id": "4266532621802885278",
      "type": "instance",
      "zone": "europe-west3-c",
      "name": "aef-socket--server--prod-20201002t220627-hdzf"
    },
    "event_timestamp_us": "1602285178688704"
  },
  "resource": {
    "type": "gce_instance",
    "labels": {
      "instance_id": "4266532621802885278",
      "project_id": "automatic-rock-252916",
      "zone": "europe-west3-c"
    }
  },
  "timestamp": "2020-10-09T23:12:58.688704Z",
  "severity": "ERROR",
  "labels": {
    "compute.googleapis.com/resource_zone": "europe-west3-c",
    "compute.googleapis.com/resource_type": "instance",
    "compute.googleapis.com/resource_name": "aef-socket--server--prod-20201002t220627-hdzf",
    "compute.googleapis.com/resource_id": "4266532621802885278"
  },
  "logName": "projects/automatic-rock-252916/logs/compute.googleapis.com%2Factivity_log",
  "receiveTimestamp": "2020-10-09T23:12:58.720000801Z"
}

Note "socket--server--prod" in labels, that's the name of our App Engine Flex service and the only reason I suspect this error might be somehow related to our problem, because again that error isn't from App Engine logs, it's just something I found when querying all GCP logs for all error logs.

We've managed to solve the problem this time by redeploying it with

gcloud app deploy socket-server.prod.yaml    --stop-previous-version

command, but the actual problem is the fact that

1. Instance crashed without any obvious reason. We didn't experience high load, and last time we redeployed that service was 7 days before it crashed so I'm all out of ideas aside from some internal GCP problem (and ZONE_RESOURCE_POOL_EXHAUSTED error code suggests that as well).

2. Even though redeploying service was enough to fix the problem, App Engine didn't do it. Until today I was sure that automatic scaling and restarting of crashed instances was one of the key features of App Engine, but now I am very confused. I mean, we're paying for App Engine services about $250 monthly yet it can't guarantee our services don't crash randomly and without any notice. In this case, our live production service was down for multiple hours and we only learned about that from user reports.

3. There is no customer support. The only support contacts I found in GCP console was about billing issues, and a link to stackoverflow/serverfault so here I am.

So my questions are:

1. Is all of this ok? Am I supposed to be checking on my services regularly myself if I want them to run non-stop or is there some way to configure Flex in such way that it restarts automatically?

2. What is that ZONE_RESOURCE_POOL_EXHAUSTED error code, can it be related to the crash of my service and is it possible to do anything about it (from what I've googled so far, it isn't, but here's hoping).

3. Did you have similar issues with App Engine and did you manage to resolve them?

4. And last, but not least, does GCP owe us money for the downtime if it was their fault and if yes, how to check and/or prove it was their fault?

Sorry if this isn't strict enough question post, but I'm not sure where else can I post to get any help. If you know a better way, please let me know.

Answer 1

• Is all of this ok? Am I supposed to be checking on my services regularly myself if I want them to run non-stop or is there some way to configure Flex in such way that it restarts automatically?

You can use one of GCP services, Google Cloud Monitoring to get data you deem important for you and set up alerts when the is a significant change on your instance.

• What is that ZONE_RESOURCE_POOL_EXHAUSTED error code, can it be related to the crash of my service and is it possible to do anything about it (from what I've googled so far, it isn't, but here's hoping).

ZONE_RESOURCE_POOL_EXHAUSTED error code happens when Google Cloud resources you need are not currently available. It is recommended that you deploy and balance your workload across multiple zones and regions to reduce the likelihood of an outage and have access to multiple resource pools when you need to expand quickly. Google public documentation outlines how to build resilient and scalable architectures on Google Cloud Platform. Additionally, you can use Google Cloud on-demand that guarantees Google Cloud capacity.

• And last, but not least, does GCP owe us money for the downtime if it was their fault and if yes, how to check and/or prove it was their fault?

You can use Google Cloud Support, if you have subscribed to any paid support else you can file a Public Issue tracker (PIT) whenever you have an issue free of cost. Lastly, you don't have to prove that it is anyone's fault. Google have multiple to adjust credit for their customers as soon they the identify the root cause of the issue.