The zone does not have enough resources available to fulfill the request

Question

Been receiving The zone 'projects/PROJECT/zones/australia-southeast1-c' does not have enough resources available to fulfill the request. Try a different zone, or try again later. for many hours now when attempting to increase the size of a GKE cluster.

This error is also returned when starting up a compute vm instance in the australia-southeast1-c zone.

I have checked project quotas. I have been researching and it seems this is due to capacity in the data centres.

What is the expected strategy for handling this error (besides trying again later)?

Will a regional cluster eliminate this problem or is it possible to receive this error across all zones in the austraila-souteast region?

Answer 1

You have correctly diagnosed this error and the message given is really the best advice.

Ultimately platforms like GCP despite their on demand 'cloud' nature are still comprised of physical data centres with finite resources underneath.

If there is lots of demand for particular configurations or services in a specific zone it can be possible to exhaust capacity. In these situations you have a few choices:

1. Wait and try later - systems are being started and stopped all the time and so capacity can simply become available again due to a drop in demand. Also the provider - Google in this case - will have seen that the region is reaching capacity and will be taking steps to expand that capacity. Though keep in mind procuring and installing new equipment is still a very physical activity and can take many days or weeks.
2. Change what you are asking for - it could be that by changing the configuration you request resources that are less in demand and so still have capacity.
3. Move to another location, either another zone or even an entire region - clearly zones are easier than regions.

Your idea of using a regional cluster (or a even a multi-zone cluster) is a sound one in theory, though keep in mind you cannot specify the number of nodes in a particular zone, instead GCP creates the same number in each zone, so you may still hit capacity issues in the zone with constraints, although you could have the option to simply not use that zone at that point.

Ordinarily these kind of capacity issues are rare, certainly Google will be monitoring overall consumption to ensure there is enough headroom and if there are spikes hitting capacity they will be for short periods and typically processes to add capacity will have been triggered long in advance. That being said in these tricky COVID-19 times, many providers are seeing a surge in utilisation which was not predicted - this coupled with potentially a slower supply chain to add capacity may mean that it will be longer to mitigate such issues.