1

I'm managing an Elasticsearch deployment in Kubernetes. I see that the disk storage is getting close to full, so I would like to increase the size of the persistent volumes.

I want to change this value in the Stateful Set:

spec.
volumeClaimTemplates[0].spec.resources.requests.storage : "10Gi"

But when I do this using the Kubernetes-dashboard, I get the following message:

Internal server error

StatefulSet.apps "es-cluster" is invalid: spec: Forbidden: updates to statefulset > spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden.

This makes me think that I will have to delete my existing Stateful Set and deploy a new one.

Is it possible to increase the per-pod disk storage without a service interruption or loss of data?

Additional detail

I have multiple Elasticsearch data pods and am using replica count=1, so if I am able to take them down and upgrade the disk storage one pod at a time, there shouldn't be a problem. I just don't see how to do this given the restriction indicated above.

1

Even tho Resizing Persistent Volumes using Kubernetes from Kubernetes 1.11, there seems to be some issues with it.

As discussed in GitHub: StatefulSet: support resize pvc storage in K8s v1.11 #68737

Due to this limitation, many database Operators for Kubernetes don't support PVC resizing. It is a critical issue because when your database becomes bigger than you expected - you have no choice and it is needed to backup DB and recreate new DB from the backup.

You should resize it by deleting the Statefulset, that would mean you will delete all Pods and it will cause downtime.

A work around was posted by DaveWHarvey

I got around this limitation on elasticsearch because of work I had to do to avoid EBS volumes being unable to be assigned because they are in the wrong availability zone, i.e. I had created a statefulset per AZ. If I want to change some storage characteristic, I create a new "AZ" using the same storage class, and then migrate all the data to pods in that new AZ, then destroy the old AZ.

Hope this helps you a bit.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.