1

I deployed an Elasticsearch cluster on GKE, using this project as a starting point: https://github.com/pires/kubernetes-elasticsearch-cluster

I would like to configure two Curator jobs by using Kubernetes CronJob objects, to snapshot the indices, and then to delete/prune old indices.

I would like to store the snapshots on a GCS bucket. I created a new service account and downloaded the JSON credentials key, to be used with the elasticsearch keystore. See plugin docs: https://www.elastic.co/guide/en/elasticsearch/plugins/current/repository-gcs-usage.html

I am not sure how/where to add this key, to be used by the Curator CronJob performing the backup. The Elasticsearch documentation mentions running the elasticsearch-keystore binary on the credentials key file.

curator.yaml:

apiVersion: batch/v1beta1
    kind: CronJob
    metadata:
      name: curator
    spec:
      schedule: 0 11 * * *
      jobTemplate:
        spec:
          template:
            spec:
              containers:
              - name: curator
                image: quay.io/pires/docker-elasticsearch-curator:5.4.1
                args:
                - --config
                - /etc/config/config.yml
                - /etc/config/action_file.yml
                env:
                  - name:
                volumeMounts:
                  - name: config-volume
                    mountPath: /etc/config
              volumes:
                - name: config-volume
                  configMap:
                    name: curator-config
              restartPolicy: OnFailure

curator-config.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: curator-config
data:
  action_file.yml:
    # Remember, leave a key empty if there is no value.  None will be a string,
    # not a Python "NoneType"
    #
    # Also remember that all examples have 'disable_action' set to True.  If you
    # want to use this action as a template, be sure to set this to False after
    # copying it.
    actions:
  1:
    action: snapshot
    options:
      repository: gcs_repository
      name: ${SNAPSHOT_NAME:snapshot-%Y-%m-%d}
      continue_if_exception: false
    filters:
      - filtertype: age
        source: name
        direction: older
        timestring: '%Y-%m-%d'
        unit: days
        unit_count: ${DAYS}
  2:
    action: delete_indices
    options:
      continue_if_exception: false
    filters:
      - filtertype: age
        source: name
        direction: older
        timestring: '%Y-%m-%d'
        unit: days
        unit_count: ${DAYS}

$kubectl get pods

NAME                                             READY     STATUS    RESTARTS   AGE
cerebro-59648dc47c-vr964                         1/1       Running   0          25d
es-client-7bff44b8f5-2wqcs                       1/1       Running   0          12d
es-client-7bff44b8f5-vnrhg                       1/1       Running   0          12d
es-data-0                                        1/1       Running   0          52d
es-data-1                                        1/1       Running   0          52d
es-data-2                                        1/1       Running   0          52d
es-master-6bf767f949-8fpjl                       1/1       Running   0          52d
es-master-6bf767f949-brjpq                       1/1       Running   0          52d
es-master-6bf767f949-gx2jp                       1/1       Running   0          52d
fluentd-gcp-v2.0-7mncl                           1/1       Running   0          43m
fluentd-gcp-v2.0-rsfmc                           1/1       Running   0          43m
fluentd-gcp-v2.0-tbh9t                           1/1       Running   0          43m
kibana-595858b4b7-5npcr                          1/1       Running   0          52d
nginx-ingress-controller-86c8447687-z4rjq        1/1       Running   2          52d
nginx-ingress-default-backend-6664bc64c9-q2hnm   1/1       Running   338        52d 

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.