0

We have configured vault to run as a pod in the cluster. In the below deployment YAML file, we have included the vault initialisation and unsealing to happen when the pod comes up initially. But when the pod gets restarted, the pod is going to crashLoopBackOff state because the vault is getting reinitialised. This is because we have included both the initialisation and unsealing command in the postStart lifecycle command of the deployment file. Is there any way in which we could initialise the pod only once and later when pod restarts, unseal the vault using the existing keys?

Deployment file:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: vault
  name: vault
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: vault
    spec:
      containers:
      - image: vault
        name: vault
        imagePullPolicy: Always
        ports:
        - containerPort: 8200
          name: vaultport
          protocol: TCP
        securityContext:
         capabilities:
           add:
             - IPC_LOCK
        env:
        - name: VAULT_ADDR
          value: "http://0.0.0.0:8200"
        command: ["vault", "server"]
        args: 
          - "-config=/vault/config/config.hcl"
        volumeMounts:
          - name: vault-unseal
            mountPath: /vault/file/unseal.sh
            subPath: unseal.sh
          - name: vault-config
            mountPath: /vault/config/config.hcl
            subPath: config.hcl
        lifecycle:
          postStart:
            exec:
              command: ["/bin/sh", "-c", "vault operator init > /vault/file/keys.txt; sh /vault/file/unseal.sh" ]
      volumes:
      - name: vault-unseal
        configMap:
          name: vault-unseal
      - name: vault-config
        configMap:
          name: vault-config 
      imagePullSecrets:
      - name: regcred

Output of kubectl describe pod:

Name:           vault-677bfd9c9c-dwsgv
Namespace:      xxx
Priority:       0
Node:           xxxxxxx-5b587f98-ljf4/10.0.0.11
Start Time:     Thu, 30 Jan 2020 06:26:21 +0000
Labels:         app=vault
                pod-template-hash=677bfd9c9c
Annotations:    <none>
Status:         Running
IP:             10.4.2.10
IPs:            <none>
Controlled By:  ReplicaSet/vault-677bfd9c9c
Containers:
  vault:
    Container ID:  xxxxxxxxxxx
    Image:         xxxxxxxxxxxxxxxx
    Image ID:      xxxxxxxxxxxxxxxxxxxxxxxxx
    Port:          8200/TCP
    Host Port:     0/TCP
    Command:
      vault
      server
    Args:
      -config=/vault/config/config.hcl
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 30 Jan 2020 06:26:26 +0000
      Finished:     Thu, 30 Jan 2020 06:26:27 +0000
    Ready:          False
    Restart Count:  1
    Environment:
      VAULT_ADDR:  http://0.0.0.0:8200
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-kxfdb (ro)
      /vault/config/config.hcl from vault-config (rw,path="config.hcl")
      /vault/file from vault-data (rw)
      /vault/file/unseal.sh from vault-unseal (rw,path="unseal.sh")
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  vault-unseal:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      vault-unseal
    Optional:  false
  vault-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      vault-config
    Optional:  false
  vault-data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  vault-data
    ReadOnly:   false
  default-token-kxfdb:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-kxfdb
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason               Age                From                                                          Message
  ----     ------               ----               ----                                                          -------
  Normal   Scheduled            18s                default-scheduler                                             Successfully assigned xxx/xxxxxxxxxx
  Normal   Pulling              13s (x2 over 15s)  kubelet, gke-cluster-testing--np-testing-featu-5b587f98-ljf4  pulling image "xxxxxxxxx"
  Normal   Pulled               13s (x2 over 15s)  kubelet, gke-cluster-testing--np-testing-featu-5b587f98-ljf4  Successfully pulled image "xxxxxxx"
  Normal   Created              13s (x2 over 15s)  kubelet, gke-cluster-testing--np-testing-featu-5b587f98-ljf4  Created container
  Normal   Started              13s (x2 over 14s)  kubelet, gke-cluster-testing--np-testing-featu-5b587f98-ljf4  Started container
  Warning  FailedPostStartHook  12s (x2 over 14s)  kubelet, gke-cluster-testing--np-testing-featu-5b587f98-ljf4  Exec lifecycle hook ([/bin/sh -c vault operator init > /vault/file/keys.txt; sh /vault/file/unseal.sh]) for Container "vault" in Pod "vault-677bfd9c9c-dwsgv_xxx(6ebdc17a-4329-11ea-9fc1-4201c0a80004)" failed - error: command '/bin/sh -c vault operator init > /vault/file/keys.txt; sh /vault/file/unseal.sh' exited with 2: Error initializing: Error making API request.

URL: PUT http://0.0.0.0:8200/v1/sys/init
Code: 400. Errors:

* Vault is already initialized
An error occurred attempting to ask for an unseal key. The raw error message
is shown below, but usually this is because you attempted to pipe a value
into the unseal command or you are executing outside of a terminal (tty). You
should run the unseal command from a terminal for maximum security. If this
is not an option, the unseal key can be provided as the first argument to the
unseal command. The raw error was:  file descriptor 0 is not a terminal
An error occurred attempting to ask for an unseal key. The raw error message
is shown below, but usually this is because you attempted to pipe a value
into the unseal command or you are executing outside of a terminal (tty). You
should run the unseal command from a terminal for maximum security. If this
is not an option, the unseal key can be provided as the first argument to the
unseal command. The raw error was:  file descriptor 0 is not a terminal
An error occurred attempting to ask for an unseal key. The raw error message
is shown below, but usually this is because you attempted to pipe a value
into the unseal command or you are executing outside of a terminal (tty). You
should run the unseal command from a terminal for maximum security. If this
is not an option, the unseal key can be provided as the first argument to the
unseal command. The raw error was:  file descriptor 0 is not a terminal
Token (will be hidden):
Error authenticating: An error occurred attempting to ask for a token. The raw error message is shown below, but usually this is because you attempted to pipe a value into the command or you are executing outside of a terminal (tty). If you want to pipe the value, pass "-" as the argument to read from stdin. The raw error was: file descriptor 0 is not a terminal
, message: "Unseal Key (will be hidden): \nUnseal Key (will be hidden): \nUnseal Key (will be hidden): \nKey                Value\n---                -----\nSeal Type          shamir\nInitialized        true\nSealed             true\nTotal Shares       5\nThreshold          3\nUnseal Progress    0/3\nUnseal Nonce       n/a\nVersion            1.3.2\nHA Enabled         false\n++++++++++++ Vault Status +++++++++\nKey                Value\n---                -----\nSeal Type          shamir\nInitialized        true\nSealed             true\nTotal Shares       5\nThreshold          3\nUnseal Progress    0/3\nUnseal Nonce       n/a\nVersion            1.3.2\nHA Enabled         false\nError initializing: Error making API request.\n\nURL: PUT http://0.0.0.0:8200/v1/sys/init\nCode: 400. Errors:\n\n* Vault is already initialized\nAn error occurred attempting to ask for an unseal key. The raw error message\nis shown below, but usually this is because you attempted to pipe a value\ninto the unseal command or you are executing outside of a terminal (tty). You\nshould run the unseal command from a terminal for maximum security. If this\nis not an option, the unseal key can be provided as the first argument to the\nunseal command. The raw error was:  file descriptor 0 is not a terminal\nAn error occurred attempting to ask for an unseal key. The raw error message\nis shown below, but usually this is because you attempted to pipe a value\ninto the unseal command or you are executing outside of a terminal (tty). You\nshould run the unseal command from a terminal for maximum security. If this\nis not an option, the unseal key can be provided as the first argument to the\nunseal command. The raw error was:  file descriptor 0 is not a terminal\nAn error occurred attempting to ask for an unseal key. The raw error message\nis shown below, but usually this is because you attempted to pipe a value\ninto the unseal command or you are executing outside of a terminal (tty). You\nshould run the unseal command from a terminal for maximum security. If this\nis not an option, the unseal key can be provided as the first argument to the\nunseal command. The raw error was:  file descriptor 0 is not a terminal\nToken (will be hidden): \nError authenticating: An error occurred attempting to ask for a token. The raw error message is shown below, but usually this is because you attempted to pipe a value into the command or you are executing outside of a terminal (tty). If you want to pipe the value, pass \"-\" as the argument to read from stdin. The raw error was: file descriptor 0 is not a terminal\n"
  Normal   Killing  12s (x2 over 14s)  kubelet, gke-cluster-testing--np-testing-featu-5b587f98-ljf4  Killing container with id docker://vault:FailedPostStartHook
  Warning  BackOff  10s (x2 over 11s)  kubelet, gke-cluster-testing--np-testing-featu-5b587f98-ljf4  Back-off restarting failed container
1
  • Could you also share Kubernetes version and some details about env (On-Prem, local)? – PjoterS Jan 31 '20 at 12:55
1

Yes, in this case you need to make your script smarter to check first if the vault is already initialized. Here is a very simple example with a bash script.

Also, a common pattern for achieving this is by using a sidecar (that eventually goes to "sleep" indefinitely), but the postStart hook should also work as long as you make your script resilient enough considering that in some cases it will run before the vault container is up and will end up killing it and restarting until they both eventually sync up.

I'd highly recommend this talk by Seth Vargo where auto-initialization is done with a more elaborate Golang program.

1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.