0

I have a complicated system deployed in k8s, and in one of the pods, there are 2 containers, mongo and metrics. some related info from kubectl -n app_ns describe pod app-mongo is as below(removed some un-related info):

  containers:
  - args:
    - --config=/data/configdb/mongod.conf
    - --dbpath=/data/db
    - --replSet=app-mongo
    - --port=27017
    - --bind_ip=0.0.0.0
    - --auth
    - --keyFile=/data/configdb/key.txt
    command:
    - mongod
    image: mongo:3.6.8
    imagePullPolicy: IfNotPresent
    livenessProbe:
      exec:
        command:
        - mongo
        - --eval
        - db.adminCommand('ping')
      failureThreshold: 3
      initialDelaySeconds: 30
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 5
    name: mongodb
    ports:
    - containerPort: 27017
      name: mongodb
      protocol: TCP
    readinessProbe:
      exec:
        command:
        - mongo
        - --eval
        - db.adminCommand('ping')
      failureThreshold: 3
      initialDelaySeconds: 5
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 1

  - command:
    - sh
    - -ec
    - /bin/mongodb_exporter -mongodb.uri mongodb://$METRICS_USER:$METRICS_PASSWORD@localhost:27017
      -mongodb.socket-timeout=15s -mongodb.sync-timeout=1m -web.metrics-path=/metrics
      -web.listen-address=:9216
    image: ssalaues/mongodb-exporter:0.6.1
    imagePullPolicy: IfNotPresent
    livenessProbe:
      exec:
        command:
        - sh
        - -ec
        - /bin/mongodb_exporter -mongodb.uri mongodb://$METRICS_USER:$METRICS_PASSWORD@localhost:27017
          -test
      failureThreshold: 3
      initialDelaySeconds: 30
      periodSeconds: 14
      successThreshold: 1
      timeoutSeconds: 23
    name: metrics

now, the problem is, the pod app-mongo will restart from time to time.

kubectl -n app-ns get pod -w
NAME        READY   STATUS     RESTARTS   AGE
mongodb-0   2/2     Running    0          46s
mongodb-0   2/2     Running    1          2m19s
mongodb-0   2/2     Running           2          4m52s
mongodb-0   2/2     Running           3          6m57s
mongodb-0   2/2     Running           4          9m47s
mongodb-0   2/2     Running           5          12m
mongodb-0   2/2     Running           6          14m
mongodb-0   1/2     CrashLoopBackOff   6          16m
mongodb-0   2/2     Running            7          18m

you can see, it restarted many times, and the time seems random. Sometime it will restart after 4 or 5 monutes, sometime it will restart after half an hour.

And I noticed that, the metrics comtainer will restart like this. but the mongo container seems ok. it will just restart 1 or 2 times in a day.

and I also go into the metrics container like this:

kubectl -n app_ns exec -it  app-mongo  --container=metrics sh

and in the shell, I run:

/bin/mongodb_exporter -mongodb.uri mongodb://root:the_password@localhost:27017 -test

Sometime it will return the correct message, but sometime, it will return:

ERRO[0000] Cannot connect to server using url mongodb://****:****@localhost:27017: no reachable servers  source="connection.go:84"
ERRO[0000] Can't connect to MongoDB: Cannot connect using uri: mongodb://root:mongopassw0rd@localhost:27017  source="mongodb_exporter.go:104"
/

And in this time, I found that it can not connect to localhost 27017 with telnet.

And, the docker version is 19.03, kubernetes version is 1.15.11. OS is centos 7.6.

I have checked many log, and status, and anything I can think, but cannot find any problems. If you need anything else to diagnose the problem, I can provide. Any suggestion and clue will be very helpful. Thanks.

  • That's a tough one; you may want to check the docker logs for the sandbox pod (it's the one running the pause: image), of course check the kubelet logs, and for debugging purposes you may want to run a 2nd Pod consisting of nginx and curl localhost in a loop, to see if this is something specific to the mongo+exporter combination, or something bigger affecting the whole Node – mdaniel Jun 20 at 16:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.