1

We have an api in running which receives once a day multiple batches of large data that are inserted in a mongodb. We use the cvallance/mongo-k8s-sidecar for the replicationset configuration

This works perfectly on a local mongodatabase.

there is also no production traffic on the database which could raise raise conditions or so.

Now we deployed it to a google container engine. There the import works in general too. But from time to time we got timeoutexceptions like this:

Cannot run replSetReconfig because the node is currently updating its configuration

or

MongoDB.Driver.MongoCommandException: Command insert failed: BSONObj size: 16793637 (0x1004025) is invalid. Size must be between 0 and 16793600(16MB) First element: insert: "LandingPageConnectionSet_Stage".

or

Error in workloop { MongoError: connection 0 to 127.0.0.1:27017 timed out at Function.MongoError.create (/opt/cvallance/mongo-k8s-sidecar/node_modules/mongodb-core/lib/error.js:29:11) at Socket. (/opt/cvallance/mongo-k8s-sidecar/node_modules/mongodb-core/lib/connection/connection.js:198:20) at Object.onceWrapper (events.js:254:19) at Socket.emit (events.js:159:13) at Socket._onTimeout (net.js:411:8) at ontimeout (timers.js:478:11) at tryOnTimeout (timers.js:302:5) at Timer.listOnTimeout (timers.js:262:5)

I can see that the cpu seems to not be at its limits.

Kubernetes configuration for mongodb

---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: fast
provisioner: kubernetes.io/gce-pd
parameters:
  type: pd-ssd
---
apiVersion: v1
kind: Service
metadata:
  name: mongo
  labels:
    name: mongo
spec:
  ports:
  - port: 27017
    targetPort: 27017
  clusterIP: None
  selector:
    role: mongo
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: mongo
spec:
  serviceName: "mongo"
  replicas: 3
  template:
    metadata:
      labels:
        role: mongo
        environment: test
    spec:
      terminationGracePeriodSeconds: 10
      containers:
        - name: mongo
          image: mongo:3.6
          command:
            - mongod
            - "--replSet"
            - rs0
            - "--bind_ip"
            - 0.0.0.0
            - "--smallfiles"
            - "--noprealloc"
          ports:
            - containerPort: 27017
          volumeMounts:
            - name: mongo-persistent-storage
              mountPath: /data/db
        - name: mongo-sidecar
          image: cvallance/mongo-k8s-sidecar
          env:
            - name: MONGO_SIDECAR_POD_LABELS
              value: "role=mongo,environment=test"
  volumeClaimTemplates:
  - metadata:
      name: mongo-persistent-storage
      annotations:
        volume.beta.kubernetes.io/storage-class: "fast"
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 32Gi

we also little changed the config by limitting the wiretiger cachesize and removing the smallfiles options so the part in the config looked like this:

   - mongod
    - "--replSet"
    - rs0
    - "--bind_ip"
    - 0.0.0.0
    - "--noprealloc"
    - "--wiredTigerCacheSizeGB"
    - "1.5"
  • 1
    Hi, I would like to take a look in your logs and activities to check if I can spot the issue, if you want you can open a private Google issue and I will take a look into your project. Register it here specifying your project numeber issuetracker.google.com/issues/new?component=187164 and post in the comment the link (Disclaimer: I work for Google Cloud Platform Support) – GalloCedrone Mar 2 '18 at 14:15
  • 1
    @GalloCedrone I created it. issueid: 74101711 – Boas Enkler Mar 2 '18 at 18:17
1

I checked the logs and the kubernetes Dashboard with Boas Enkler.

In the Kubernetes dashboard regarding the status of the PODs there were the following hints:

Pod Name: kube-lego-*****-***     
Status: Evicted 
Reason: The node was low on resource: memory.

You could have retrieved the very same information through kubectl describe pod [podname]

Notice that quoting the documentation: "If the kubelet is unable to reclaim sufficient resources on the node, kubelet begins evicting Pods."

Therefore I believed that the error with Mongodb since it was working on premise without any issue, to doublecheck we went through the Kernel logs showed by the console serial output and we found:

Memory cgroup out of memory: Kill process 4**7 (mongod) score 1494 or sacrifice child
...
Memory cgroup out of memory: Kill process 1**8 (mongod) score 1538 or sacrifice child

We noticed as well that there was no Memory Request field in the YAML file of the deployment. This is an issue since it could happen that even if there are three nodes with no workload can happen that all the PODs are started on the very same node since they theoretically fit.

In order to mitigate this behaviour there are some possible solution:

  • Scale vertically the cluster and introduce memory request values

  • Instruct the mongodb process to consume an amount of memory smaller than the Requested one.

  • The introduction of memory limit is essential if you have more container running on the same node and you want to avoid that they are killed by it. Consider that in this way it will be killed sometimes even if there is still memory available on the node.

  • as a hint for other we scaled the cluster and don't get the error anymore but still the isseu could happen as it is not easily possible to limit mongodb ressources. there are some tasks in the pipeline of mongodb but currently there is no hint when or ever this will be done – Boas Enkler Apr 18 '18 at 11:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.