We have an api in running which receives once a day multiple batches of large data that are inserted in a mongodb.
We use the cvallance/mongo-k8s-sidecar
for the replicationset configuration
This works perfectly on a local mongodatabase.
there is also no production traffic on the database which could raise raise conditions or so.
Now we deployed it to a google container engine. There the import works in general too. But from time to time we got timeoutexceptions like this:
Cannot run replSetReconfig because the node is currently updating its configuration
or
MongoDB.Driver.MongoCommandException: Command insert failed: BSONObj size: 16793637 (0x1004025) is invalid. Size must be between 0 and 16793600(16MB) First element: insert: "LandingPageConnectionSet_Stage".
or
Error in workloop { MongoError: connection 0 to 127.0.0.1:27017 timed out at Function.MongoError.create (/opt/cvallance/mongo-k8s-sidecar/node_modules/mongodb-core/lib/error.js:29:11) at Socket. (/opt/cvallance/mongo-k8s-sidecar/node_modules/mongodb-core/lib/connection/connection.js:198:20) at Object.onceWrapper (events.js:254:19) at Socket.emit (events.js:159:13) at Socket._onTimeout (net.js:411:8) at ontimeout (timers.js:478:11) at tryOnTimeout (timers.js:302:5) at Timer.listOnTimeout (timers.js:262:5)
I can see that the cpu seems to not be at its limits.
Kubernetes configuration for mongodb
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: fast
provisioner: kubernetes.io/gce-pd
parameters:
type: pd-ssd
---
apiVersion: v1
kind: Service
metadata:
name: mongo
labels:
name: mongo
spec:
ports:
- port: 27017
targetPort: 27017
clusterIP: None
selector:
role: mongo
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: mongo
spec:
serviceName: "mongo"
replicas: 3
template:
metadata:
labels:
role: mongo
environment: test
spec:
terminationGracePeriodSeconds: 10
containers:
- name: mongo
image: mongo:3.6
command:
- mongod
- "--replSet"
- rs0
- "--bind_ip"
- 0.0.0.0
- "--smallfiles"
- "--noprealloc"
ports:
- containerPort: 27017
volumeMounts:
- name: mongo-persistent-storage
mountPath: /data/db
- name: mongo-sidecar
image: cvallance/mongo-k8s-sidecar
env:
- name: MONGO_SIDECAR_POD_LABELS
value: "role=mongo,environment=test"
volumeClaimTemplates:
- metadata:
name: mongo-persistent-storage
annotations:
volume.beta.kubernetes.io/storage-class: "fast"
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 32Gi
we also little changed the config by limitting the wiretiger cachesize and removing the smallfiles options so the part in the config looked like this:
- mongod
- "--replSet"
- rs0
- "--bind_ip"
- 0.0.0.0
- "--noprealloc"
- "--wiredTigerCacheSizeGB"
- "1.5"