1

I can run the exact same command over and over again and sometimes it works, sometimes it doesn't:

root@bak-rrk9m:/# mysql -Bsuroot -p"$MYSQL_ROOT_PASSWORD" -h"$KMB_MARIADB_SERVICE_SERVICE_HOST" -e "show databases"
ERROR 2002 (HY000): Can't connect to MySQL server on '10.245.54.251' (115)
root@bak-rrk9m:/# mysql -Bsuroot -p"$MYSQL_ROOT_PASSWORD" -h"$KMB_MARIADB_SERVICE_SERVICE_HOST" -e "show databases"
#mysql50#lost+found
busman
demo_busman
information_schema
mysql
performance_schema
root@bak-rrk9m:/# mysql -Bsuroot -p"$MYSQL_ROOT_PASSWORD" -h"$KMB_MARIADB_SERVICE_SERVICE_HOST" -e "show databases"
#mysql50#lost+found
busman
demo_busman
information_schema
mysql
performance_schema
root@bak-rrk9m:/# mysql -Bsuroot -p"$MYSQL_ROOT_PASSWORD" -h"$KMB_MARIADB_SERVICE_SERVICE_HOST" -e "show databases"
#mysql50#lost+found
busman
demo_busman
information_schema
mysql
performance_schema
root@bak-rrk9m:/# mysql -Bsuroot -p"$MYSQL_ROOT_PASSWORD" -h"$KMB_MARIADB_SERVICE_SERVICE_HOST" -e "show databases"
#mysql50#lost+found
busman
demo_busman
information_schema
mysql
performance_schema
root@bak-rrk9m:/# mysql -Bsuroot -p"$MYSQL_ROOT_PASSWORD" -h"$KMB_MARIADB_SERVICE_SERVICE_HOST" -e "show databases"
#mysql50#lost+found
busman
demo_busman
information_schema
mysql
performance_schema
root@bak-rrk9m:/# mysql -Bsuroot -p"$MYSQL_ROOT_PASSWORD" -h"$KMB_MARIADB_SERVICE_SERVICE_HOST" -e "show databases"
#mysql50#lost+found
busman
demo_busman
information_schema
mysql
performance_schema
root@bak-rrk9m:/# mysql -Bsuroot -p"$MYSQL_ROOT_PASSWORD" -h"$KMB_MARIADB_SERVICE_SERVICE_HOST" -e "show databases"
ERROR 2002 (HY000): Can't connect to MySQL server on '10.245.54.251' (115)
root@bak-rrk9m:/# mysql -Bsuroot -p"$MYSQL_ROOT_PASSWORD" -h"$KMB_MARIADB_SERVICE_SERVICE_HOST" -e "show databases"
ERROR 2002 (HY000): Can't connect to MySQL server on '10.245.54.251' (115)
root@bak-rrk9m:/# mysql -Bsuroot -p"$MYSQL_ROOT_PASSWORD" -h"$KMB_MARIADB_SERVICE_SERVICE_HOST" -e "show databases"
ERROR 2002 (HY000): Can't connect to MySQL server on '10.245.54.251' (115)
root@bak-rrk9m:/# mysql -Bsuroot -p"$MYSQL_ROOT_PASSWORD" -h"$KMB_MARIADB_SERVICE_SERVICE_HOST" -e "show databases"
ERROR 2002 (HY000): Can't connect to MySQL server on '10.245.54.251' (115)
root@bak-rrk9m:/# mysql -Bsuroot -p"$MYSQL_ROOT_PASSWORD" -h"$KMB_MARIADB_SERVICE_SERVICE_HOST" -e "show databases"
#mysql50#lost+found
busman
demo_busman
information_schema
mysql
performance_schema
root@bak-rrk9m:/# mysql -Bsuroot -p"$MYSQL_ROOT_PASSWORD" -h"$KMB_MARIADB_SERVICE_SERVICE_HOST" -e "show databases"
ERROR 2002 (HY000): Can't connect to MySQL server on '10.245.54.251' (115)

My website, OTOH, doesn't seem to be getting any errors.

What's going on? How do I fix this?

It's the same issue whether I use the hostname or internal/cluster IP address.


Got some logs from the MariaDB pod. This might be it:

2020-06-15 0:51:51 12069 [Warning] Aborted connection 12069 to db: 'demo_kmbookings' user: 'root' host: '10.244.0.84' (Got an error writing communication packets)


Found some article with some suggestions but nothing concreate.

Thought maybe it was running out of RAM while creating those dumps since I cheaped out and only gave it 1 GiB or so. Fortunately Kubernetes made it easy to rebuild my whole cluster on a bigger node, but that didn't help.

Noticed now that my MariaDB service selector didn't actually match my deploy template. I fixed that and now it's running successfully. Will have to try a few more times to be sure, but it makes me wonder how this ever worked.

  • Well, 115? that's an odd "should never happen" sort of error. You tagged this kubernetes. Can you provide your pod definition? e.g. kubectl get pod my-pod -o yaml Is your cluster running ok? See this document for troubleshooting. – Michael Hampton Jun 15 at 0:43
  • @MichaelHampton Here's my backup pod: mpen.xyz/share/2020/06/2020-06-14_17-54-30.txt My cluster seems fine. I have a website running on the cluster connected to the same DB, and I haven't noticed any errors with it. I added an error for the mariadb pod to the question. – mpen Jun 15 at 0:56
  • 1
    Nice of them to not actually mention what the error was! – Michael Hampton Jun 15 at 1:12
  • Did you notice any pod restarts on your mariadb pods? That would explain intermittent connectivity issues since you have replicas: 1; also, FWIW, it's dangerous to use Deployment with a stateful Pod -- that's why StatefulSet exists and is used for databases – mdaniel Jun 15 at 2:08
  • 1
    There are some subtle differences between Deployments and StatefulSets, but the biggest differences are that if you try to scale the Deployment to 2 replicas, it will never work because only one Pod can have that PV at a time, and if you try to change the image or environment variable in your Deployment, it will create a ReplicaSet which will never start because the new Pods also cannot bind to that PV – mdaniel Jun 15 at 15:50
0

I'm pretty sure this was a Kubernetes configuration issue. My MariaDB deployment looked like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mariadb-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      pod: b437e465-2526-41bb-ae19-534b3a60f2eb
  template:
    metadata:
      labels:
        pod: b437e465-2526-41bb-ae19-534b3a60f2eb
    spec:
      containers:
        - name: mariadb
          image: mariadb
          ports:
            - containerPort: 3306
          envFrom:
            - secretRef:
                name: mariadb-env
          volumeMounts:
            - name: mariadb-volume
              mountPath: /var/lib/mysql
            - name: config-volume
              mountPath: /etc/mysql/mariadb.conf.d/zzz-kymark.cnf
              subPath: my.cnf
      volumes:
        - name: mariadb-volume
          persistentVolumeClaim:
            claimName: mariadb-pvc
        - name: config-volume
          configMap:
            name: mariadb-config

But my service definition looked like this:

apiVersion: v1
kind: Service
metadata:
  name: mariadb-service
spec:
  selector:
    app: kymark-mariadb-pod
  ports:
    - protocol: TCP
      port: 3306

Notice the selector doesn't match.

I haven't studied up on how that all works yet, but I assume if the service selector doesn't match the pod definition, then Kubernetes doesn't know how to set up the network properly.

What I don't get is why it would still work sometimes. Where was Kubernetes routing all the traffic to? Why did my website still work?

I guess the lessons here are:

  1. Check the logs for your database. The client error doesn't contain much information.
  2. Double-check your Kubernetes config and selectors

There were no errors to suggest my config was wrong, but here we are.

| improve this answer | |
0

The selector used in service spec must match with the deployment spec, otherwise your service won't redirect the traffic for the correct pod.

Is common some people use the same selector for various application inside the cluster in the beggining because they don't understand how it works.

I've extracted some points from documentation:

The .spec.selector field defines how the Deployment finds which Pods to manage. In this case, you simply select a label that is defined in the Pod template (app: nginx). However, more sophisticated selection rules are possible, as long as the Pod template itself satisfies the rule.

Note: You must specify an appropriate selector and Pod template labels in a Deployment (in this case, app: nginx). Do not overlap labels or selectors with other controllers (including other Deployments and StatefulSets). Kubernetes doesn't stop you from overlapping, and if multiple controllers have overlapping selectors those controllers might conflict and behave unexpectedly.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.