Made a mistake in a puppet config and inadvertently let puppet upgrade a kubernetes cluster to 1.13.0. Since k8 wants a somewhat involved upgrade process the whole thing was fairly botched.
After running the process manually and getting all impacted nodes and master to the same version it seems that kubelet (and thus the api server) will run for 20-30 minutes and then stop responding.
As it starts to get unhappy the log begins seeing more entries like these:
level=error msg="Error running exec in container: rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused \"process_linux.go:110: decoding init error from pipe caused \\"read parent: connection reset by peer\\"\"\n"
file.go:182] Not recursing into manifest path "/etc/kubernetes/manifests/<path>"
with various namespace entries for path. This cycle repeats every 20 seconds until kubelet eventually stops responding. If I manually restart kubelet it works for a while and begins the cycle anew.
Did I miss something in the upgrade process?
[root@kube-master ~]# rpm -qa | grep docker docker-client-1.13.1-84.git07f3374.el7.centos.x86_64 docker-1.13.1-84.git07f3374.el7.centos.x86_64 docker-common-1.13.1-84.git07f3374.el7.centos.x86_64