I am being billed for node usage if I cordon and drain the node?

Question

I am learning GKE (Google Kubernetes Engine), and I have a cluster with 2 node pools:

➜  ~ gcloud container node-pools list
NAME                MACHINE_TYPE   DISK_SIZE_GB  NODE_VERSION
pool-2              n1-standard-1  10            1.14.10-gke.24
pool-n1-standard-2  n1-standard-2  10            1.14.10-gke.24

I have 1 node on each node pool:

➜  ~ kubectl get node
NAME                                             STATUS   ROLES    AGE     VERSION
gke-cluster-1-pool-2-bec144d8-rqb8               Ready    <none>   8d      v1.14.10-gke.24
gke-cluster-1-pool-n1-standard-2-892e9394-b8x8   Ready    <none>   5h11m   v1.14.10-gke.24

➜  ~ kubectl describe node | grep -i cloud.google.com/gke-nodepool
                    cloud.google.com/gke-nodepool=pool-2
                    cloud.google.com/gke-nodepool=pool-n1-standard-2

I want my pods running for now on a n1-standard-1 machine, while I develop the system, so I cordoned the more expensive node and then drained it:

➜  ~ kubectl cordon gke-cluster-1-pool-n1-standard-2-892e9394-b8x8
node/gke-cluster-1-pool-n1-standard-2-892e9394-b8x8 cordoned

➜  ~ kubectl drain gke-cluster-1-pool-n1-standard-2-892e9394-b8x8
node/gke-cluster-1-pool-n1-standard-2-892e9394-b8x8 already cordoned
node/gke-cluster-1-pool-n1-standard-2-892e9394-b8x8 drained

After that I expect that all pods get evicted, but listing pods on this node still shows some "system" pod running:

➜  ~ kubectl get pods --all-namespaces -o wide --field-selector spec.nodeName=gke-cluster-1-pool-n1-standard-2-892e9394-b8x8
NAMESPACE     NAME                                                        READY   STATUS    RESTARTS   AGE     IP           NODE                                             NOMINATED NODE   READINESS GATES
kube-system   kube-proxy-gke-cluster-1-pool-n1-standard-2-892e9394-b8x8   1/1     Running   0          5h22m   10.138.0.7   gke-cluster-1-pool-n1-standard-2-892e9394-b8x8   <none>           <none>

So my question are:

• what is this pod?
• and more important I am being charged the full cost of running a n1-standard-2 machine as stated in the pricing page? ($0.0950/Hour as of now)

Answer 1

1. kube-proxy is a network proxy that runs in all GKE nodes. In GKE, it has Iptables based rules to allow network communication of pods in the cluster. You can connect to the pod with kubectl exec -ti -n kube-system KUBE-PROXY-POD-NAME /bin/sh and execute iptables -L -n to see the Iptables rules. More on kube-proxy.

2. Pricing - yes Google will charge you full cost for any running instance. It doesn't matter whether you have an application or pod running in it or not. The fee is for the reserved compute/memory/storage of the VM. If you don't want to be charged a fee, delete the instance. You can also lower your fee by using preemptible vms for gke nodes.