0

I have CoreOS stable v1122.2.0 installed.

I have etcd2 configured with tls and is working properly. i created the certificates based on https://github.com/coreos/etcd/tree/master/hack/tls-setup using sub-domains that I created for my servers instead of specific IP address for calico tls to work.

etcd2 and calcio-node are configured and work properly. now I want to configure Kubernetes. I used the instructions at https://coreos.com/kubernetes/docs/latest/deploy-master.html, for now I'm configuring only one coreos server.

when I start kubelet and execute journalctl -f -u kubelet I get the following messages:

 Sep 23 23:30:11 coreos-2.tux-in.com kubelet-wrapper[1473]: E0923 23:30:11.495381    1473 reflector.go:205] pkg/kubelet/kubelet.go:286: Failed to list *api.Node: Get http://127.0.0.1:8080/api/v1/nodes?fieldSelector=metadata.name%3Dcoreos-2.tux-in.com&resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
 Sep 23 23:30:11 coreos-2.tux-in.com kubelet-wrapper[1473]: E0923 23:30:11.889187    1473 reflector.go:205] pkg/kubelet/kubelet.go:267: Failed to list *api.Service: Get http://127.0.0.1:8080/api/v1/services?resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
 Sep 23 23:30:12 coreos-2.tux-in.com kubelet-wrapper[1473]: E0923 23:30:12.292061    1473 reflector.go:205] pkg/kubelet/config/apiserver.go:43: Failed to list *api.Pod: Get http://127.0.0.1:8080/api/v1/pods?fieldSelector=spec.nodeName%3Dcoreos-2.tux-in.com&resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
 Sep 23 23:30:12 coreos-2.tux-in.com kubelet-wrapper[1473]: E0923 23:30:12.307222    1473 event.go:207] Unable to write event: 'Post http://127.0.0.1:8080/api/v1/namespaces/default/events: dial tcp 127.0.0.1:8080: getsockopt: connection refused' (may retry after sleeping)
 Sep 23 23:30:12 coreos-2.tux-in.com kubelet-wrapper[1473]: E0923 23:30:12.495982    1473 reflector.go:205] pkg/kubelet/kubelet.go:286: Failed to list *api.Node: Get http://127.0.0.1:8080/api/v1/nodes?fieldSelector=metadata.name%3Dcoreos-2.tux-in.com&resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
 Sep 23 23:30:12 coreos-2.tux-in.com kubelet-wrapper[1473]: E0923 23:30:12.889756    1473 reflector.go:205] pkg/kubelet/kubelet.go:267: Failed to list *api.Service: Get http://127.0.0.1:8080/api/v1/services?resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
 Sep 23 23:30:13 coreos-2.tux-in.com kubelet-wrapper[1473]: E0923 23:30:13.292671    1473 reflector.go:205] pkg/kubelet/config/apiserver.go:43: Failed to list *api.Pod: Get http://127.0.0.1:8080/api/v1/pods?fieldSelector=spec.nodeName%3Dcoreos-2.tux-in.com&resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
 Sep 23 23:30:13 coreos-2.tux-in.com kubelet-wrapper[1473]: E0923 23:30:13.496732    1473 reflector.go:205] pkg/kubelet/kubelet.go:286: Failed to list *api.Node: Get http://127.0.0.1:8080/api/v1/nodes?fieldSelector=metadata.name%3Dcoreos-2.tux-in.com&resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
 Sep 23 23:30:13 coreos-2.tux-in.com kubelet-wrapper[1473]: E0923 23:30:13.589335    1473 kubelet.go:1938] Failed creating a mirror pod for "kube-apiserver-coreos-2.tux-in.com_kube-system(9b41319800532574b4c4ac760c920bee)": Post http://127.0.0.1:8080/api/v1/namespaces/kube-system/pods: dial tcp 127.0.0.1:8080: getsockopt: connection refused
 Sep 23 23:30:13 coreos-2.tux-in.com kubelet-wrapper[1473]: E0923 23:30:13.890294    1473 reflector.go:205] pkg/kubelet/kubelet.go:267: Failed to list *api.Service: Get http://127.0.0.1:8080/api/v1/services?resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
 Sep 23 23:30:13 coreos-2.tux-in.com kubelet-wrapper[1473]: I0923 23:30:13.979257    1473 docker_manager.go:2289] checking backoff for container "kube-apiserver" in pod "kube-apiserver-coreos-2.tux-in.com"
 Sep 23 23:30:13 coreos-2.tux-in.com kubelet-wrapper[1473]: I0923 23:30:13.980071    1473 docker_manager.go:2303] Back-off 2m40s restarting failed container=kube-apiserver pod=kube-apiserver-coreos-2.tux-in.com_kube-system(9b41319800532574b4c4ac760c920bee)
 Sep 23 23:30:13 coreos-2.tux-in.com kubelet-wrapper[1473]: E0923 23:30:13.980144    1473 pod_workers.go:183] Error syncing pod 9b41319800532574b4c4ac760c920bee, skipping: failed to "StartContainer" for "kube-apiserver" with CrashLoopBackOff: "Back-off 2m40s restarting failed container=kube-apiserver pod=kube-apiserver-coreos-2.tux-in.com_kube-system(9b41319800532574b4c4ac760c920bee)"

this is my /var/lib/coreos-install/user_data file:

 #cloud-config

 hostname: "coreos-2.tux-in.com"
 write_files:
  - path: "/etc/ssl/etcd/ca.pem"
    permissions: "0666"
    owner: "etcd:etcd"
    content: |
     ...
  - path: "/etc/ssl/etcd/etcd1.pem"
    permissions: "0666"
    owner: "etcd:etcd"
    content: |
     ...
  - path: "/etc/ssl/etcd/etcd1-key.pem"
    permissions: "0666"
    owner: "etcd:etcd"
    content: |
     ...
  - path: "/etc/kubernetes/ssl/ca.pem"
    permissions: "0600"
    owner: "root:root"
    content: |
     ...
  - path: "/etc/kubernetes/ssl/apiserver.pem"
    permissions: "0600"
    owner: "root:root"
    content: |
     ...
  - path: "/etc/kubernetes/ssl/apiserver-key.pem"
    permissions: "0600"
    owner: "root:root"
    content: |
     ...
  - path: "/etc/kubernetes/cni/net.d/10-calico.conf"
    content: |
      {
          "name": "calico",
          "type": "flannel",
          "delegate": {
              "type": "calico",
              "etcd_endpoints": "https://coreos-2.tux-in.com:2379",
              "log_level": "none",
              "log_level_stderr": "info",
              "hostname": "coreos-2.tux-in.com",
              "policy": {
                  "type": "k8s",
                  "k8s_api_root": "http://127.0.0.1:8080/api/v1/"
              }
          }
      }
  - path: "/etc/kubernetes/manifests/policy-controller.yaml"
    content: |
     apiVersion: v1
      kind: Pod
      metadata:
        name: calico-policy-controller
        namespace: calico-system
      spec:
        hostNetwork: true
        containers:
          # The Calico policy controller.
          - name: k8s-policy-controller
            image: calico/kube-policy-controller:v0.2.0
            env:
              - name: ETCD_ENDPOINTS
                value: "https://coreos-2.tux-in.com:2379"
              - name: K8S_API
                value: "http://127.0.0.1:8080"
              - name: LEADER_ELECTION
                value: "true"
          # Leader election container used by the policy controller.
          - name: leader-elector
            image: quay.io/calico/leader-elector:v0.1.0
            imagePullPolicy: IfNotPresent
            args:
              - "--election=calico-policy-election"
              - "--election-namespace=calico-system"
              - "--http=127.0.0.1:4040"
  - path: "/etc/kubernetes/manifests/kube-scheduler.yaml"
    content: |
      apiVersion: v1
      kind: Pod
      metadata:
        name: kube-scheduler
        namespace: kube-system
      spec:
        hostNetwork: true
        containers:
        - name: kube-scheduler
          image: quay.io/coreos/hyperkube:v1.3.6_coreos.0
          command:
          - /hyperkube
          - scheduler
          - --master=http://127.0.0.1:8080
          - --leader-elect=true
          livenessProbe:
            httpGet:
              host: 127.0.0.1
              path: /healthz
              port: 10251
            initialDelaySeconds: 15
            timeoutSeconds: 1
  - path: "/etc/kubernetes/manifests/kube-controller-manager.yaml"
    content: |
      apiVersion: v1
      kind: Pod
      metadata:
        name: kube-controller-manager
        namespace: kube-system
      spec:
        hostNetwork: true
        containers:
        - name: kube-controller-manager
          image: quay.io/coreos/hyperkube:v1.3.6_coreos.0
          command:
          - /hyperkube
          - controller-manager
          - --master=http://127.0.0.1:8080
          - --leader-elect=true
          - --service-account-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem
          - --root-ca-file=/etc/kubernetes/ssl/ca.pem
          livenessProbe:
            httpGet:
              host: 127.0.0.1
              path: /healthz
              port: 10252
            initialDelaySeconds: 15
            timeoutSeconds: 1
          volumeMounts:
          - mountPath: /etc/kubernetes/ssl
            name: ssl-certs-kubernetes
            readOnly: true
          - mountPath: /etc/ssl/certs
            name: ssl-certs-host
            readOnly: true
        volumes:
        - hostPath:
            path: /etc/kubernetes/ssl
          name: ssl-certs-kubernetes
        - hostPath:
            path: /usr/share/ca-certificates
          name: ssl-certs-host
  - path: "/etc/kubernetes/manifests/kube-proxy.yaml"
    content: |
      apiVersion: v1
      kind: Pod
      metadata:
        name: kube-proxy
        namespace: kube-system
      spec:
        hostNetwork: true
        containers:
        - name: kube-proxy
          image: quay.io/coreos/hyperkube:v1.3.6_coreos.0
          command:
          - /hyperkube
          - proxy
          - --master=http://127.0.0.1:8080
          - --proxy-mode=iptables
          securityContext:
            privileged: true
          volumeMounts:
          - mountPath: /etc/ssl/certs
            name: ssl-certs-host
            readOnly: true
        volumes:
        - hostPath:
            path: /usr/share/ca-certificates
          name: ssl-certs-host
  - path: "/etc/kubernetes/manifests/kube-apiserver.yaml"
    content: |
      apiVersion: v1
      kind: Pod
      metadata:
        name: kube-apiserver
        namespace: kube-system
      spec:
        hostNetwork: true
        containers:
        - name: kube-apiserver
          image: quay.io/coreos/hyperkube:v1.3.6_coreos.0
          command:
          - /hyperkube
          - apiserver
          - --bind-address=0.0.0.0
          - --etcd-servers=https://coreos-2.tux-in.com:2379
          - --allow-privileged=true
          - --service-cluster-ip-range=10.0.0.0/24
          - --secure-port=443
          - --advertise-address=coreos-2.tux-in.com
          - --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota
          - --tls-cert-file=/etc/kubernetes/ssl/apiserver.pem
          - --tls-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem
          - --client-ca-file=/etc/kubernetes/ssl/ca.pem
          - --service-account-key-file=/etc/kubernetes/ssl/apiserver-key.pem
          - --runtime-config=extensions/v1beta1=true,extensions/v1beta1/networkpolicies=true
          ports:
          - containerPort: 443
            hostPort: 443
            name: https
          - containerPort: 8080
            hostPort: 8080
            name: local
          volumeMounts:
          - mountPath: /etc/kubernetes/ssl
            name: ssl-certs-kubernetes
            readOnly: true
          - mountPath: /etc/ssl/certs
            name: ssl-certs-host
            readOnly: true
        volumes:
        - hostPath:
            path: /etc/kubernetes/ssl
          name: ssl-certs-kubernetes
        - hostPath:
            path: /usr/share/ca-certificates
          name: ssl-certs-host
 ssh_authorized_keys:
          - ...
 coreos:
   etcd2:
     # generate a new token for each unique cluster from https://discovery.etcd.io/new?size=3
     # specify the initial size of your cluster with ?size=X
     discovery: ...
     advertise-client-urls: https://coreos-2.tux-in.com:2379,https://coreos-2.tux-in.com:4001
     initial-advertise-peer-urls: https://coreos-2.tux-in.com:2380
     # listen on both the official ports and the legacy ports
     # legacy ports can be omitted if your application doesn't depend on them
     listen-client-urls: https://0.0.0.0:2379,https://0.0.0.0:4001
     listen-peer-urls: https://coreos-2.tux-in.com:2380
   flannel:
     etcd_endpoints: "https://coreos-2.tux-in.com:2379"
     etcd_cafile: /etc/ssl/etcd/ca.pem
     etcd_certfile: /etc/ssl/etcd/etcd1.pem
     etcd_keyfile: /etc/ssl/etcd/etcd1-key.pem
   update:
     reboot-strategy: etcd-lock
   units:
     - name: 00-enp4s0.network
       runtime: true
       content: |
        [Match]
        Name=enp4s0

        [Network]
        Address=10.79.218.2/24
        Gateway=10.79.218.232
        DNS=8.8.8.8
     - name: var-lib-rkt.mount
       enable: true
       command: start
       content: |
         [Mount]
         What=/dev/disk/by-uuid/daca9515-5040-4f1d-ac0b-b69de3b91343
         Where=/var/lib/rkt
         Type=btrfs
         Options=loop,discard
     - name: etcd2.service
       command: start
       drop-ins:
        - name: 30-certs.conf
          content: |
           [Service]
           Environment="ETCD_CERT_FILE=/etc/ssl/etcd/etcd1.pem"
           Environment="ETCD_KEY_FILE=/etc/ssl/etcd/etcd1-key.pem"
           Environment="ETCD_TRUSTED_CA_FILE=/etc/ssl/etcd/ca.pem"
           Environment="ETCD_CLIENT_CERT_AUTH=true"
           Environment="ETCD_PEER_CERT_FILE=/etc/ssl/etcd/etcd1.pem"
           Environment="ETCD_PEER_KEY_FILE=/etc/ssl/etcd/etcd1-key.pem"
           Environment="ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/etcd/ca.pem"
           Environment="ETCD_PEER_CLIENT_CERT_AUTH=true"
     - name: flanneld.service
       command: start
       drop-ins:
        - name: 50-network-config.conf
          content: |
           [Service]
           ExecStartPre=/usr/bin/etcdctl --ca-file=/etc/ssl/etcd/ca.pem --cert-file=/etc/ssl/etcd/etcd1.pem --key-file=/etc/ssl/etcd/etcd1-key.pem --endpoint=https://coreos-2.tux-in.com:2379 set /coreos.com/network/config '{"Network":"10.1.0.0/16", "Backend": {"Type": "vxlan"}}'
     - name: calico-node.service
       command: start
       content: |
        [Unit]
        Description=Calico per-host agent
        Requires=network-online.target
        After=network-online.target

        [Service]
        Slice=machine.slice
        Environment=CALICO_DISABLE_FILE_LOGGING=true
        Environment=HOSTNAME=coreos-2.tux-in.com
        Environment=IP=10.79.218.2
        Environment=FELIX_FELIXHOSTNAME=coreos-2.tux-in.com
        Environment=CALICO_NETWORKING=false
        Environment=NO_DEFAULT_POOLS=true
        Environment=ETCD_ENDPOINTS=https://coreos-2.tux-in.com:2379
        Environment=ETCD_AUTHORITY=coreos-2.tux-in.com:2379
        Environment=ETCD_SCHEME=https
        Environment=ETCD_CA_CERT_FILE=/etc/ssl/etcd/ca.pem
        Environment=ETCD_CERT_FILE=/etc/ssl/etcd/etcd1.pem
        Environment=ETCD_KEY_FILE=/etc/ssl/etcd/etcd1-key.pem
        ExecStart=/usr/bin/rkt run --volume=resolv-conf,kind=host,source=/etc/resolv.conf,readOnly=true \
        --volume=etcd-tls-certs,kind=host,source=/etc/ssl/etcd,readOnly=true --inherit-env --stage1-from-dir=stage1-fly.aci \
        --volume=modules,kind=host,source=/lib/modules,readOnly=false \
        --mount=volume=modules,target=/lib/modules \
        --trust-keys-from-https quay.io/calico/node:v0.19.0 \
        --mount=volume=etcd-tls-certs,target=/etc/ssl/etcd \
        --mount=volume=resolv-conf,target=/etc/resolv.conf

        KillMode=mixed
        Restart=always
        TimeoutStartSec=0

        [Install]
        WantedBy=multi-user.target
     - name: kubelet.service
       command: start
       content: |
        [Service]
        ExecStartPre=/usr/bin/mkdir -p /etc/kubernetes/manifests
        ExecStartPre=/usr/bin/mkdir -p /var/log/containers

        Environment=KUBELET_VERSION=v1.3.7_coreos.0
        Environment="RKT_OPTS=--volume var-log,kind=host,source=/var/log \
          --mount volume=var-log,target=/var/log \
          --volume dns,kind=host,source=/etc/resolv.conf \
          --mount volume=dns,target=/etc/resolv.conf"

        ExecStart=/usr/lib/coreos/kubelet-wrapper \
          --api-servers=http://127.0.0.1:8080 \
          --network-plugin-dir=/etc/kubernetes/cni/net.d \
          --network-plugin=cni \
          --register-schedulable=false \
          --allow-privileged=true \
          --config=/etc/kubernetes/manifests \
          --hostname-override=coreos-2.tux-in.com \
          --cluster-dns=8.8.8.8 \
          --cluster-domain=tux-in.com
        Restart=always
        RestartSec=10
        [Install]
        WantedBy=multi-user.target

is 127.0.0.1:8080 should be opened by the kubelet-apiserver ? what am I missing here?

thanks!

1

In many cases, the API server is started by Kubelet, resulting in initial connectivity errors before the API endpoint is available. If this error is persistent after some time, you may wish to see if your API server is starting and whether or not.

Kubelet will automatically start the services located in /etc/kubernetes/manifests which is likely the place your kube-apiserver.yaml belongs.

If your API Server is not starting, you will need to:

1: Check Kubelet command line options to ensure manifests are enabled with the --config=/etc/kubernetes/manifests options. This can be checked with ps aux | grep kubelet

2: Check the Kubelet & API container logs to see what is breaking during API Startup. This would generally be a certificate mismatch, failed dependency, etcd service not listening, etc.

Kubelet service logs:

   $ journalctl -fu kubelet.service

This example I gather logs from the API server via 'docker logs' and show my Kubelet starting API server. Notice the similar connectivity issues before the server is listening, and eventual startup.

   $ docker ps -l
    543022a70bc6        gcr.io/google_containers/hyperkube:v1.3.7   "/hyperkube apiserver"   3 seconds ago        Exited (1) 3 seconds ago

   $ docker logs 543022a70bc6
    I0920 00:26:33.903861       1 genericapiserver.go:606] Will report 10.0.104.100 as public IP address.
    E0920 00:26:33.937478       1 reflector.go:205] k8s.io/kubernetes/plugin/pkg/admission/namespace/exists/admission.go:86: Failed to list *api.Namespace: Get http://0.0.0.0:8080/api/v1/namespaces?resourceVersion=0: dial tcp 0.0.0.0:8080: getsockopt: connection refused
    E0920 00:26:33.937651       1 reflector.go:205] k8s.io/kubernetes/plugin/pkg/admission/namespace/lifecycle/admission.go:116: Failed to list *api.Namespace: Get http://0.0.0.0:8080/api/v1/namespaces?resourceVersion=0: dial tcp 0.0.0.0:8080: getsockopt: connection refused
    E0920 00:26:33.937821       1 reflector.go:205] k8s.io/kubernetes/plugin/pkg/admission/limitranger/admission.go:154: Failed to list *api.LimitRange: Get http://0.0.0.0:8080/api/v1/limitranges?resourceVersion=0: dial tcp 0.0.0.0:8080: getsockopt: connection refused
    E0920 00:26:33.939508       1 reflector.go:216] k8s.io/kubernetes/plugin/pkg/admission/serviceaccount/admission.go:119: Failed to list *api.Secret: Get http://0.0.0.0:8080/api/v1/secrets?fieldSelector=type%3Dkubernetes.io%2Fservice-account-token&resourceVersion=0: dial tcp 0.0.0.0:8080: getsockopt: connection refused
    E0920 00:26:33.939741       1 reflector.go:216] k8s.io/kubernetes/plugin/pkg/admission/serviceaccount/admission.go:103: Failed to list *api.ServiceAccount: Get http://0.0.0.0:8080/api/v1/serviceaccounts?resourceVersion=0: dial tcp 0.0.0.0:8080: getsockopt: connection refused
    E0920 00:26:33.947780       1 reflector.go:205] k8s.io/kubernetes/plugin/pkg/admission/resourcequota/controller.go:121: Failed to list *api.ResourceQuota: Get http://0.0.0.0:8080/api/v1/resourcequotas?resourceVersion=0: dial tcp 0.0.0.0:8080: getsockopt: connection refused
    [restful] 2016/09/20 00:26:34 log.go:30: [restful/swagger] listing is available at https://10.0.104.100:6443/swaggerapi/
    [restful] 2016/09/20 00:26:34 log.go:30: [restful/swagger] https://10.0.104.100:6443/swaggerui/ is mapped to folder /swagger-ui/
    I0920 00:26:34.235914       1 genericapiserver.go:690] Serving securely on 0.0.0.0:6443
    I0920 00:26:34.235941       1 genericapiserver.go:734] Serving insecurely on 0.0.0.0:8080
|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.