0

When setting up Elasticsearch cluster on Kubernetes with 2x data, 2x master and 2x client nodes (each container on sepearete physical node) i get the following error in client-node:

[2019-01-28T12:25:08,574][WARN ][o.e.d.z.UnicastZenPing   ] [elasticsearch-client-5f7759d57c-95wwm] [81] failed send ping to {#zen_unicast_elasticsearch-discovery_0#}{LuD6SiE5RCG7Ol9xb0LBFw}{elasticsearch-discovery}{10.233.12.222:9300}
java.lang.IllegalStateException: handshake failed, mismatched cluster name [Cluster [elasticsearch-default]] - {#zen_unicast_elasticsearch-discovery_0#}{LuD6SiE5RCG7Ol9xb0LBFw}{elasticsearch-discovery}{10.233.12.222:9300}
        at org.elasticsearch.transport.TransportService.handshake(TransportService.java:406) ~[elasticsearch-6.2.3.jar:6.2.3]
        at org.elasticsearch.transport.TransportService.handshake(TransportService.java:369) ~[elasticsearch-6.2.3.jar:6.2.3]
        at org.elasticsearch.discovery.zen.UnicastZenPing$PingingRound.getOrConnect(UnicastZenPing.java:400) ~[elasticsearch-6.2.3.jar:6.2.3]
        at org.elasticsearch.discovery.zen.UnicastZenPing$3.doRun(UnicastZenPing.java:503) [elasticsearch-6.2.3.jar:6.2.3]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.3.jar:6.2.3]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.3.jar:6.2.3]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]

This error does not have any sense because i dont have cluster name [elasticsearch-default]..

Also, the client node is responding when checking against elasticsearch api endpoint and show correct cluster name that is set in the config:

{
  "name" : "elasticsearch-client-5f7759d57c-q2vd5",
  "cluster_name" : "elasticsearch-test2",
  "cluster_uuid" : "_na_",
  "version" : {
    "number" : "6.2.3",
    "build_hash" : "c59ff00",
    "build_date" : "2018-03-13T10:06:29.741383Z",
    "build_snapshot" : false,
    "lucene_version" : "7.2.1",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"

But when trying healthcheck endpoint it responds with:

curl -X GET "10.233.43.73:9200/_cluster/health?pretty"

{
  "error" : {
    "root_cause" : [
      {
        "type" : "master_not_discovered_exception",
        "reason" : null
      }
    ],
    "type" : "master_not_discovered_exception",
    "reason" : null
  },
  "status" : 503
}

Additionally, the master-node is showing this error in the log:

[WARN ][o.e.t.n.Netty4Transport  ] [49d98428-ddef-479e-85b2-e00fb6f0acef] exception caught on transport layer [NettyTcpChannel{localAddress=/10.233.67.119:9300, remoteAddress=/10.233.69.29:35712}], closing connection
io.netty.handler.codec.DecoderException: java.io.StreamCorruptedException: invalid internal transport message format, got (ff,f4,ff,fd)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:392) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:359) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:342) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.handler.logging.LoggingHandler.channelInactive(LoggingHandler.java:167) [netty-handler-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1354) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:917) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:822) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [netty-common-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403) [netty-common-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
Caused by: java.io.StreamCorruptedException: invalid internal transport message format, got (ff,f4,ff,fd)
        at org.elasticsearch.transport.TcpTransport.validateMessageHeader(TcpTransport.java:1283) ~[elasticsearch-6.2.3.jar:6.2.3]
        at org.elasticsearch.transport.netty4.Netty4SizeHeaderFrameDecoder.decode(Netty4SizeHeaderFrameDecoder.java:36) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[?:?]
        ... 20 more

ES image: txn2/k8s-es:v6.2.3

ES config file:

cluster:
  name: ${CLUSTER_NAME}

node:
  master: ${NODE_MASTER}
  data: ${NODE_DATA}
  name: ${NODE_NAME}
  ingest: ${NODE_INGEST}
  max_local_storage_nodes: ${MAX_LOCAL_STORAGE_NODES}

processors: ${PROCESSORS:1}

network.host: ${NETWORK_HOST}

path:
  data: /data/data
  logs: /data/log
  repo: ${REPO_LOCATIONS}

bootstrap:
  memory_lock: ${MEMORY_LOCK}

http:
  enabled: ${HTTP_ENABLE}
  compression: true
  cors:
    enabled: true
    allow-origin: "*"

discovery:
  zen:
    ping.unicast.hosts: ${DISCOVERY_SERVICE}
    minimum_master_nodes: ${NUMBER_OF_MASTERS}

all env variables are being read correctly

bash-4.4# printenv | grep CLUSTER
CLUSTER_NAME=elasticsearch-test2

the error from master-node is indicating like there is some network problem like the containers couldnt "talk" to each other, but i even tested if the connection is possible from client-node container to elasticsearch-discovery service ClusterIP:

sudo nsenter -t 4552 -n telnet 10.233.12.222 9300
Trying 10.233.12.222...
Connected to 10.233.12.222.
Escape character is '^]'.

Kubernetes configuration: 7 node cluster (3 masters + 4 minions)

clientVersion:
  buildDate: 2018-04-27T09:10:24Z
  compiler: gc
  gitCommit: 81753b10df112992bf51bbc2c2f85208aad78335
  gitTreeState: clean
  gitVersion: v1.10.2
  goVersion: go1.9.3
  major: "1"
  minor: "10"
  platform: linux/amd64
serverVersion:
  buildDate: 2018-04-27T09:10:24Z
  compiler: gc
  gitCommit: 81753b10df112992bf51bbc2c2f85208aad78335
  gitTreeState: clean
  gitVersion: v1.10.2
  goVersion: go1.9.3
  major: "1"
  minor: "10"
  platform: linux/amd64
Docker version: 17.3.2

I can also add that i managed to get it to work using exactly the same image on another cluster (2 masters + 2 minions. exaclty the same kubernetes and docker versions).

Improve this question

1 Answer 1

Reset to default
0

I guess you are using the this Helm chart?

I had this same error and it seems to be because of only 2 masters and there is a quorum issue when the new client tries to join. My solution was to add another master FIRST. Once a client is failing to connect to the cluster it was too late, and scaling the master statefulset failed too.

My solution, unfortunately, was to delete the cluster (but not the Persistent Volumes) and re-create it. The PVs were re-attached to the ES nodes and the downtime was minimal.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .