Split brains is a recurring problem when running any kind of clusters. A sudden server crash or network partition might lead to inconsistent state and data corruption. Elasticsearch addresses this problem by allowing multiple nodes to be configured as master. Running an odd number of master node and properly setting
(number of master nodes / 2) + 1 is an easy way to prevents split brain disasters.
However, there’s still a case your cluster might find itself in an inconsistent state.
When your master node leaves the cluster for some reasons and won’t reconnect by itself, it keeps a list of indexes existing before the split. Our clusters are living things, and we create and delete indexes all day long. When your long lost master comes back from the dead, you’ll notice some strange messages in the logs:
[2016–10–09 16:35:12,071][INFO ][gateway.local.state.meta ] [esmaster01]  dangling index, exists on local file system, but not in cluster metadata, auto import to cluster state [YES]
These are the indexes your master used to know about before coming back. Elasticsearch considers these indexes actually exists and will import them into the elected master.
[2016–10–09 16:35:16,715][DEBUG][gateway.local.state.meta ] [esmaster01]  no longer dangling (created), removing
That’s the moment your cluster turns red and newly created indexes appear when running
GET /_cat/indices, except the data don’t exist anymore. The only way to bring it back to green is to delete those fantom indexes one by one using
DELETE. Nothing complicated except a large number of freshly created indexes might put your elected master to their knees.
According to Elasticsearch documentation, this feature has 2 purposes:
If a new master node is started which is unaware of the other indices in the cluster, adding the old nodes will cause the old indices to be imported, instead of being deleted. An old index can be added to an existing cluster by copying it to the data/ directory of a new node, starting the node and letting it join the cluster. Once the index has been replicated to other nodes in the cluster, the new node can be shut down and removed.
Elasticsearch behaviour can be controlled using
gateway.local.autoimportdangled which is set to
yes by default.
However, to avoid any surprise after a master node crash, I prefer to shutdown Elasticsearch, delete all the data directory and start the node as a fresh one. Indeed it might not fill all the cases, but it avoids most conflicts due to a zombi node coming back from the dead.
Photo: Brain, by Adeel Anwer, CC.