Elasticsearch Shards: Definition, Sizes, and Optimization

Updated September 2022

Optimizing Elasticsearch for shard size is an important component for achieving maximum performance from your cluster. To get started let’s review a few definitions that are an important part of the Elasticsearch jargon. If you are already familiar with Elasticsearch, you can continue straight to the next section.

Defining Elasticsearch Jargon: Cluster, Replicas, Shards, and More

Elasticsearch clusters are the gathering of three or more nodes, and each cluster has a unique name for accurate identification. Each node represents a single Elasticsearch instance, and the minimum number of nodes for a cluster is three because Elasticsearch is a distributed system. [If you’re not familiar with why three (3) is the minimum number of nodes, read this.]

Anytime there is a collection of documents in Elasticsearch, it is referred to as an index, and each index is typically split into sub-elements referred to as shards. These shards are then distributed across multiple nodes in the cluster. Elasticsearch automatically manages and balances how the shards are arranged in the nodes.

Elasticsearch also automatically creates one (1) primary shard and one (1) replica for every index. We recommend that our clients have one (1) replica in every production cluster as a backup. Beyond serving as a backup, the replicas also aid in search performance, providing higher throughput and additional capacity. You can add or remove replicas at anytime to scale out query processing.

Unlike replicas, shard allocation cannot be modified later. So without further ado, let’s dig into shards.

Optimizing Elasticsearch Shard Size and Number

Determining shard allocation at the get-go is important because if you want to change the number of shards after the cluster is in production, it is necessary to reindex all of the source documents. If you are new to Elasticsearch, just know that reindexing is a long process.

You might be thinking, “Just allocate a crazy number of shards so that’s never a problem, right?” Not quite, like all things in life there needs to be a balance.

Each new shard comes with a cost, taking up memory and CPU resources. Additionally, keep in mind that each search request will require an interaction with every shard in the index. If shards are competing for the same hardware resources (that is part of the same node), then this will impair performance.

You will want to limit your maximum shard size to 30-80 GB if running a recent version of Elasticsearch. In fact, a single shard can hold as much as 100s of GB and still perform well. (If running below version 6.0 then estimate 30-50 GB.)

Using the 30-80 GB value, you can calculate how many shards you’ll need.

For instance, let’s assume you rotate indices monthly and expect around 600 GB of data per month. In this example, you would allocate 8 to 20 shards.

Typically it is better for performance to have a fewer number of large shards, as opposed to a large number of small shards since each shard will take up a finite amount of resources.

For the example above, we would likely stick to the lower end of the range and use about 10 shards.

There are exceptions, of course.

Let’s say your use case focuses on low latency indexing, quick queries, and you have 40 nodes. In that scenario, we would increase the shard count to 40 for better allocation of shards across our hardware at the expense of less efficient shard allocation.

With a total of 40 shards, each of our nodes will have 1/40th of the data to process. Whereas with 8 total shards, we would have 8 nodes processing 1/8th of the data each and the other 32 nodes not doing anything.

Elastic Stack Consulting Services

If you are interested in 24/7 support, consulting, and/or fully managed Elasticsearch services on your environment, you can find more information on our Elasticsearch consulting page.