OpenSearch heap sizing is the process of setting the correct amount of RAM for the Java Virtual Machine (JVM) that runs OpenSearch. Correctly sizing the Java Heap is the most essential configuration step for OpenSearch performance. In the OpenSearch Cluster Setup guide, we learned the initial installation and specific heap tuning for stability. If the heap is too small, you risk OutOfMemoryError crashes. If it is too large, you starve the Lucene filesystem cache, which leads to decreased performance.

In this guide from PerLod Hosting, we want to explore how to size OpenSearch heap settings correctly to avoid OOM and performance degradation.

OpenSearch Heap Sizing Rules

Configuring the heap is a balancing act between the application and the operating system. OpenSearch needs enough RAM to process queries, but the OS needs the rest to cache the actual data files. These four rules help you find that perfect size to prevent crashes and keep search speeds fast:

1. The 50% Rule: Allocate no more than 50% of your total physical RAM to the OpenSearch heap. OpenSearch runs on the Java Virtual Machine (JVM), but it relies heavily on the underlying operating system for file caching.

• If you have 64GB RAM: Allocate 30GB to 31GB to the heap and leave the rest for the OS.
• If you have 16GB RAM: Allocate 8GB to Heap.

Note: If you set up your OpenSearch based on our cluster setup guide, ensure your instance type allows for this memory split.

2. The 32GB Limit (Compressed Oops): Never set your heap size above 32GB. The JVM uses a technique called Compressed Ordinary Object Pointers (Compressed Oops) to compress 64bit pointers to 32 bits.

• If you cross the limit (usually around 32GB), the JVM switches to 64bit pointers, which increases memory usage by about 1.5x.
• The safe heap size can be at 30GB or 31GB. If you need more capacity, scale out by adding nodes rather than scaling up a single heap.

3. Lock the Heap (Xms = Xmx): Always set the minimum (-Xms) and maximum (-Xmx) heap size to the same value.

If Xms is lower than Xmx, the JVM must resize the heap during runtime, which causes performance pauses.

Correct value:

-Xms4g -Xmx4g

4. Disable Swapping: Heap sizing is useless if the OS swaps the JVM memory to disk, so you must disable swap on your Linux host:

sudo swapoff -a

Also, remove the swap entries from the /etc/fstab file.

In the opensearch.yml file, you must set:

bootstrap.memory_lock: true

Note: Hardening your memory configuration is as essential as securing your network layers. For production, combine this step with the security practices detailed in our Secure OpenSearch Cluster guide.

Common OpenSearch Heap Sizing Mistakes

Even with the correct rules in mind, it is easy to make small configuration errors that have a big impact on performance. Avoid these common configuration mistakes to ensure your memory settings actually work as intended.

1. Cross the Zero-Based Limit: The JVM runs fastest when the heap is under 30GB. If you try to maximize it to exactly 32GB, like 31.9GB, the system often loses efficiency features. To be safe and keep the speed high, just stop at 30GB.

2. Starve the OS Cache: Allocating 80 to 90% of RAM to OpenSearch leaves no RAM for the OS to cache Lucene segments.

OpenSearch must read data from the disk frequently, which causes slow search speeds. If your heap is full but CPU is low, and disk I/O is high, you starve the OS cache.

3. Configuration Conflict: OpenSearch picks settings in this order of priority:​

• OPENSEARCH_JAVA_OPTS environment variable, which has the highest priority.
• jvm.options file. If you edit jvm.options but have an old environment variable set, your file changes will be ignored.

How to Apply OpenSearch Heap Settings?

At this point, you can follow the steps below to apply and verify the OpenSearch heap settings.

You can configure the OpenSearch heap settings using the jvm.options file. Open the file with your desired text editor:

sudo nano /etc/opensearch/jvm.options

Uncomment and update the Xms and Xmx lines as shown below:

-Xms4g
-Xmx4g

Once you are done, restart OpenSearch to apply the changes:

sudo systemctl restart opensearch

After applying the configuration, you can use the OpenSearch API to confirm the node accepted the new heap size:

curl -XGET "https://localhost:9200/_cat/nodes?h=name,heap.current,heap.max,heap.percent&v" -u 'admin:YourPassword' --insecure

Security Note: If you have followed our OpenSearch security guide, your cluster will require HTTPS and authentication (-u). If you are still in a testing phase without security, use http.

Expected Output:

name      heap.current heap.max heap.percent
node-1    1.2gb        4gb      30

• heap.max: Should match your setting.
• heap.percent: A healthy idle node value is between 30% to 70%.

To verify that pointer compression is active, you can check the node logs during startup:

grep "compressed ordinary object pointers" /var/log/opensearch/opensearch.log

Expected Output:

[INFO ][o.e.e.NodeEnvironment] [node-1] heap size [29.8gb], compressed ordinary object pointers [true]

Tip: If you are managing multiple nodes, manually grepping logs on every server is inefficient. We recommend centralized logging; check our Monitoring Linux Logs with OpenSearch and Filebeat guide to learn how to ship these logs to a dashboard for easier verification.

FAQs

Conclusion

Configuring the OpenSearch heap settings correctly is the most effective way to prevent crashes and ensure low-latency searches. By following the 50% rule and strictly capping your heap at 30GB, you allow the JVM and the operating system to work together efficiently. Always validate your changes using the OpenSearch API, and ensure your underlying OS is prepared.

A well-sized heap results in a stable, fast, and reliable search engine.

Heap sizing is important, but so is hardware. As your cluster grows, it is recommended to move master and coordinating tasks to dedicated servers to keep everything stable.

We hope you enjoy this OpenSearch heap sizing guide. Subscribe to our X and Facebook channels to get the latest articles.