Opensearch optimization for search


I’m running an opensearch host (single-host) that we use for log analysis.
On that instance, we will be ingesting like 20Go of logs per day.

I’m looking for suggestions how I can optimize it for search efficiency. Of course if there are possibility to have ingestion performances, I’m also interested.
I use only datastreams with 0 replica (as it is a single-host), and 1 shard.
We will be keep the logs for 90 days, then delete the oldest ones.

So actually adding shards may be able to help a bit. When you send a search it it will spin up a thread per shard if I’m not mistaken. You wouldnt want 100 shards because then it is going to take more time to merge all the results at the end. 2-3 additional ones may help.

Do you have an idea of how long searches are taking and what your target time would be?

oh thats a good point, thank you! FYI, My container is currently running with 16 CPUs and 16Gb of Ram (dedicated to OS ), 2To of SSD.
Ideally i would aim for aim for 1sec or less.
Actually how can you measure accurately the time that takes a search? Yet I only query my datastreams via opensearch-dashboards, so I can only estimate the time that takes a query and I counted 5secs to retrieve 3 days of data (45Go).
That is 162,528,687 hits and each document containing approximately 11fields.

Most of the time OS is not queried intensively, only when we need to investigate logs. But at that moment, it is nice to not wait too long for the reply.

There are lots of things that you might want to do, from changing when indices get rotated to the number of shards to the merge policy and such. We listed many of these options in quite a lot of detail a long time ago here: Elasticsearch for logs and metrics: A deep dive – Velocity 2016, O’REILLY CONFERENCES - YouTube

With the number of shards in particular, it depends on what you need to optimize for:

  • latency for searching one index → in this case more than 1 shard per index would help because, as David mentioned, you’d search with one thread per shard
  • latency for searches on more data (N indices) → in this case you’d naturally parallelize the further you go back, so 1 shard works best. It’s just that the number of shards you hit depends on how often you rotate. The rollover policy will also influence the likely size of the “current” index - which is typically N times slower than the rest because of caching. Typically, the sweetspot is between 5GB and 20GB per shard. Towards 5GB if you don’t have a lot of data, towards 20GB if you do. Given your hardware specs, you can’t store too much data on this node, so I’d tend to go towards 5GB

@radu.gheorghe Thank you for your detailed reply! Sorry I was on vacation, so I could not reply to that feed.
So i was want to use a 90 days retention policy, I have decided to use 1 index per day, to make the rotation per day easier.
Now i have about 80 indices, of +/- 10Gb each with 3 primary shards. If I understand your explanation correctly, I should go back to 1 shard per index in that case, is that correct ?

Yes, it sounds like 1 shard per index is your best bet, efficiency-wise.

But if I were you I would set up Index State Management - OpenSearch documentation and make it rotate at 10GB. This way if, in the future, you have spikes of ingest, you won’t end up with shards that are too big (which will slow down indexing. And while you index to them, searches will be slower, too).

OK so the conditions in the policy management are OR? so if index reach 7d OR 10Go?
I thought an alternative that i thought could be more performant, if I fot your point:
I can create an index rollover of a week (where i roughly exect) 70Go over 7 days, but with 5-6 shards per index. This way I still have shards of +/- 15Go but less indices. Wouldn’t that be even more efficient search wise? I would have less indices but more shards:
8 indices for 8 weeks, each having 6 shards

Yes, you can say 7d OR 10GB, whichever comes first.

Having an index of 70GB with 7 shards would be slower than the equivalent 7 indices of 10GB each, because in the first case you’re writing to all shards at once, invalidating caches. Searches will be much slower because of that. Otherwise, when you search for the whole week, you’d still hit 7 shards in total.

IMO the rule of thumb is to lower the “surface” of your searches and indexing as much as it’s reasonable.