Hello,
I’m running an opensearch host (single-host) that we use for log analysis.
On that instance, we will be ingesting like 20Go of logs per day.
I’m looking for suggestions how I can optimize it for search efficiency. Of course if there are possibility to have ingestion performances, I’m also interested.
I use only datastreams with 0 replica (as it is a single-host), and 1 shard.
We will be keep the logs for 90 days, then delete the oldest ones.
So actually adding shards may be able to help a bit. When you send a search it it will spin up a thread per shard if I’m not mistaken. You wouldnt want 100 shards because then it is going to take more time to merge all the results at the end. 2-3 additional ones may help.
Do you have an idea of how long searches are taking and what your target time would be?
oh thats a good point, thank you! FYI, My container is currently running with 16 CPUs and 16Gb of Ram (dedicated to OS ), 2To of SSD.
Ideally i would aim for aim for 1sec or less.
Actually how can you measure accurately the time that takes a search? Yet I only query my datastreams via opensearch-dashboards, so I can only estimate the time that takes a query and I counted 5secs to retrieve 3 days of data (45Go).
That is 162,528,687 hits and each document containing approximately 11fields.
Most of the time OS is not queried intensively, only when we need to investigate logs. But at that moment, it is nice to not wait too long for the reply.
There are lots of things that you might want to do, from changing when indices get rotated to the number of shards to the merge policy and such. We listed many of these options in quite a lot of detail a long time ago here: Elasticsearch for logs and metrics: A deep dive – Velocity 2016, O’REILLY CONFERENCES - YouTube
With the number of shards in particular, it depends on what you need to optimize for:
@radu.gheorghe Thank you for your detailed reply! Sorry I was on vacation, so I could not reply to that feed.
So i was want to use a 90 days retention policy, I have decided to use 1 index per day, to make the rotation per day easier.
Now i have about 80 indices, of +/- 10Gb each with 3 primary shards. If I understand your explanation correctly, I should go back to 1 shard per index in that case, is that correct ?
Yes, it sounds like 1 shard per index is your best bet, efficiency-wise.
But if I were you I would set up Index State Management - OpenSearch documentation and make it rotate at 10GB. This way if, in the future, you have spikes of ingest, you won’t end up with shards that are too big (which will slow down indexing. And while you index to them, searches will be slower, too).
OK so the conditions in the policy management are OR? so if index reach 7d OR 10Go?
I thought an alternative that i thought could be more performant, if I fot your point:
I can create an index rollover of a week (where i roughly exect) 70Go over 7 days, but with 5-6 shards per index. This way I still have shards of +/- 15Go but less indices. Wouldn’t that be even more efficient search wise? I would have less indices but more shards:
8 indices for 8 weeks, each having 6 shards
Yes, you can say 7d OR 10GB, whichever comes first.
Having an index of 70GB with 7 shards would be slower than the equivalent 7 indices of 10GB each, because in the first case you’re writing to all shards at once, invalidating caches. Searches will be much slower because of that. Otherwise, when you search for the whole week, you’d still hit 7 shards in total.
IMO the rule of thumb is to lower the “surface” of your searches and indexing as much as it’s reasonable.