Limits to automatic id created at once?

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser): v3.5.0

Describe the issue:

I have recently moved from elasticsearch to opensearch and have noticed about 25% of documents are getting lost on a high throughput cluster. I also changed from indexing to alias to using streams. I let document IDs be autogenerated.

Lower cluster volume seems to be fine. There are occasional logs missing there too.

I do occasionally see failures due to duplicate IDs which should not be happening as I don’t define the ID usually.

Configuration:

Filebeat → logstash → opensearch

Relevant Logs or Screenshots:

2

@doug_f How many resources did you assign to OpenSearch nodes?
What’s the Java heap size value?

What roles has your OpenSearch node assigned?

3

Here are the nodes:

3 manager 16GB heap (cluster_manager)

6 local 31GB heap (data, ingest, remote_cluster_client)

6 Searchable Snapshot (warm) with 31GB heap

6 client nodes with 16GB heap (remote_cluster_client)

All nodes are at 1-30% CPU according to Prometheus.

4

@doug_f I’ve seen this ID issue before when Security Analytics monitors had over 30 mappings. Plugin was creating plenty of fan_outs tasks and cluster was falling apart even when Java heap size was at 32 GB and CPU nodes weren’t too busy.

Also plugin was creating a lot of long running fanout tasks.

Have you tried to separate ingest and data nodes?