How an explosion takes place on an opensearch index/cluster... 🥳

General Feedback
,
1

First, I’m using opensearch as a content marketing agency, i’m not a tech related user of Opensearch.

At a limit set at 500.000 fields/columns, by mistake, an explosion took place…
In the past, i successful used Openseach at 5000 columns/fields.
This time, i exaggerated and risked the 500.000.

The index disappeared.
Some other indexes within the (small 3 - nodes) cluster, also disappeared.
My guess is that all indexes disappeared at the time of the BIG-BANG document that Opensearch probably tried to index, document that probably had lots of columns/fields.

The exact number of fields the BIG document had, I have not yet discovered.
As i mentioned, i used Opensearch with aprox 5000 columns/fields.
I usually use on average per document 2500 fields.
I know, its kind of against patterns of best use.

My field setting on the part of documents that have lots of fields dynamically added, is set as dynamic field type. Tried, in my work with Opensearch to use nested field type on that specific field, that i need to have and be able to index under it a big number of columns/fields, but things are complicated.

In the 2.9 release saw that the risk of explosion have been reduced, or maybe i misunderstood the improvement brought by the release.

The nice part, as far as i understand the Opensearch, is that after the crash/implosion caused by lots of fields, the dashboard i think recreated or maybe the opensearch recreated the needed internal indexes that are i think needed for the cluster and maybe the dashboards to work, and even if the explosion took place, the system was still up and running.

When tried to restore the backup, the red flag was on the specific index where the explosion took place, and i had 2 versions that were saved, but none of them worked to be restored. Most likely because of the huge number of fields. The restauration took place, but the index remained in red state, and no orange status was obtained, probably the flow of information was impossible to rebuild/restore the index with its documents.

But, overall, Opensearch is a great tool for me as a content marketing that uses Opensearch to manage pieces of content used in digital communications cross channels, on different webpages, sites, domains.

:sunglasses::partying_face::crazy_face:

2

0_o

Just to make sure I am understanding you accidentally indexed a document with 500k fields and that crashed your cluster?? I am going to have to test this because that should not be allowed… I wonder if it caused an out of memory bug in an area we dont check.

1 Like
3

Hello,

I left the max fields by mistake at the value of 500.000 (I’m sure this was the setting value left before the inplosion). And started the ongoing indexing of documents, some of them have i think around >10.000 maybe 20.000 fields or maybe more, I don’t think I have documents that reach 100.000 fields.

Usually i leave the max fields at 2000-5000 and in the indexing process no crash appears/appeared, but, at the same time, I’m not able to index some documents that have more then 5000-10.000 fields.

Saw in the release notes of Opensearch 2.9 some sort of note regarding the maximum fields within a document, and hoped that I’ll be able to have tons of columns/fields nested within a document, without using the nested field type, just by using the dynamic type field.

Also, I don’t know if nested fields offer a bigger number of fields (objects/arrays) indexed within documents.

So, I set the config at 500.000 max fields, started the indexing of documents (documents that are very variate in terms of structure), and the crash took place at an unknown number of fields that probably where supposed to be created at the BIG-BANG document with lots of columns/fields.

In my use case of Opensearch, i index pieces of content, that represent for example a book and its propositions. So, having one document as a book with a huge number of propositions within it is very important for my use-case.