Index that I use for product search has disappeared twice

OpenSearch
,
1

Hi, I have an issue with OpenSearch hosted on AWS, for some reason, my index that I use for product search has disappeared twice in stage. The dev environment is completely fine.It happened first time around a month ago, and today I discovered that it disappeared again. No lifecycle policy was attached.After first error I thought about enabling audit logs, but I then got busy with another task, and even if I enable them now my understanding is that I won’t be able to see what actually happened, and I can’t wait another month for this to happen again.

OpenSearch version:
2.17.0

Environment:

  • Stage: index disappeared twice (~28 days apart)
  • Dev: index is fine

Index settings:

  • No ISM or ILM policy
  • Manually created
  • Standard mappings

I am the only one who is using OpenSearch, it is also accessed by a Lambda that manages products (updated items, add new ones, etc), but it doesn’t have a feature to remove an index. Another lambda does actual product retrieval logic, but as it queries search, I don’t think it has anything to do with the issue.Are there internal mechanisms or bugs that could explain this?
How to trace deletions more reliably?
Have any ideas what I should do next?

2

@vika_ts How is your OpenSearch cluster deployed? EKS, EC2 or managed OpenSearch?

3

@pablo
Hi! Thanks so much for taking the time to respond and help dig into this.
I’m using Amazon Managed OpenSearch Service

4

@vika_ts Thanks for the clarification. Have you tried to contact AWS support?
The indices don’t disappear by themselves. Even if the data would be corrupted, OpenSearch cluster would still have the information about the index.

Do you keep all your environments in the same area/zone?