Having huge index without data?

OpenSearch
, ,
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
2.8
build date 2023-06-03

Describe the issue:
The size of the index is 50GB. The query reports to have 62642231 documents but I can’t see them in the OpensearchDashboard. Only a few (normal) documents are displayed there.

Also strange, that I use one index per day. The index for July 12 queried in the screenshot shows data from July 2nd.

How can i find these “occupying” documents and how to be sure, that it is not some kind of CVE like this I Geeting some vulnerability issue in opensearchproject/opensearch-Dashboard:2.8.0 docker Image

Configuration:

Relevant Logs or Screenshots:

Screenshot 2024-07-13 191740
Screenshot 2024-07-13 1917401892×302 48.2 KB

2

You can check the document count by calling

GET fluent-bit-sec-2024.07.12/_count

.
You can also execute aggregation on the _index field to check if there’re some old documents existing in the new index:

GET  fluent-bit-sec-2024.07.12/_search
{
  "size": 0,
  "aggs": {
    "test": {
      "terms": {
        "field": "_index"
      }
    }
  }
}
1 Like
3

Thank you very much!

The first command

GET fluent-bit-sec-2024.07.17/_count

returned:

{
  "count": 28763834,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  }
}

Looks fine with a lot of entries.

But if I search the data using the DQL search, I get NO results. :cry:

image
image1259×238 25 KB

The second command returned:

{
  "took": 1699,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 10000,
      "relation": "gte"
    },
    "max_score": null,
    "hits": []
  },
  "aggregations": {
    "test": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "key": "fluent-bit-sec-2024.07.17",
          "doc_count": 21963753
        }
      ]
    }
  }
}

Do you havy another idea what else I could check?

4

Maybe it’s caused by the timezone issue, you can expand the time range to check if any results could be returned or check the query DSL in the browser’s console:

image
image2660×880 253 KB

5

Check for hidden or corrupt documents causing discrepancies in the OpenSearch Dashboard. Ensure index settings and mappings are correct and consistent. Review logs and security settings for potential vulnerabilities or errors in the 2.8.0 Docker image, and investigate any unusual patterns or entries.

6

Thank you very much. Your suggestions sound promising.
Could you give me a hint on how to achieve these?

  • Ensure index settings and mappings are correct and consistent
  • potential vulnerabilities or errors in the 2.8.0 Docker image
  • investigate any unusual patterns or entries with Millions of documents?