Slow query performance issue

Plugins Performance Analyzer
21

All my data node keep hiting 100% of opensearch_os_mem_used_percent. Master and client are fine, and they use 20% to 40% memery

image
image1920×533 127 KB

For example, data node DD0Tc2xHS2qCuzDfQQ9njA
image
image3309×919 243 KB

I uses opensearch-operator to manage them, and the memory limit for data nodes is 16g

[root@syslog-k3s-1 ~]# kubectl top pods -n my-system-os 
NAME                                        CPU(cores)   MEMORY(bytes)   
my-system-os-coordinators-0                89m          5539Mi          
my-system-os-coordinators-1                83m          5499Mi          
my-system-os-coordinators-2                136m         5498Mi          
my-system-os-dashboards-5755bb6cf4-n9jbg   5m           152Mi           
my-system-os-masters-0                     16m          3103Mi          
my-system-os-masters-1                     13m          3395Mi          
my-system-os-masters-2                     28m          3235Mi          
my-system-os-nodes-0                       378m         13891Mi         
my-system-os-nodes-1                       17m          12218Mi         
my-system-os-nodes-10                      473m         12883Mi         
my-system-os-nodes-11                      24m          12482Mi         
my-system-os-nodes-12                      22m          10467Mi         
my-system-os-nodes-13                      370m         14640Mi         
my-system-os-nodes-14                      30m          11832Mi         
my-system-os-nodes-15                      448m         14324Mi         
my-system-os-nodes-2                       626m         13116Mi         
my-system-os-nodes-3                       18m          12594Mi         
my-system-os-nodes-4                       23m          11615Mi         
my-system-os-nodes-5                       27m          13973Mi         
my-system-os-nodes-6                       18m          11388Mi         
my-system-os-nodes-7                       587m         14321Mi         
my-system-os-nodes-8                       19m          11732Mi         
my-system-os-nodes-9                       22m          12071Mi         
[root@syslog-k3s-1 ~]#
22

I wonder if I should keep increasing the memory limit? When my old ElasticSearch cluster was working, it didn’t need 16g memory for data nodes.

23

@deng47 it seems like the heap pressure does indeed slow down your queries, could you try to increase the heap as you suggested?

24

I increased the resources.limits.memory of data nodes from 16g to 32g, and updated the JVM options from -Xms8g Xms8g to -Xmx16g -Xms16g with opensearch-operator.

apiVersion: opensearch.opster.io/v1
kind: OpenSearchCluster
metadata:
  name: my-system-os
  namespace: my-system-os
spec:
  general:
    version: "2.4.1"
    httpPort: 9200
    vendor: opensearch
    serviceName: my-system-os
    pluginsList: ["repository-s3", "https://github.com/aiven/prometheus-exporter-plugin-for-opensearch/releases/download/2.4.1.0/prometheus-exporter-2.4.1.0.zip"]
    additionalConfig:
      plugins.security.ssl.transport.enforce_hostname_verification: "false"
    additionalVolumes:
    - name: ldap-cert
      path: /usr/share/opensearch/config/ldap/
      restartPods: true
      secret:
        secretName: ldap-cert
  security:
    config:
      securityConfigSecret:
      ##Pre create this secret with required security configs, to override the default settings
        name: securityconfig-secret
      adminSecret:
        name: opensearch-admin-cert
    tls:
      transport:
        generate: false
        perNode: false
        secret:
          name: opensearch-cert
        caSecret:
          name: opensearch-ca-cert
        nodesDn: [ SKIP ]
        adminDn: [ "CN=A" ]
      http:
        generate: false
        secret:
          name: opensearch-cert
        caSecret:
          name: opensearch-ca-cert
  dashboards:
    version: "2.4.1"
    enable: true
    replicas: 1
    tls:
      enable: false
    additionalConfig:
      opensearch_security.multitenancy.enabled: "true"
      data.search.usageTelemetry.enabled: "true"
      opensearch.requestTimeout: "600000"
      opensearch.shardTimeout: "600000"
    tolerations:
    - key: "node-role.kubernetes.io/worker"
      operator: "Exists"
      effect: "NoSchedule"
    resources:
      requests:
         memory: "1Gi"
         cpu: "500m"
      limits:
         memory: "8Gi"
         cpu: "4"
  confMgmt:
    smartScaler: true
  nodePools:
    - component: masters
      replicas: 3
      jvm: -Xmx2g -Xms2g -Dopensearch.allow_insecure_settings=true
      diskSize: "10Gi"
      NodeSelector:
      tolerations:
      - key: "node-role.kubernetes.io/worker"
        operator: "Exists"
        effect: "NoSchedule"
      resources:
         requests:
            memory: "2Gi"
            cpu: "500m"
         limits:
            memory: "16Gi"
            cpu: "4"
      roles:
        - "cluster_manager"
    - component: nodes
      replicas: 16
      jvm: -Xmx16g -Xms16g -Dopensearch.allow_insecure_settings=true
      diskSize: "8Ti"
      NodeSelector:
      tolerations:
      - key: "node-role.kubernetes.io/worker"
        operator: "Exists"
        effect: "NoSchedule"
      resources:
         requests:
            memory: "2Gi"
            cpu: "500m"
         limits:
            memory: "32Gi"
            cpu: "4"
      roles:
        - "data"
    - component: coordinators
      replicas: 3
      jvm: -Xmx4g -Xms4g -Dopensearch.allow_insecure_settings=true
      diskSize: "10Gi"
      NodeSelector:
      tolerations:
      - key: "node-role.kubernetes.io/worker"
        operator: "Exists"
        effect: "NoSchedule"
      resources:
         requests:
            memory: "2Gi"
            cpu: "500m"
         limits:
            memory: "16Gi"
            cpu: "4"
      roles:
        - "ingest"

However, the same query still took 17+ seconds

{
  "took" : 17291,
  "timed_out" : false,
  "_shards" : {
    "total" : 93,
    "successful" : 93,
    "skipped" : 45,
    "failed" : 0
  },

Search time spent on my-system-syslog-000014 took 20+ seconds

{
        "id" : "[5pwR-7jcQraOj--MGQVrjw][my-system-syslog-000014][0]",
        "inbound_network_time_in_millis" : 1,
        "outbound_network_time_in_millis" : 1,
        "searches" : [
          {
            "query" : [
              {
                "type" : "ConstantScoreQuery",
                "description" : "ConstantScore(timestamp:[1674187200000 TO 1674273600000])",
                "time_in_nanos" : 21233855750,
                "breakdown" : {
                  "set_min_competitive_score_count" : 0,
                  "match_count" : 0,
                  "shallow_advance_count" : 0,
                  "set_min_competitive_score" : 0,
                  "next_doc" : 21225484244,
                  "match" : 0,
                  "next_doc_count" : 270770395,
                  "score_count" : 0,
                  "compute_max_score_count" : 0,
                  "compute_max_score" : 0,
                  "advance" : 28289,
                  "advance_count" : 32,
                  "score" : 0,
                  "build_scorer_count" : 64,
                  "create_weight" : 49649,
                  "shallow_advance" : 0,
                  "create_weight_count" : 1,
                  "build_scorer" : 8293568
                },
                "children" : [
                  {
                    "type" : "IndexOrDocValuesQuery",
                    "description" : "timestamp:[1674187200000 TO 1674273600000]",
                    "time_in_nanos" : 7086083341,
                    "breakdown" : {
                      "set_min_competitive_score_count" : 0,
                      "match_count" : 0,
                      "shallow_advance_count" : 0,
                      "set_min_competitive_score" : 0,
                      "next_doc" : 7077839224,
                      "match" : 0,
                      "next_doc_count" : 270770395,
                      "score_count" : 0,
                      "compute_max_score_count" : 0,
                      "compute_max_score" : 0,
                      "advance" : 20009,
                      "advance_count" : 32,
                      "score" : 0,
                      "build_scorer_count" : 64,
                      "create_weight" : 11819,
                      "shallow_advance" : 0,
                      "create_weight_count" : 1,
                      "build_scorer" : 8212289
                    }
                  }
                ]
              }
            ],
            "rewrite_time" : 399924,
            "collector" : [
              {
                "name" : "SimpleFieldCollector",
                "reason" : "search_top_hits",
                "time_in_nanos" : 9125626231
              }
            ]
          }
        ],
        "aggregations" : [ ]
      },
       {
        "id" : "[dYwPIUy0SdGb36b4HCl7yw][my-system-syslog-000014][2]",
        "inbound_network_time_in_millis" : 6,
        "outbound_network_time_in_millis" : 0,
        "searches" : [
          {
            "query" : [
              {
                "type" : "ConstantScoreQuery",
                "description" : "ConstantScore(timestamp:[1674187200000 TO 1674273600000])",
                "time_in_nanos" : 21350447485,
                "breakdown" : {
                  "set_min_competitive_score_count" : 0,
                  "match_count" : 0,
                  "shallow_advance_count" : 0,
                  "set_min_competitive_score" : 0,
                  "next_doc" : 21329952918,
                  "match" : 0,
                  "next_doc_count" : 270741156,
                  "score_count" : 0,
                  "compute_max_score_count" : 0,
                  "compute_max_score" : 0,
                  "advance" : 85577,
                  "advance_count" : 36,
                  "score" : 0,
                  "build_scorer_count" : 72,
                  "create_weight" : 34579,
                  "shallow_advance" : 0,
                  "create_weight_count" : 1,
                  "build_scorer" : 20374411
                },
                "children" : [
                  {
                    "type" : "IndexOrDocValuesQuery",
                    "description" : "timestamp:[1674187200000 TO 1674273600000]",
                    "time_in_nanos" : 7100630096,
                    "breakdown" : {
                      "set_min_competitive_score_count" : 0,
                      "match_count" : 0,
                      "shallow_advance_count" : 0,
                      "set_min_competitive_score" : 0,
                      "next_doc" : 7080218607,
                      "match" : 0,
                      "next_doc_count" : 270741156,
                      "score_count" : 0,
                      "compute_max_score_count" : 0,
                      "compute_max_score" : 0,
                      "advance" : 76388,
                      "advance_count" : 36,
                      "score" : 0,
                      "build_scorer_count" : 72,
                      "create_weight" : 11619,
                      "shallow_advance" : 0,
                      "create_weight_count" : 1,
                      "build_scorer" : 20323482
                    }
                  }
                ]
              }
            ],
            "rewrite_time" : 120867,
            "collector" : [
              {
                "name" : "SimpleFieldCollector",
                "reason" : "search_top_hits",
                "time_in_nanos" : 8954326204
              }
            ]
          }
        ],
        "aggregations" : [ ]
      },
      {
        "id" : "[g1LncdRoRCez4uKHcw12YQ][my-system-syslog-000014][1]",
        "inbound_network_time_in_millis" : 1,
        "outbound_network_time_in_millis" : 0,
        "searches" : [
          {
            "query" : [
              {
                "type" : "ConstantScoreQuery",
                "description" : "ConstantScore(timestamp:[1674187200000 TO 1674273600000])",
                "time_in_nanos" : 21383526392,
                "breakdown" : {
                  "set_min_competitive_score_count" : 0,
                  "match_count" : 0,
                  "shallow_advance_count" : 0,
                  "set_min_competitive_score" : 0,
                  "next_doc" : 21369364100,
                  "match" : 0,
                  "next_doc_count" : 270753955,
                  "score_count" : 0,
                  "compute_max_score_count" : 0,
                  "compute_max_score" : 0,
                  "advance" : 19119,
                  "advance_count" : 34,
                  "score" : 0,
                  "build_scorer_count" : 68,
                  "create_weight" : 18220,
                  "shallow_advance" : 0,
                  "create_weight_count" : 1,
                  "build_scorer" : 14124953
                },
                "children" : [
                  {
                    "type" : "IndexOrDocValuesQuery",
                    "description" : "timestamp:[1674187200000 TO 1674273600000]",
                    "time_in_nanos" : 7146843105,
                    "breakdown" : {
                      "set_min_competitive_score_count" : 0,
                      "match_count" : 0,
                      "shallow_advance_count" : 0,
                      "set_min_competitive_score" : 0,
                      "next_doc" : 7132722812,
                      "match" : 0,
                      "next_doc_count" : 270753955,
                      "score_count" : 0,
                      "compute_max_score_count" : 0,
                      "compute_max_score" : 0,
                      "advance" : 11039,
                      "advance_count" : 34,
                      "score" : 0,
                      "build_scorer_count" : 68,
                      "create_weight" : 5430,
                      "shallow_advance" : 0,
                      "create_weight_count" : 1,
                      "build_scorer" : 14103824
                    }
                  }
                ]
              }
            ],
            "rewrite_time" : 72299,
            "collector" : [
              {
                "name" : "SimpleFieldCollector",
                "reason" : "search_top_hits",
                "time_in_nanos" : 9013413861
              }
            ]
          }
        ],
        "aggregations" : [ ]
      },
25

The memory usage of some data nodes is hitting 100% again. Increasing the mem limit and heap size seems doesn’t help at all

image
image3327×927 222 KB 

[root@syslog-k3s-1 my-system-os]# kubectl top pods -n my-system-os 
NAME                                        CPU(cores)   MEMORY(bytes)   
my-system-os-coordinators-0                72m          5495Mi          
my-system-os-coordinators-1                86m          5506Mi          
my-system-os-coordinators-2                119m         5403Mi          
my-system-os-dashboards-5755bb6cf4-n9jbg   1m           156Mi           
my-system-os-masters-0                     24m          3075Mi          
my-system-os-masters-1                     13m          2619Mi          
my-system-os-masters-2                     21m          2678Mi          
my-system-os-nodes-0                       21m          17533Mi         
my-system-os-nodes-1                       17m          17902Mi         
my-system-os-nodes-10                      966m         24319Mi         
my-system-os-nodes-11                      20m          17627Mi         
my-system-os-nodes-12                      39m          18743Mi         
my-system-os-nodes-13                      365m         23464Mi         
my-system-os-nodes-14                      49m          17761Mi         
my-system-os-nodes-15                      930m         25942Mi         
my-system-os-nodes-2                       476m         22952Mi         
my-system-os-nodes-3                       26m          17699Mi         
my-system-os-nodes-4                       25m          17953Mi         
my-system-os-nodes-5                       37m          17793Mi         
my-system-os-nodes-6                       31m          18007Mi         
my-system-os-nodes-7                       495m         24740Mi         
my-system-os-nodes-8                       22m          17742Mi         
my-system-os-nodes-9                       22m          17741Mi
26

Indeed, increasing the memory limit was a low hanging fruit to pinpoint where exactly is the problem. We probably need to dig much deeper to understand what possibly could make such big difference between ES and OS. Could you first please share the index mapping if possible? It’s not a dynamic one, right?

27

Hi @reta I created a template like below. I use the same template on both ES and OS

PUT _template/my-system-syslog-template
{
    "order" : 10,
    "index_patterns" : [
      "my-system-syslog*"
    ],
    "settings" : {
      "index" : {
        "codec" : "best_compression",
        "mapping" : {
          "ignore_malformed" : "true"
        },
        "refresh_interval" : "30s",
        "number_of_shards" : "3",
        "number_of_replicas" : "1",
        "plugins" : {
          "index_state_management" : {
            "rollover_alias" : "my-system-syslog"
          }
        }
      }
    },
    "mappings" : {
      "dynamic" : "true",
      "properties" : {
        "severity" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "ignore_above" : 256,
              "type" : "keyword"
            }
          }
        },
        "syslog_pid" : {
          "type" : "keyword"
        },
        "procid" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "ignore_above" : 256,
              "type" : "keyword"
            }
          }
        },
        "syslog_program" : {
          "type" : "keyword"
        },
        "message" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "ignore_above" : 256,
              "type" : "keyword"
            }
          }
        },
        "priority" : {
          "type" : "short"
        },
        "syslog_message" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "ignore_above" : 256,
              "type" : "keyword"
            }
          }
        },
        "type" : {
          "type" : "keyword"
        },
        "hostname" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "ignore_above" : 256,
              "type" : "keyword"
            }
          }
        },
        "@timestamp" : {
          "type" : "date"
        },
        "@version" : {
          "type" : "keyword"
        },
        "syslog_rawmesg" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "ignore_above" : 256,
              "type" : "keyword"
            }
          }
        },
        "tag" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "ignore_above" : 256,
              "type" : "keyword"
            }
          }
        },
        "facility" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "ignore_above" : 256,
              "type" : "keyword"
            }
          }
        },
        "syslog_pri" : {
          "type" : "keyword"
        },
        "timestamp" : {
          "type" : "date"
        }
      }
    },
    "aliases" : { }
  }
28

I set "dynamic" : "true" in my template. Here is the actual mapping of my-system-syslog-000014

GET my-system-syslog-000014
{
  "my-system-syslog-000014" : {
    "aliases" : {
      "my-system-syslog" : {
        "is_write_index" : false
      }
    },
    "mappings" : {
      "dynamic" : "true",
      "properties" : {
        "@timestamp" : {
          "type" : "date"
        },
        "@version" : {
          "type" : "keyword"
        },
        "facility" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "hostname" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "message" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "priority" : {
          "type" : "short"
        },
        "procid" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "severity" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "syslog_message" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "syslog_pid" : {
          "type" : "keyword"
        },
        "syslog_pri" : {
          "type" : "keyword"
        },
        "syslog_program" : {
          "type" : "keyword"
        },
        "syslog_rawmesg" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "tag" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "timestamp" : {
          "type" : "date"
        },
        "type" : {
          "type" : "keyword"
        }
      }
    },
    "settings" : {
      "index" : {
        "codec" : "best_compression",
        "mapping" : {
          "ignore_malformed" : "true"
        },
        "refresh_interval" : "30s",
        "number_of_shards" : "3",
        "plugins" : {
          "index_state_management" : {
            "rollover_alias" : "my-system-syslog"
          }
        },
        "provided_name" : "my-system-syslog-000014",
        "creation_date" : "1674214457357",
        "number_of_replicas" : "1",
        "uuid" : "nn_99edDRHiwBE9j5tEJ7g",
        "version" : {
          "created" : "136257927"
        }
      }
    }
  }
}

And Some sample messages

      {
        "_index" : "my-system-syslog-000014",
        "_id" : "UbkDz4UBWDrnINIatM5C",
        "_score" : 5.782648,
        "_source" : {
          "procid" : "2245045",
          "message" : " ethtool: could not get ethtool features for 8bv2bfr0",
          "hostname" : "my-system-host015",
          "tag" : "systemd-udevd[2245045]",
          "priority" : "28",
          "severity" : "warning",
          "facility" : "daemon",
          "timestamp" : "2023-01-20T11:36:01.185840+00:00"
        }
      },
      {
        "_index" : "my-system-syslog-000014",
        "_id" : "2xEt0YUBZv6xTjl5C-kq",
        "_score" : 5.782648,
        "_source" : {
          "procid" : "14067",
          "message" : "   /dev/loop0: Failed to get external handle [udev].",
          "hostname" : "host007",
          "tag" : "lvm[14067]",
          "priority" : "30",
          "severity" : "info",
          "facility" : "daemon",
          "timestamp" : "2023-01-20T16:30:26.748550+00:00"
        }
      }
29

Thanks @deng47 , I think I have a hint now: Document performance changes for `BEST_COMPRESSION` since ES 7.9 ¡ Issue #348 ¡ opensearch-project/documentation-website ¡ GitHub, I believe that OS and ES (at this version ranges) are using different Apache Lucene versions (8.8 on ES and 8.10 on OS), but the performance degradation in your case is really huge (like 30x). Do you have a chance to try out OS 1.0 [1] for a sake of experiment, it should be using same Apache Lucene version as ES 7.13. Thank you.

[1] Opensearch 1.0.0 ¡ OpenSearch

30

Thank you for the hint. I updated the syslog template and set the compression type to default. It’s a v2.4.1 OpenSearch. I assume the template will apply to the next new syslog index? I will create a new OS 1.0 cluster as well. It probably will take 2~3 days. So I have enough data to test it. I wonder what compression type I should try in OS 1.0? Default or best compression?

31

Thank you

I assume the template will apply to the next new syslog index?

Yes

So I have enough data to test it. I wonder what compression type I should try in OS 1.0? Default or best compression?

The one you used with ES, I think it is best compression, right?

Thank you.

32

It seems OS 1.0 doesn’t make much difference.
I deployed an OS 1.0 with helm chart(I failed to get opensearch-operator work with OS 1.0)

# cat Chart.yaml 
apiVersion: v2
appVersion: "1.0.0"
description: A Helm chart for OpenSearch
maintainers:
- name: DandyDeveloper
- name: gaiksaya
- name: peternied
- name: peterzhuamazon
- name: TheAlgo
name: opensearch
type: application
version: 1.5.4

There are 1 billion messages in my syslog index

image
image3137×708 88.9 KB 

GET /my-system-syslog/_search
{
  "profile": true,
  "version": true,
  "size": 2,
  "sort": [
    {
      "timestamp": {
        "order": "desc",
        "unmapped_type": "boolean"
      }
    }
  ],
  "stored_fields": [
    "*"
  ],
  "docvalue_fields": [
    {
      "field": "@timestamp",
      "format": "date_time"
    },
    {
      "field": "timestamp",
      "format": "date_time"
    }
  ],
  "_source": {
    "excludes": []
  },
  "query": {
    "bool": {
      "must": [],
      "filter": [
        {
          "match_all": {}
        },
        {
          "range": {
            "timestamp": {              
              "gte": "2023-02-03T21:29:31.382Z",
              "lte": "2023-02-05T21:29:31.382Z",
              "format": "strict_date_optional_time"
            }
          }
        }
      ],
      "should": [],
      "must_not": []
    }
  }
}

It took 14 seconds

{
  "took" : 14134,
  "timed_out" : false,
  "_shards" : {
    "total" : 6,
    "successful" : 6,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 10000,
      "relation" : "gte"
    },
    "max_score" : null,
    "hits" : [SKIP]
  },
  "profile" : {
    "shards" : [
      {
        "id" : "[4PYK1iBXR5WbA3Q9VkhunQ][my-system-syslog-000001][1]",
        "searches" : [
          {
            "query" : [
              {
                "type" : "BooleanQuery",
                "description" : "#(ConstantScore(timestamp:[1675459771382 TO 1675632571382]))^0.0 LongDistanceFeatureQuery(field=,origin=1675619292000,pivotDistance=162094814)",
                "time_in_nanos" : 16434629194,
                "breakdown" : {
                  "set_min_competitive_score_count" : 21211,
                  "match_count" : 0,
                  "shallow_advance_count" : 0,
                  "set_min_competitive_score" : 69085101,
                  "next_doc" : 5790855442,
                  "match" : 0,
                  "next_doc_count" : 116020638,
                  "score_count" : 116020638,
                  "compute_max_score_count" : 0,
                  "compute_max_score" : 0,
                  "advance" : 132798,
                  "advance_count" : 32,
                  "score" : 10571629429,
                  "build_scorer_count" : 64,
                  "create_weight" : 101179,
                  "shallow_advance" : 0,
                  "create_weight_count" : 1,
                  "build_scorer" : 2825245
                },
                "children" : [
                  {
                    "type" : "BoostQuery",
                    "description" : "(ConstantScore(timestamp:[1675459771382 TO 1675632571382]))^0.0",
                    "time_in_nanos" : 9266925468,
                    "breakdown" : {
                      "set_min_competitive_score_count" : 0,
                      "match_count" : 0,
                      "shallow_advance_count" : 0,
                      "set_min_competitive_score" : 0,
                      "next_doc" : 0,
                      "match" : 0,
                      "next_doc_count" : 0,
                      "score_count" : 0,
                      "compute_max_score_count" : 0,
                      "compute_max_score" : 0,
                      "advance" : 9265347053,
                      "advance_count" : 116685242,
                      "score" : 0,
                      "build_scorer_count" : 96,
                      "create_weight" : 44459,
                      "shallow_advance" : 0,
                      "create_weight_count" : 1,
                      "build_scorer" : 1533956
                    },
                    "children" : [
                      {
                        "type" : "IndexOrDocValuesQuery",
                        "description" : "timestamp:[1675459771382 TO 1675632571382]",
                        "time_in_nanos" : 3060103937,
                        "breakdown" : {
                          "set_min_competitive_score_count" : 0,
                          "match_count" : 0,
                          "shallow_advance_count" : 0,
                          "set_min_competitive_score" : 0,
                          "next_doc" : 0,
                          "match" : 0,
                          "next_doc_count" : 0,
                          "score_count" : 0,
                          "compute_max_score_count" : 0,
                          "compute_max_score" : 0,
                          "advance" : 3058607641,
                          "advance_count" : 116685242,
                          "score" : 0,
                          "build_scorer_count" : 96,
                          "create_weight" : 13340,
                          "shallow_advance" : 0,
                          "create_weight_count" : 1,
                          "build_scorer" : 1482956
                        }
                      }
                    ]
                  },
                  {
                    "type" : "LongDistanceFeatureQuery",
                    "description" : "LongDistanceFeatureQuery(field=,origin=1675619292000,pivotDistance=162094814)",
                    "time_in_nanos" : 7347862375,
                    "breakdown" : {
                      "set_min_competitive_score_count" : 21211,
                      "match_count" : 0,
                      "shallow_advance_count" : 96,
                      "set_min_competitive_score" : 67815398,
                      "next_doc" : 0,
                      "match" : 0,
                      "next_doc_count" : 0,
                      "score_count" : 116020638,
                      "compute_max_score_count" : 64,
                      "compute_max_score" : 14260,
                      "advance" : 3179829184,
                      "advance_count" : 116020666,
                      "score" : 4100024866,
                      "build_scorer_count" : 64,
                      "create_weight" : 7510,
                      "shallow_advance" : 11040,
                      "create_weight_count" : 1,
                      "build_scorer" : 160117
                    }
                  }
                ]
              }
            ],
            "rewrite_time" : 400274,
            "collector" : [
              {
                "name" : "SimpleFieldCollector",
                "reason" : "search_top_hits",
                "time_in_nanos" : 7581615580
              }
            ]
          }
        ],
        "aggregations" : [ ]
      },
      {
        "id" : "[7l_kGMWuQYK3__CZcpPxGw][my-system-syslog-000002][1]",
        "searches" : [
          {
            "query" : [
              {
                "type" : "BooleanQuery",
                "description" : "#(ConstantScore(timestamp:[1675459771382 TO 1675632571382]))^0.0 LongDistanceFeatureQuery(field=,origin=1675632601000,pivotDistance=166634855)",
                "time_in_nanos" : 278114037,
                "breakdown" : {
                  "set_min_competitive_score_count" : 1445,
                  "match_count" : 0,
                  "shallow_advance_count" : 0,
                  "set_min_competitive_score" : 23167770,
                  "next_doc" : 93320061,
                  "match" : 0,
                  "next_doc_count" : 1602259,
                  "score_count" : 1602259,
                  "compute_max_score_count" : 0,
                  "compute_max_score" : 0,
                  "advance" : 85770,
                  "advance_count" : 17,
                  "score" : 154936698,
                  "build_scorer_count" : 34,
                  "create_weight" : 1018291,
                  "shallow_advance" : 0,
                  "create_weight_count" : 1,
                  "build_scorer" : 5585447
                },
                "children" : [
                  {
                    "type" : "BoostQuery",
                    "description" : "(ConstantScore(timestamp:[1675459771382 TO 1675632571382]))^0.0",
                    "time_in_nanos" : 140371660,
                    "breakdown" : {
                      "set_min_competitive_score_count" : 0,
                      "match_count" : 0,
                      "shallow_advance_count" : 0,
                      "set_min_competitive_score" : 0,
                      "next_doc" : 0,
                      "match" : 0,
                      "next_doc_count" : 0,
                      "score_count" : 0,
                      "compute_max_score_count" : 0,
                      "compute_max_score" : 0,
                      "advance" : 137972071,
                      "advance_count" : 1607802,
                      "score" : 0,
                      "build_scorer_count" : 51,
                      "create_weight" : 390933,
                      "shallow_advance" : 0,
                      "create_weight_count" : 1,
                      "build_scorer" : 2008656
                    },
                    "children" : [
                      {
                        "type" : "IndexOrDocValuesQuery",
                        "description" : "timestamp:[1675459771382 TO 1675632571382]",
                        "time_in_nanos" : 51900530,
                        "breakdown" : {
                          "set_min_competitive_score_count" : 0,
                          "match_count" : 0,
                          "shallow_advance_count" : 0,
                          "set_min_competitive_score" : 0,
                          "next_doc" : 0,
                          "match" : 0,
                          "next_doc_count" : 0,
                          "score_count" : 0,
                          "compute_max_score_count" : 0,
                          "compute_max_score" : 0,
                          "advance" : 50390765,
                          "advance_count" : 1607802,
                          "score" : 0,
                          "build_scorer_count" : 51,
                          "create_weight" : 13620,
                          "shallow_advance" : 0,
                          "create_weight_count" : 1,
                          "build_scorer" : 1496145
                        }
                      }
                    ]
                  },
                  {
                    "type" : "LongDistanceFeatureQuery",
                    "description" : "LongDistanceFeatureQuery(field=,origin=1675632601000,pivotDistance=166634855)",
                    "time_in_nanos" : 137354314,
                    "breakdown" : {
                      "set_min_competitive_score_count" : 1445,
                      "match_count" : 0,
                      "shallow_advance_count" : 51,
                      "set_min_competitive_score" : 22530344,
                      "next_doc" : 0,
                      "match" : 0,
                      "next_doc_count" : 0,
                      "score_count" : 1602259,
                      "compute_max_score_count" : 34,
                      "compute_max_score" : 7600,
                      "advance" : 54069950,
                      "advance_count" : 1602275,
                      "score" : 59247827,
                      "build_scorer_count" : 34,
                      "create_weight" : 558269,
                      "shallow_advance" : 11440,
                      "create_weight_count" : 1,
                      "build_scorer" : 928884
                    }
                  }
                ]
              }
            ],
            "rewrite_time" : 133758,
            "collector" : [
              {
                "name" : "SimpleFieldCollector",
                "reason" : "search_top_hits",
                "time_in_nanos" : 147102579
              }
            ]
          }
        ],
        "aggregations" : [ ]
      },
      {
        "id" : "[OC4PGqpoR8myLtzjIKIQUQ][my-system-syslog-000001][0]",
        "searches" : [
          {
            "query" : [
              {
                "type" : "BooleanQuery",
                "description" : "#(ConstantScore(timestamp:[1675459771382 TO 1675632571382]))^0.0 LongDistanceFeatureQuery(field=,origin=1675619292000,pivotDistance=162447828)",
                "time_in_nanos" : 12710895801,
                "breakdown" : {
                  "set_min_competitive_score_count" : 17205,
                  "match_count" : 0,
                  "shallow_advance_count" : 0,
                  "set_min_competitive_score" : 81664641,
                  "next_doc" : 4481178182,
                  "match" : 0,
                  "next_doc_count" : 89038241,
                  "score_count" : 89038241,
                  "compute_max_score_count" : 0,
                  "compute_max_score" : 0,
                  "advance" : 94489,
                  "advance_count" : 27,
                  "score" : 8142561445,
                  "build_scorer_count" : 54,
                  "create_weight" : 17790,
                  "shallow_advance" : 0,
                  "create_weight_count" : 1,
                  "build_scorer" : 5379254
                },
                "children" : [
                  {
                    "type" : "BoostQuery",
                    "description" : "(ConstantScore(timestamp:[1675459771382 TO 1675632571382]))^0.0",
                    "time_in_nanos" : 7087665282,
                    "breakdown" : {
                      "set_min_competitive_score_count" : 0,
                      "match_count" : 0,
                      "shallow_advance_count" : 0,
                      "set_min_competitive_score" : 0,
                      "next_doc" : 0,
                      "match" : 0,
                      "next_doc_count" : 0,
                      "score_count" : 0,
                      "compute_max_score_count" : 0,
                      "compute_max_score" : 0,
                      "advance" : 7083507788,
                      "advance_count" : 89656781,
                      "score" : 0,
                      "build_scorer_count" : 81,
                      "create_weight" : 6390,
                      "shallow_advance" : 0,
                      "create_weight_count" : 1,
                      "build_scorer" : 4151104
                    },
                    "children" : [
                      {
                        "type" : "IndexOrDocValuesQuery",
                        "description" : "timestamp:[1675459771382 TO 1675632571382]",
                        "time_in_nanos" : 2355574999,
                        "breakdown" : {
                          "set_min_competitive_score_count" : 0,
                          "match_count" : 0,
                          "shallow_advance_count" : 0,
                          "set_min_competitive_score" : 0,
                          "next_doc" : 0,
                          "match" : 0,
                          "next_doc_count" : 0,
                          "score_count" : 0,
                          "compute_max_score_count" : 0,
                          "compute_max_score" : 0,
                          "advance" : 2351474763,
                          "advance_count" : 89656781,
                          "score" : 0,
                          "build_scorer_count" : 81,
                          "create_weight" : 1460,
                          "shallow_advance" : 0,
                          "create_weight_count" : 1,
                          "build_scorer" : 4098776
                        }
                      }
                    ]
                  },
                  {
                    "type" : "LongDistanceFeatureQuery",
                    "description" : "LongDistanceFeatureQuery(field=,origin=1675619292000,pivotDistance=162447828)",
                    "time_in_nanos" : 5694928749,
                    "breakdown" : {
                      "set_min_competitive_score_count" : 17205,
                      "match_count" : 0,
                      "shallow_advance_count" : 81,
                      "set_min_competitive_score" : 80569774,
                      "next_doc" : 0,
                      "match" : 0,
                      "next_doc_count" : 0,
                      "score_count" : 89038241,
                      "compute_max_score_count" : 54,
                      "compute_max_score" : 14450,
                      "advance" : 2420651208,
                      "advance_count" : 89038264,
                      "score" : 3193528578,
                      "build_scorer_count" : 54,
                      "create_weight" : 500,
                      "shallow_advance" : 11180,
                      "create_weight_count" : 1,
                      "build_scorer" : 153059
                    }
                  }
                ]
              }
            ],
            "rewrite_time" : 60209,
            "collector" : [
              {
                "name" : "SimpleFieldCollector",
                "reason" : "search_top_hits",
                "time_in_nanos" : 5666904156
              }
            ]
          }
        ],
        "aggregations" : [ ]
      },
      {
        "id" : "[OC4PGqpoR8myLtzjIKIQUQ][my-system-syslog-000002][2]",
        "searches" : [
          {
            "query" : [
              {
                "type" : "BooleanQuery",
                "description" : "#(ConstantScore(timestamp:[1675459771382 TO 1675632571382]))^0.0 LongDistanceFeatureQuery(field=,origin=1675632601000,pivotDistance=83371834)",
                "time_in_nanos" : 313157824,
                "breakdown" : {
                  "set_min_competitive_score_count" : 1629,
                  "match_count" : 0,
                  "shallow_advance_count" : 0,
                  "set_min_competitive_score" : 4022779,
                  "next_doc" : 141710118,
                  "match" : 0,
                  "next_doc_count" : 1827789,
                  "score_count" : 1827789,
                  "compute_max_score_count" : 0,
                  "compute_max_score" : 0,
                  "advance" : 46149,
                  "advance_count" : 21,
                  "score" : 166486663,
                  "build_scorer_count" : 42,
                  "create_weight" : 43210,
                  "shallow_advance" : 0,
                  "create_weight_count" : 1,
                  "build_scorer" : 848905
                },
                "children" : [
                  {
                    "type" : "BoostQuery",
                    "description" : "(ConstantScore(timestamp:[1675459771382 TO 1675632571382]))^0.0",
                    "time_in_nanos" : 144388324,
                    "breakdown" : {
                      "set_min_competitive_score_count" : 0,
                      "match_count" : 0,
                      "shallow_advance_count" : 0,
                      "set_min_competitive_score" : 0,
                      "next_doc" : 0,
                      "match" : 0,
                      "next_doc_count" : 0,
                      "score_count" : 0,
                      "compute_max_score_count" : 0,
                      "compute_max_score" : 0,
                      "advance" : 144045381,
                      "advance_count" : 1832162,
                      "score" : 0,
                      "build_scorer_count" : 63,
                      "create_weight" : 20770,
                      "shallow_advance" : 0,
                      "create_weight_count" : 1,
                      "build_scorer" : 322173
                    },
                    "children" : [
                      {
                        "type" : "IndexOrDocValuesQuery",
                        "description" : "timestamp:[1675459771382 TO 1675632571382]",
                        "time_in_nanos" : 48486175,
                        "breakdown" : {
                          "set_min_competitive_score_count" : 0,
                          "match_count" : 0,
                          "shallow_advance_count" : 0,
                          "set_min_competitive_score" : 0,
                          "next_doc" : 0,
                          "match" : 0,
                          "next_doc_count" : 0,
                          "score_count" : 0,
                          "compute_max_score_count" : 0,
                          "compute_max_score" : 0,
                          "advance" : 48178892,
                          "advance_count" : 1832162,
                          "score" : 0,
                          "build_scorer_count" : 63,
                          "create_weight" : 11560,
                          "shallow_advance" : 0,
                          "create_weight_count" : 1,
                          "build_scorer" : 295723
                        }
                      }
                    ]
                  },
                  {
                    "type" : "LongDistanceFeatureQuery",
                    "description" : "LongDistanceFeatureQuery(field=,origin=1675632601000,pivotDistance=83371834)",
                    "time_in_nanos" : 122815074,
                    "breakdown" : {
                      "set_min_competitive_score_count" : 1629,
                      "match_count" : 0,
                      "shallow_advance_count" : 63,
                      "set_min_competitive_score" : 3923567,
                      "next_doc" : 0,
                      "match" : 0,
                      "next_doc_count" : 0,
                      "score_count" : 1827789,
                      "compute_max_score_count" : 42,
                      "compute_max_score" : 6520,
                      "advance" : 55254900,
                      "advance_count" : 1827804,
                      "score" : 63559998,
                      "build_scorer_count" : 42,
                      "create_weight" : 1990,
                      "shallow_advance" : 6400,
                      "create_weight_count" : 1,
                      "build_scorer" : 61699
                    }
                  }
                ]
              }
            ],
            "rewrite_time" : 90208,
            "collector" : [
              {
                "name" : "SimpleFieldCollector",
                "reason" : "search_top_hits",
                "time_in_nanos" : 116284605
              }
            ]
          }
        ],
        "aggregations" : [ ]
      },
      {
        "id" : "[ghVyg3VTSXe-lgPY4YFGmw][my-system-syslog-000001][2]",
        "searches" : [
          {
            "query" : [
              {
                "type" : "BooleanQuery",
                "description" : "#(ConstantScore(DocValuesFieldExistsQuery [field=timestamp]))^0.0 LongDistanceFeatureQuery(field=,origin=1675619292000,pivotDistance=77414340)",
                "time_in_nanos" : 6695145955,
                "breakdown" : {
                  "set_min_competitive_score_count" : 14623,
                  "match_count" : 0,
                  "shallow_advance_count" : 0,
                  "set_min_competitive_score" : 67124691,
                  "next_doc" : 2309582355,
                  "match" : 0,
                  "next_doc_count" : 45968168,
                  "score_count" : 45968168,
                  "compute_max_score_count" : 0,
                  "compute_max_score" : 0,
                  "advance" : 170738,
                  "advance_count" : 32,
                  "score" : 4312414611,
                  "build_scorer_count" : 64,
                  "create_weight" : 32639,
                  "shallow_advance" : 0,
                  "create_weight_count" : 1,
                  "build_scorer" : 5820921
                },
                "children" : [
                  {
                    "type" : "BoostQuery",
                    "description" : "(ConstantScore(DocValuesFieldExistsQuery [field=timestamp]))^0.0",
                    "time_in_nanos" : 3739395509,
                    "breakdown" : {
                      "set_min_competitive_score_count" : 0,
                      "match_count" : 0,
                      "shallow_advance_count" : 0,
                      "set_min_competitive_score" : 0,
                      "next_doc" : 0,
                      "match" : 0,
                      "next_doc_count" : 0,
                      "score_count" : 0,
                      "compute_max_score_count" : 0,
                      "compute_max_score" : 0,
                      "advance" : 3738108348,
                      "advance_count" : 46757412,
                      "score" : 0,
                      "build_scorer_count" : 96,
                      "create_weight" : 12010,
                      "shallow_advance" : 0,
                      "create_weight_count" : 1,
                      "build_scorer" : 1275151
                    },
                    "children" : [
                      {
                        "type" : "DocValuesFieldExistsQuery",
                        "description" : "DocValuesFieldExistsQuery [field=timestamp]",
                        "time_in_nanos" : 1232753680,
                        "breakdown" : {
                          "set_min_competitive_score_count" : 0,
                          "match_count" : 0,
                          "shallow_advance_count" : 0,
                          "set_min_competitive_score" : 0,
                          "next_doc" : 0,
                          "match" : 0,
                          "next_doc_count" : 0,
                          "score_count" : 0,
                          "compute_max_score_count" : 0,
                          "compute_max_score" : 0,
                          "advance" : 1232386755,
                          "advance_count" : 46757412,
                          "score" : 0,
                          "build_scorer_count" : 96,
                          "create_weight" : 4830,
                          "shallow_advance" : 0,
                          "create_weight_count" : 1,
                          "build_scorer" : 362095
                        }
                      }
                    ]
                  },
                  {
                    "type" : "LongDistanceFeatureQuery",
                    "description" : "LongDistanceFeatureQuery(field=,origin=1675619292000,pivotDistance=77414340)",
                    "time_in_nanos" : 2967003456,
                    "breakdown" : {
                      "set_min_competitive_score_count" : 14623,
                      "match_count" : 0,
                      "shallow_advance_count" : 96,
                      "set_min_competitive_score" : 65593429,
                      "next_doc" : 0,
                      "match" : 0,
                      "next_doc_count" : 0,
                      "score_count" : 45968168,
                      "compute_max_score_count" : 64,
                      "compute_max_score" : 14890,
                      "advance" : 1242610219,
                      "advance_count" : 45968194,
                      "score" : 1657651767,
                      "build_scorer_count" : 64,
                      "create_weight" : 2129,
                      "shallow_advance" : 13280,
                      "create_weight_count" : 1,
                      "build_scorer" : 1117742
                    }
                  }
                ]
              }
            ],
            "rewrite_time" : 103167,
            "collector" : [
              {
                "name" : "SimpleFieldCollector",
                "reason" : "search_top_hits",
                "time_in_nanos" : 4000925774
              }
            ]
          }
        ],
        "aggregations" : [ ]
      },
      {
        "id" : "[ghVyg3VTSXe-lgPY4YFGmw][my-system-syslog-000002][0]",
        "searches" : [
          {
            "query" : [
              {
                "type" : "BooleanQuery",
                "description" : "#(ConstantScore(timestamp:[1675459771382 TO 1675632571382]))^0.0 LongDistanceFeatureQuery(field=,origin=1675632601000,pivotDistance=167185269)",
                "time_in_nanos" : 311026237,
                "breakdown" : {
                  "set_min_competitive_score_count" : 1953,
                  "match_count" : 0,
                  "shallow_advance_count" : 0,
                  "set_min_competitive_score" : 12863749,
                  "next_doc" : 118061126,
                  "match" : 0,
                  "next_doc_count" : 1800554,
                  "score_count" : 1800554,
                  "compute_max_score_count" : 0,
                  "compute_max_score" : 0,
                  "advance" : 89729,
                  "advance_count" : 19,
                  "score" : 174271053,
                  "build_scorer_count" : 38,
                  "create_weight" : 779717,
                  "shallow_advance" : 0,
                  "create_weight_count" : 1,
                  "build_scorer" : 4960863
                },
                "children" : [
                  {
                    "type" : "BoostQuery",
                    "description" : "(ConstantScore(timestamp:[1675459771382 TO 1675632571382]))^0.0",
                    "time_in_nanos" : 147050314,
                    "breakdown" : {
                      "set_min_competitive_score_count" : 0,
                      "match_count" : 0,
                      "shallow_advance_count" : 0,
                      "set_min_competitive_score" : 0,
                      "next_doc" : 0,
                      "match" : 0,
                      "next_doc_count" : 0,
                      "score_count" : 0,
                      "compute_max_score_count" : 0,
                      "compute_max_score" : 0,
                      "advance" : 145031649,
                      "advance_count" : 1805433,
                      "score" : 0,
                      "build_scorer_count" : 57,
                      "create_weight" : 321465,
                      "shallow_advance" : 0,
                      "create_weight_count" : 1,
                      "build_scorer" : 1697200
                    },
                    "children" : [
                      {
                        "type" : "IndexOrDocValuesQuery",
                        "description" : "timestamp:[1675459771382 TO 1675632571382]",
                        "time_in_nanos" : 49632241,
                        "breakdown" : {
                          "set_min_competitive_score_count" : 0,
                          "match_count" : 0,
                          "shallow_advance_count" : 0,
                          "set_min_competitive_score" : 0,
                          "next_doc" : 0,
                          "match" : 0,
                          "next_doc_count" : 0,
                          "score_count" : 0,
                          "compute_max_score_count" : 0,
                          "compute_max_score" : 0,
                          "advance" : 48655828,
                          "advance_count" : 1805433,
                          "score" : 0,
                          "build_scorer_count" : 57,
                          "create_weight" : 12060,
                          "shallow_advance" : 0,
                          "create_weight_count" : 1,
                          "build_scorer" : 964353
                        }
                      }
                    ]
                  },
                  {
                    "type" : "LongDistanceFeatureQuery",
                    "description" : "LongDistanceFeatureQuery(field=,origin=1675632601000,pivotDistance=167185269)",
                    "time_in_nanos" : 130619084,
                    "breakdown" : {
                      "set_min_competitive_score_count" : 1953,
                      "match_count" : 0,
                      "shallow_advance_count" : 57,
                      "set_min_competitive_score" : 12232800,
                      "next_doc" : 0,
                      "match" : 0,
                      "next_doc_count" : 0,
                      "score_count" : 1800554,
                      "compute_max_score_count" : 38,
                      "compute_max_score" : 5640,
                      "advance" : 51796573,
                      "advance_count" : 1800571,
                      "score" : 65231595,
                      "build_scorer_count" : 38,
                      "create_weight" : 416593,
                      "shallow_advance" : 14280,
                      "create_weight_count" : 1,
                      "build_scorer" : 921603
                    }
                  }
                ]
              }
            ],
            "rewrite_time" : 150776,
            "collector" : [
              {
                "name" : "SimpleFieldCollector",
                "reason" : "search_top_hits",
                "time_in_nanos" : 165877216
              }
            ]
          }
        ],
        "aggregations" : [ ]
      }
    ]
  }
}

The mapping of the index

{
  "my-system-syslog-000001" : {
    "aliases" : {
      "my-system-syslog" : {
        "is_write_index" : false
      }
    },
    "mappings" : {
      "dynamic" : "true",
      "properties" : {
        "@timestamp" : {
          "type" : "date"
        },
        "@version" : {
          "type" : "keyword"
        },
        "facility" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "hostname" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "message" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "priority" : {
          "type" : "short"
        },
        "procid" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "severity" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "syslog_message" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "syslog_pid" : {
          "type" : "keyword"
        },
        "syslog_pri" : {
          "type" : "keyword"
        },
        "syslog_program" : {
          "type" : "keyword"
        },
        "syslog_rawmesg" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "tag" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "timestamp" : {
          "type" : "date"
        },
        "type" : {
          "type" : "keyword"
        }
      }
    },
    "settings" : {
      "index" : {
        "codec" : "best_compression",
        "mapping" : {
          "ignore_malformed" : "true"
        },
        "refresh_interval" : "30s",
        "number_of_shards" : "3",
        "plugins" : {
          "index_state_management" : {
            "rollover_alias" : "my-system-syslog"
          }
        },
        "provided_name" : "my-system-syslog-000001",
        "creation_date" : "1675478923643",
        "number_of_replicas" : "1",
        "uuid" : "ANC07EvlQFKvpGuZ2smvKQ",
        "version" : {
          "created" : "135217827"
        }
      }
    }
  }
}

The rollover didn’t work correctly, so the shard size is 50g

my-system-syslog-000001 1 p STARTED 343689584 50.4gb 10.42.10.191 dev-os-data-6
my-system-syslog-000001 1 r STARTED 343689584 50.6gb 10.42.8.87   dev-os-data-7
my-system-syslog-000001 2 p STARTED 343768745 50.5gb 10.42.9.227  dev-os-data-0
my-system-syslog-000001 2 r STARTED 343768745 50.5gb 10.42.12.228 dev-os-data-4
my-system-syslog-000001 0 p STARTED 343682415 50.5gb 10.42.15.63  dev-os-data-1
my-system-syslog-000001 0 r STARTED 343682415 50.5gb 10.42.18.110 dev-os-data-3
my-system-syslog-000002 1 r STARTED   6455011    1gb 10.42.18.110 dev-os-data-3
my-system-syslog-000002 1 p STARTED   6442701  1.1gb 10.42.3.223  dev-os-data-5
my-system-syslog-000002 2 p STARTED   6458507    1gb 10.42.15.63  dev-os-data-1
my-system-syslog-000002 2 r STARTED   6458507  1.1gb 10.42.10.191 dev-os-data-6
my-system-syslog-000002 0 r STARTED   6456466    1gb 10.42.9.227  dev-os-data-0
my-system-syslog-000002 0 p STARTED   6444154  1.1gb 10.42.7.93   dev-os-data-2
33

There are 8 data nodes in my OS 1.0 cluster. The heap size is 8g. Memory usage looks fine, never hit the memory limit which is much better than my OS 2.4.1 cluster

NAME                                                   CPU(cores)   MEMORY(bytes)   
dev-dashboard-opensearch-dashboards-77fb584595-gngf8   1m           142Mi           
dev-os-client-0                                        18m          7005Mi          
dev-os-client-1                                        22m          7009Mi          
dev-os-data-0                                          35m          11820Mi         
dev-os-data-1                                          23m          13184Mi         
dev-os-data-2                                          29m          9838Mi          
dev-os-data-3                                          42m          11864Mi         
dev-os-data-4                                          15m          12587Mi         
dev-os-data-5                                          18m          10015Mi         
dev-os-data-6                                          19m          13219Mi         
dev-os-data-7                                          14m          12954Mi         
dev-os-master-0                                        19m          4900Mi          
dev-os-master-1                                        17m          4826Mi          
dev-os-master-2                                        15m          4869Mi          
elastic-exporter-587b48df77-dpr2d                      11m          20Mi

image
image1920×887 175 KB

34

New mapping has been applied to new indices in my OS 2.4.1 cluster. Index.codec is set to default in my-system-syslog-000034.

GET /my-system-syslog/_search
{
  "profile": true,
  "version": true,
  "size": 2,
  "sort": [
    {
      "timestamp": {
        "order": "desc",
        "unmapped_type": "boolean"
      }
    }
  ],
  "stored_fields": [
    "*"
  ],
  "docvalue_fields": [
    {
      "field": "@timestamp",
      "format": "date_time"
    },
    {
      "field": "timestamp",
      "format": "date_time"
    }
  ],
  "_source": {
    "excludes": []
  },
  "query": {
    "bool": {
      "must": [],
      "filter": [
        {
          "match_all": {}
        },
        {
          "range": {
            "timestamp": {              
              "gte": "2023-02-03T16:00:00.000Z",
              "lte": "2023-02-04T16:00:00.000Z",
              "format": "strict_date_optional_time"
            }
          }
        }
      ],
      "should": [],
      "must_not": []
    }
  }
}

However, I see no improvement. It still takes 17 seconds

  "took" : 17835,
  "timed_out" : false,
  "_shards" : {
    "total" : 105,
    "successful" : 105,
    "skipped" : 30,
    "failed" : 0
  },

 {
        "id" : "[IHm5ZVmkR4KH1p6kcQZUoQ][my-system-syslog-000034][1]",
        "inbound_network_time_in_millis" : 0,
        "outbound_network_time_in_millis" : 2,
        "searches" : [
          {
            "query" : [
              {
                "type" : "ConstantScoreQuery",
                "description" : "ConstantScore(*:*)",
                "time_in_nanos" : 19252485526,
                "breakdown" : {
                  "set_min_competitive_score_count" : 0,
                  "match_count" : 0,
                  "shallow_advance_count" : 0,
                  "set_min_competitive_score" : 0,
                  "next_doc" : 19252277490,
                  "match" : 0,
                  "next_doc_count" : 245672284,
                  "score_count" : 0,
                  "compute_max_score_count" : 0,
                  "compute_max_score" : 0,
                  "advance" : 7010,
                  "advance_count" : 32,
                  "score" : 0,
                  "build_scorer_count" : 64,
                  "create_weight" : 8190,
                  "shallow_advance" : 0,
                  "create_weight_count" : 1,
                  "build_scorer" : 192836
                },
                "children" : [
                  {
                    "type" : "MatchAllDocsQuery",
                    "description" : "*:*",
                    "time_in_nanos" : 6356615696,
                    "breakdown" : {
                      "set_min_competitive_score_count" : 0,
                      "match_count" : 0,
                      "shallow_advance_count" : 0,
                      "set_min_competitive_score" : 0,
                      "next_doc" : 6356438889,
                      "match" : 0,
                      "next_doc_count" : 245672284,
                      "score_count" : 0,
                      "compute_max_score_count" : 0,
                      "compute_max_score" : 0,
                      "advance" : 4040,
                      "advance_count" : 32,
                      "score" : 0,
                      "build_scorer_count" : 64,
                      "create_weight" : 950,
                      "shallow_advance" : 0,
                      "create_weight_count" : 1,
                      "build_scorer" : 171817
                    }
                  }
                ]
              }
            ],
            "rewrite_time" : 152447,
            "collector" : [
              {
                "name" : "SimpleFieldCollector",
                "reason" : "search_top_hits",
                "time_in_nanos" : 8431130256
              }
            ]
          }
        ],
        "aggregations" : [ ]
      },
35

@deng47 thanks a lot for the experiment, the results are very surprising, I would have expected the OS 1.0 to be at least on part with ES 7.13.x but that is not the case. The notable difference between 1.x and 2.x profiles is the presence of the LongDistanceFeatureQuery - I don’t understand where it is coming from.

Anyway, with the mappings I have at hand, I will try to reproduce the same query profiles, and than we could look for further steps.

Thank you.

36

Hi @reta Do you have any updates? Thank you

37

Hi @deng47 , sorry no updates from me at this moment, I haven’t had time to try to reproduce the issue yet.

38

Hey @deng47 ,

I found a time today to work on the issue but on the dataset I have (generated http_logs), the problem is not reproducible, so it might be related to your specific environment. Do you have a opportunity to spin off ES and OS clusters side by side so we could compare the stats from both and pinpoint the possible regression (or misconfiguration)?

Thank you

39

Hi @reta Could you tell me how did you deploy opensearch? Do you use helm chart or operator? What version do you use? Thank you

40

Sure, I use 2.5.0, bare metal deployment on local machine (single node). I have been using the http_logs dataset from opensearch-benchmark project [1], altering the mappings a bit to much yours. The environment is different from yours but I believe the slow query / aggregation should be reproducible unless this is indeed the deployment issue. Thank you.

[1] GitHub - opensearch-project/opensearch-benchmark: OpenSearch Benchmark - a community driven, open source project to run performance tests for OpenSearch