Fetching the hits value from opensearch through logstash

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
2.5

Describe the issue:
I want to fetch the hits value of opensearch through logstash. Through curl command i am getting the results but using the same config in logstash does not giving the hits value/

Configuration:

curl -XGET -uusername:password "https://localhost/test*/_search" -H 'Content-Type: application/json' -d' {  "size": 0,   "query": {    "bool": {      "must": [],      "filter": [
         {          "match_phrase": {            "type": "msg"          }
        },

        {
          "range": {
            "@timestamp": {
              "gte": "now-1d/d","lt": "now/d"
            }
          }
        }
      ],
      "should": [],
      "must_not": []
    }
  },
  "track_total_hits": true
}'

For the above command

{"took":6,"timed_out":false,"_shards":{"total":30,"successful":30,"skipped":0,"failed":0},"hits":{"total":{"value":55828,"relation":"eq"},"max_score":null,"hits":[]}}

Logstash config

input {
    opensearch {
    hosts => ["https://localhost"]
    index => "test*"
    query => '{  "size": 0,
   "query": {
    "bool": {
      "must": [],
      "filter": [
         {
          "match_phrase": {
            "type": "msg"
          }
        },

        {
          "range": {
            "@timestamp": {
              "gte": "now-1d/d","lt": "now/d"
            }
          }
        }
      ],
      "should": [],
      "must_not": []
    }
  },
  "track_total_hits": true
}'
    size => 10000
    docinfo => true
    schedule => "0 1 * * *"
    user => "username"
    password => "password"
   }
}

output{
    file { path => "/usr/share/logstash/test/sample.log" }
}

But is I used the same config is logstash it is not giving the hits value.
One more thing noticed the size parameter in logstash config is not working as expected. Ideally when the size is 0 it should only show the hits result.

Could you please help us in debugging the issue?

2

What’s the result given by Logstash? You may add this config output { stdout { codec => rubydebug } } to debug Logstash.

3

@prashant I did some testing on my side and couldn’t get the hits either.

I’ve used the rubydebug codec as suggested by @gaobinlong but it only reformated the output in the output file.

{
             "timestamp" => "2024-03-27T18:10:34.957032672Z",
    "metadata_with_hash" => {
        "_index" => "test1",
         "_type" => nil,
           "_id" => "Z64bgY4BmdRLbIfXUWJN"
    },
              "@version" => "1",
                 "first" => "second",
            "@timestamp" => 2024-03-27T20:17:00.299Z
}

The hits statistics are outside of the returned documents. You can get metadata of each document but the hits section is in a different area of the query output. I suspect that the query option of the input plugin doesn’t return all the information you could get with a regular API and it returns only documents.

4

Hi @gaobinlong @pablo
Thanks for the information and efforts.
I am able to get the hits count by using http_poller plugin.

Thanks,
prashant

1 Like