Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Working with OS 2.16.0
Describe the issue:
Hi, I have problems while working with the plugin of Logstash to query Opensearch indices in order to enrich documents.
I have an index, products, with these fields: id, name, type. Another one, types, with fields: id, number, type. They have in common the field “type”, so I am going to use it to work with them. These two indices are working on my Opensearch cluster.
My goal is to add another field to the “types” index containing the total amount of products of that type. To obtain that number, I query the products index using the Logstash Opensearch filter plugin, as you can see:
Logstash conf:
filter {
if [type] == “types” {
opensearch {
hosts => [“…/localhost:9200”] # here i specify localhost 9200, I’m not able to copy in this forum the link
index => “products”
query_template => “C:.…\query.json”
fields => { “[hits][total][value]” => “count_products” }
}
}
}
output {
if [type] == “types” {
opensearch {
hosts => [“”…/localhost:9200""]
index => “types”
document_id => “%{type}”
action => “update”
doc_as_upsert => true
}
}
}
The query stored in query.json is as simple as:
{
“size”: 0,
“track_total_hits”: true,
“query”: {
“term”: {
“type.keyword”: “%{[type]}”
}
}
}
I am using this query in Opensearch dashboards console and it works as expected. The result is somethig like this, ‘value’ is the wanted count:
{
“took”: 1,
“timed_out”: false,
“_shards”: {
“total”: 2,
“successful”: 2,
“skipped”: 0,
“failed”: 0
},
“hits”: {
“total”: {
“value”: 5812,
“relation”: “eq”
},
“max_score”: null,
“hits”:
}
}
My problem is that it does not work. Neither logstash or opensearch raise any errors, but my new field is not indexed. I have also tried to reindex and still does not work. I have also tried to specify the query directly in Logstash, here:
opensearch {
hosts => [“”…/localhost:9200""]
index => “products”
query => ‘{“size”: 0, “query”: {“term”: {“type.keyword”: “%{[type]}”}}}’
fields => { “[hits][total][value]” => “count_products” }
}
And here raises a ruby exception, smth like this:
Encountered \\ <RANGE_GOOP> \\ {\\\\ term\\\\ : \\ \\ at line 1, column 10.\r\nWas expecting:\r\n \\ TO\\ …\r\n \ ,\ caused_by\ :{\ type\ :\ parse_exception\ ,\ reason\ :\ Encountered \\ <RANGE_GOOP>.
So, the query is not correctly parsed, although I can see in the logs that the TYPE IS CORRECTLY TAKEN FROM THE OTHER INDEX, so I am able to get the content of that field. The other way, with the template, it apparently parses correctly.
Anyone has some idea to fix this? Thanks
Configuration:
Relevant Logs or Screenshots: