Hi,
I’ve came across a weird behaviour when I try to rollup an index.
Here is how to reproduce the error:
Create an index test-index with the following mapping
"test-index" : {
"aliases" : { },
"mappings" : {
"dynamic_date_formats" : [
"yyyy-MM-dd HH:mm:ss||yyyy-MM-dd'T'HH:mm:ss.SSS||yyyy-MM-dd||yyyy-MM-dd'T'HH||yyyy-MM-dd'T'HH:mm"
],
"properties" : {
"Action" : {
"type" : "keyword"
}
"MaxValue" : {
"type" : "float"
},
"LengthValue" : {
"type" : "long"
},
"RequestValue" : {
"type" : "long"
},
"Type" : {
"type" : "keyword"
},
"Timestamp" : {
"type" : "date",
"format" : "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd'T'HH:mm:ss.SSS||yyyy-MM-dd||yyyy-MM-dd'T'HH||yyyy-MM-dd'T'HH:mm"
}
}
}
}
Insert the following documents into the index
curl POST https://addr:port/test-index/_doc/1?pretty
{
"Type" : "testType1",
"Timestamp" : "2022-05-31T17",
"Action" : "Action1",
"LengthValue" : 1,
"RequestValue" : 2
}
curl POST https://addr:port/test-index/_doc/2?pretty
{
"Type" : "testType2",
"Timestamp" : "2022-05-31T18",
"Action" : "Action2",
"MaxValue" : 1
}
curl POST https://addr:port/test-index/_doc/3?pretty
{
"Type" : "testType3",
"Timestamp" : "2022-05-31T19",
"Action" : "Action3"
}
Create a rollup job
curl -X PUT https://addr:port/_plugins/_rollup/jobs/rollup-test?pretty
{
"rollup": {
"source_index": "test-index",
"target_index": "rollup-test-index",
"schedule": {
"interval": {
"period": 1,
"unit": "Minutes"
}
},
"description": "Rollup job",
"enabled": true,
"page_size": 200,
"delay": 0,
"continuous": true,
"dimensions": [
{
"date_histogram": {
"source_field": "Timestamp",
"fixed_interval": "1d"
}
},
{"terms": {"source_field": "Type"}},
{"terms": {"source_field": "Action"}}
],
"metrics": [
{
"source_field": "MaxValue",
"metrics": [
{"max":{}}
]
},
{
"source_field": "LengthValue",
"metrics": [
{"max":{}},{"avg":{}}
]
},
{
"source_field": "RequestValue",
"metrics": [
{"max":{}},{"avg":{}}
]
}
]
}
}
Start the rollup job and I get the error failed to parse field [LengthValue.max] …Preview of field’s value: ‘-Infinity’", error
Even weirder, if I try to set the rollup job inside a ISM policy I do not get the same error, the rollup job inside the ISM policy manages to finish correctly, but the mapping for LengthValue is of type keyword instead of max, and if I try to query the rollup index I get OpenSearchException[OpenSearch exception [type=illegal_argument_exception, reason=Field [LengthValue.max] of type [keyword] is not supported for aggregation [max]]]
So my questions are:
Where is that -Infinity value came from?
Does all the documents must have the same format to use the max metric inside a rollup job?
Why the rollup job fails while the ISM policy manage to finish?