Error in Transform job for field missing

OpenSearch
,
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

Describe the issue : I am working on transform job where there is one field measurement based on the measurement value different fields will come . I am able to create transform job but due to that field is not present in every document it is showing following error and following
cardinality, extended_stats, geo_bounds, top_hits stats metric aggregations are not supported in transform job
i also tried putting missing:0 for avg
Configuration:

Relevant Logs or Screenshots:

Indexing failed for job metrics-hourly-memory-disk on index .opendistro-ism-config

image
image1264×754 64.2 KB

2

one more query does transform job currently support
“data_selection_query”: {
“match_all”: {}
},
only the match_all

3

@gourabh09 I’m not sure I fully understand the issue you are facing.

Only sum , avg , max , min , value_count , percentiles , and scripted_metric are supported aggregations in transformations. See docs for further details.

Are you able to use something along the lines of the following transform job?

PUT _plugins/_transform/transform-memory-hourly
{
  "transform": {
    "enabled": false,
    "schedule": {
      "interval": {
        "period": 1,
        "unit": "Hours",
        "start_time": 1708077600000
      }
    },
    "description": "Hourly memory metrics aggregation",
    "source_index": "metrics-raw",
    "target_index": "metrics-hourly-memory",
    "data_selection_query": {
      "bool": {
        "must": [
          {
            "term": {
              "measurement": "memory"
            }
          },
          {
            "exists": {
              "field": "memory_used"
            }
          }
        ]
      }
    },
    "page_size": 1000,
    "groups": [
      {
        "date_histogram": {
          "source_field": "@timestamp",
          "fixed_interval": "1h",
          "timezone": "UTC",
          "target_field": "@timestamp"
        }
      },
      {
        "terms": {
          "source_field": "host",
          "target_field": "host"
        }
      },
      {
        "terms": {
          "source_field": "region",
          "target_field": "region"
        }
      }
    ],
    "aggregations": {
      "memory_avg": {
        "avg": {
          "field": "memory_used"
        }
      },
      "memory_max": {
        "max": {
          "field": "memory_used"
        }
      },
      "memory_min": {
        "min": {
          "field": "memory_used"
        }
      },
      "memory_sum": {
        "sum": {
          "field": "memory_used"
        }
      },
      "memory_count": {
        "value_count": {
          "field": "memory_used"
        }
      }
    }
  }
}

You can then query the new index:

GET /metrics-hourly-memory/_search
{
  "size": 10,
  "sort": [
    {
      "@timestamp": {
        "order": "desc"
      }
    }
  ]
}

With example response:

{
  "took": 3,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 2,
      "relation": "eq"
    },
    "max_score": null,
    "hits": [
      {
        "_index": "metrics-hourly-memory",
        "_id": "yeL3p-33dndmQPbsitkWhg",
        "_score": null,
        "_source": {
          "transform._id": "transform-memory-hourly",
          "_doc_count": 3,
          "transform._doc_count": 3,
          "@timestamp": 1771236000000,
          "host": "server-01",
          "region": "us-east-1",
          "memory_max": 10737418240,
          "memory_avg": 9663676416,
          "memory_min": 8589934592,
          "memory_sum": 28991029248,
          "memory_count": 3
        },
        "sort": [
          1771236000000
        ]
      },
      {
        "_index": "metrics-hourly-memory",
        "_id": "HYGcHMZIJwiz13ZFxkQpRA",
        "_score": null,
        "_source": {
          "transform._id": "transform-memory-hourly",
          "_doc_count": 2,
          "transform._doc_count": 2,
          "@timestamp": 1771236000000,
          "host": "server-02",
          "region": "us-west-2",
          "memory_max": 7516192768,
          "memory_avg": 6979321856,
          "memory_min": 6442450944,
          "memory_sum": 13958643712,
          "memory_count": 2
        },
        "sort": [
          1771236000000
        ]
      }
    ]
  }
}