[transform job] there are no field in source index but theare are in that source index

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
opensearch 2.7.0

Describe the issue:
API

PUT _plugins/_transform/a-trandsform-job
{
  "transform": {
      "enabled": true,
      "continuous": true,
      "schedule": {
          "interval": {
              "start_time": 1692457200,
              "period": 1,
              "unit": "Minutes"
          }
      },
      "description": "a transform job",
      "source_index": "atenantid-a-log-*",
      "target_index": "a-trandsform-job",
      "data_selection_query": {
        "match_all": {}
      },
      "page_size": 1000,
      "groups": [
        {
            "date_histogram": {
                "calendar_interval": "1d",
                "source_field": "@timestamp",
                "target_field": "@timestamp _date_histogram_1_d_calendar",
                "timezone": "UTC"
            }
        },
        {
            "terms": {
                "source_field": "event.level",
                "target_field": "event.level_terms"
            }
        }
      ],
      "aggregations": {}
  }
}

result is error like this

{
  "error": {
    "root_cause": [
      {
        "type": "status_exception",
        "reason": "Cannot find field [event.level] that can be grouped as [terms] in [atenantid-a-log-network-2023.08.16-000003]"}
    ],
    "type": "status_exception",
    "reason": "Cannot find field [event.level] that can be grouped as [terms] in [atenantid-a-log-network-2023.08.16-000003]"},
  "status": 400
}

BUT there is [event.level] field in that index!!!
There’s not even one document that doesn’t have that field in that index…
How to solve this problems?

2

Can you show the mappings of field event.level? is it text type? If so please use event.level.keyword or other fields which are not text type instead. That’s because terms aggregation cannot perform on text field.

1 Like
3

To @gaobinlong
no, event.level’s type definitely keyword

image
image759Ă—250 27.3 KB

4

oh sorry, I got mapping table of index that in error message, there is text type of field event.level…!! thank you!! now I should go to reindex now…