Unable to run my transform job after updating to OpenSearch 2.11

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Currently on v2.11.
Previously on v2.7.

Describe the issue:
I currently have a transform job that I run periodically that used to run just fine on v2.7. Since upgrading to v2.11, the transform now fails with an error I have never seen before. I’ve had no issues running the transform aggregation without a proper field and only a script prior. If I add a bogus field to the aggregation, it runs just fine, but obviously without aggregating on the return value of the script. The aggregation still runs just fine outside of a transform in a normal search query.

Configuration:

POST _plugins/_transform/_preview
{
  "transform": {
    "enabled": true,
    "continuous": false,
    "schedule": {
      "interval": {
        "period": 1,
        "unit": "Minutes",
        "start_time": 1706576400000
      }
    },
    "description": "business metrics by S code.",
    "source_index": "crdb_2023_3",
    "target_index": "crdb_s_code_2023_9",
    "data_selection_query": {
      "match": {
        "business": "12345678"
      }
    },
    "page_size": 10,
    "groups": [
      {
        "terms": {
          "source_field": "ins_code.keyword",
          "target_field": "s_code"
        }
      }
    ],
    "aggregations": {
      "lien_avg": {
        "avg": {
            "script": {
              "lang": "painless",
              "source": """
              String val = doc[params.field].value;
              if (val != null && val.length() > 0)
              {
                double d_val = Double.parseDouble(val);
                if (d_val != 999 && d_val != 998)
                {
                  d_val
                }
                else
                {
                  null
                }
              }
              else
              {
                null;
              }
              """,
              "params": {
                "field": "lien.keyword"
              }
            }
          }
      }
    }
  }
}

Relevant Logs or Screenshots:

{
  "error": {
    "root_cause": [
      {
        "type": "status_exception",
        "reason": "Failed to parse the transformed results"
      }
    ],
    "type": "status_exception",
    "reason": "Failed to parse the transformed results",
    "caused_by": {
      "type": "null_pointer_exception",
      "reason": "sourceIdxFieldName must not be null"
    }
  },
  "status": 500
}

Could you open an issue here:Issues · opensearch-project/index-management · GitHub?

1 Like

Thanks for responding. Done.
Issue #1107

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.