Ingest Pipeline Nested Array Vector Embeddings (AWS Comprehend Medical)

Plugins Machine Learning
, ,
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
I’m running a local cluster on MacOS Sonoma 14.6.1 with the latest OpenSearch images from Docker.

Describe the issue:
I have several levels of nested JSON data (output from AWS Comprehend Medical models) that I am trying to send through the text_embedding processor. I am struggling to create a pipeline that will actually convert the nested fields into vector embeddings using foreach processors to iterate through arrays of dictionaries. I have attached a sample of the data below.

Essentially, I am trying to develop a pipeline that converts all of the text fields (at the top level, within each dictionary in the Entities array, and within each dictionary in all of the Attributes arrays) into vector embeddings.

Configuration:

{
  "id": 0,
  "text": {
    "input": "text that we send to the model goes here",
    "output": {
      "model_name": {
        "Entities": [
          {
            "Id": 5,
            "BeginOffset": 0,
            "EndOffset": 1,
            "Score": 0.9849938154220581,
            "Text": "sample text",
            "Category": "TEST_TREATMENT_PROCEDURE",
            "Type": "TEST_NAME",
            "Traits": []
          },
          {
            "Id": 3,
            "BeginOffset": 0,
            "EndOffset": 1,
            "Score": 0.8389013409614563,
            "Text": "sample text 2",
            "Category": "MEDICAL_CONDITION",
            "Type": "DX_NAME",
            "Traits": [
              { "Name": "SYMPTOM", "Score": 0.8507744073867798 },
              { "Name": "HYPOTHETICAL", "Score": 0.8027461767196655 }
            ],
            "Attributes": [
              {
                "Type": "QUALITY",
                "Score": 0.7246232032775879,
                "RelationshipScore": 1.0,
                "RelationshipType": "QUALITY",
                "Id": 4,
                "BeginOffset": 0,
                "EndOffset": 1,
                "Text": "sample text 3",
                "Category": "MEDICAL_CONDITION",
                "Traits": []
              }
            ]
          },
          {
            "Id": 1,
            "BeginOffset": 0,
            "EndOffset": 1,
            "Score": 0.9704693555831909,
            "Text": "sample text 4",
            "Category": "MEDICATION",
            "Type": "GENERIC_NAME",
            "Traits": []
          },
          {
            "Id": 2,
            "BeginOffset": 0,
            "EndOffset": 1,
            "Score": 0.9746520519256592,
            "Text": "sample text 5",
            "Category": "MEDICATION",
            "Type": "GENERIC_NAME",
            "Traits": []
          },
          {
            "Id": 6,
            "BeginOffset": 0,
            "EndOffset": 1,
            "Score": 0.787682294845581,
            "Text": "sample text 6",
            "Category": "TEST_TREATMENT_PROCEDURE",
            "Type": "TREATMENT_NAME",
            "Traits": []
          }
        ],
        "UnmappedAttributes": [],
        "ModelVersion": "2.4.0"
      }
    }
  }
}

Relevant Logs or Screenshots: