Ingestion pipeline for a nested field

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

2.5

Describe the issue:
I have the following mapping,

{
    "settings": {
        "index.knn": True,
        "default_pipeline": VECTORIZATION_PIPELINE_NAME,
    },
    "mappings": {
        "properties": {
            "title_embedding": {
                "type": "knn_vector",
                "dimension": EMBEDDING_DIMENSION,
                "method": {
                    "name": "hnsw",
                    "space_type": "l2",
                    "engine": "faiss",
                    "parameters": {
                        "encoder": {
                            "name": "flat",
                        }
                    },
                },
            },
            "short_description": {
                "type": "nested",
                "properties": {
                    "short_description_sentence": {
                        "type": "text",
                        "term_vector": "yes",
                    },
                    "short_description_sentence_embedding": {
                        "type": "knn_vector",
                        "dimension": EMBEDDING_DIMENSION,
                        "method": {
                            "name": "hnsw",
                            "space_type": "l2",
                            "engine": "faiss",
                            "parameters": {
                                "encoder": {
                                    "name": "flat",
                                }
                            },
                        },
                    },
                },
            },
            "id": {"type": "keyword"},
            "item_group_id": {"type": "keyword"},
            "category": {"type": "text"},
            "title": {"type": "text", "term_vector": "yes"},
            "url": {"type": "keyword"},
        }
    },
}

I want to embed “short_description_sentence”. I tried to using the following ingestion pipeline.

{
        "description": "A pipeline for text embedding",
        "processors": [
            {
                "text_embedding": {
                    "model_id": "7_4plIcBj4yLiPtvG5oN",
                    "field_map": {
                        "title": "title_embedding",
                        "short_description_sentence": "short_description_sentence_embedding",
                    },
                }
            },
        ],
    }

However, I only get embedding for “title” but not foreach sentence in the nested field “short_description”. How can I do that?

2

short_description_sentence is a subfield under short_description , so I think you should use "short_description.short_description_sentence": "short_description. short_description_sentence_embedding" instead.

3

@gaobinlong Thanks for taking the time to help. Unfortunately it didn’t work. I tried it with a couple of other different things, but none worked.
Below are some of what I tried.

{
    "foreach": {
        "field": "short_description",
        "processor": {
            "text_embedding": {
                "model_id": "7_4plIcBj4yLiPtvG5oN",
                "field_map": {
                    "_ingest._value.short_description_sentence": "_ingest._value.short_description_sentence_embedding",
                },
            }
        },
    }
}