# Loss of nested fields when using pipeline

**URL:** <https://forum.opensearch.org/t/loss-of-nested-fields-when-using-pipeline/23822>\
**Category:** OpenSearch\
**Tags:** troubleshoot\
**Created:** [March 18, 2025, 12:34pm UTC](https://forum.opensearch.org/t/loss-of-nested-fields-when-using-pipeline/23822 "2025-03-18T12:34:14Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![andreibalakirev](https://avatars.discourse-cdn.com/v4/letter/a/ed8c4c/32.png) [@andreibalakirev](https://forum.opensearch.org/u/andreibalakirev)\
**Post date:** [March 18, 2025, 12:34pm UTC](https://forum.opensearch.org/t/loss-of-nested-fields-when-using-pipeline/23822/1 "2025-03-18T12:34:14Z")

</div>

Hello. In opensearch version 2.19, we have a problem. Loss of nested fields when using pipeline. When we try to create an index with nested fields, only one element of the nested object gets into index.

In the example below, only the “descr” fields are included in the index. The name and id fields are missing. There is no such problem in version 2.18.

Example simulate query:

```auto
POST /_ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "text_embedding": {
          "model_id": "BeQ3O5UBwVQ2gq6gyMSO",
          "field_map": {
            "ml_text": "ml_embedding"
          }
        }
      }
    ]
  },
  "docs": [
    {
      "_index": "test_index", 
      "_source": {
        "nested_field": [
          { 
            "id": "value1", 
            "name": "value1", 
            "descr": "value1" 
            
          },
          { 
            "id": "value2", 
            "name": "value2",
            "descr": "value2" 
          }
        ],
        "categories": [
          { 
            "id": "value3", 
            "name": "value3", 
            "descr": "value3" 
            
          },
          { 
            "id": "value4", 
            "name": "value4",
            "descr": "value4" 
          }
        ]
      }
    }
  ]
}

```

Result:

```auto
{
  "docs": [
    {
      "doc": {
        "_index": "test_index",
        "_id": "_id",
        "_source": {
          "nested_field": [
            {
              "descr": "value1"
            },
            {
              "descr": "value2"
            }
          ],
          "categories": [
            {
              "descr": "value3"
            },
            {
              "descr": "value4"
            }
          ]
        },
        "_ingest": {
          "timestamp": "2025-03-18T12:15:00.656183009Z"
        }
      }
    }
  ]
}

```

---

<div class="post-metadata">

**Author:** ![ylwu](https://avatars.discourse-cdn.com/v4/letter/y/a9a28c/32.png) [@ylwu](https://forum.opensearch.org/u/ylwu)\
**Post date:** [May 1, 2025, 4:22pm UTC](https://forum.opensearch.org/t/loss-of-nested-fields-when-using-pipeline/23822/2 "2025-05-01T16:22:26Z")

</div>

This bug fixed in 2.19.2 check [[BUG] text\_embedding truncates key-value maps in a pipeline · Issue #3804 · opensearch-project/ml-commons · GitHub](https://github.com/opensearch-project/ml-commons/issues/3804#issuecomment-2843203258)

---

<div class="post-metadata">

**Author:** ![KateWinslet](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/katewinslet/32/6694_2.png) [@KateWinslet](https://forum.opensearch.org/u/KateWinslet)\
**Post date:** [May 7, 2025, 9:41am UTC](https://forum.opensearch.org/t/loss-of-nested-fields-when-using-pipeline/23822/3 "2025-05-07T09:41:22Z")

</div>

> [@andreibalakirev](#):
>
> Hello. In opensearch version 2.19, we have a problem. Loss of nested fields when using pipeline. When we try to create an index with nested fields, only one element of the nested object gets into index.  
> [Suneel Maghani](https://suneelmaghani.com)  
> In the example below, only the “descr” fields are included in the index. The name and id fields are missing. There is no such problem in version 2.18.
> 
> Example simulate query:
> 
> ```auto
> POST /_ingest/pipeline/_simulate
> {
> "pipeline": {
> "processors": [
> {
> "text_embedding": {
> "model_id": "BeQ3O5UBwVQ2gq6gyMSO",
> "field_map": {
> "ml_text": "ml_embedding"
> }
> }
> }
> ]
> },
> "docs": [
> {
> "_index": "test_index", 
> "_source": {
> "nested_field": [
> { 
> "id": "value1", 
> "name": "value1", 
> "descr": "value1" 
>             
> },
> { 
> "id": "value2", 
> "name": "value2",
> "descr": "value2" 
> }
> ],
> "categories": [
> { 
> "id": "value3", 
> "name": "value3", 
> "descr": "value3" 
>             
> },
> { 
> "id": "value4", 
> "name": "value4",
> "descr": "value4" 
> }
> ]
> }
> }
> ]
> }
> 
> ```
> 
> Result:
> 
> ```auto
> {
> "docs": [
> {
> "doc": {
> "_index": "test_index",
> "_id": "_id",
> "_source": {
> "nested_field": [
> {
> "descr": "value1"
> },
> {
> "descr": "value2"
> }
> ],
> "categories": [
> {
> "descr": "value3"
> },
> {
> "descr": "value4"
> }
> ]
> },
> "_ingest": {
> "timestamp": "2025-03-18T12:15:00.656183009Z"
> }
> }
> }
> 
> ```

In OpenSearch 2.19, the text\_embedding processor may strip non-mapped fields from nested objects, unlike in 2.18. To fix this, define nested mappings and ensure all nested fields needed are explicitly included in field\_map or handled using appropriate nested-aware processors.