Why is my ingest pipeline processor not invoked?

OpenSearch
1

I’m using OpenSearch 2.15 and I have a simple drop processor.
It seems to be correct, because the _simulate api works fine on it.
But why are none of my incoming events (sent from a filebeat instance somewhere)
not incrementing any counters?

PUT _ingest/pipeline/drop-events-too-old
{
  "description": "Drop any incoming event that is older than a specific date",
  "processors": [
    {
      "drop": {
        "if": "ctx['@timestamp'].compareTo('2024-11-14') < 0"
      }
    }
  ]
}

It works fine using simulate:

POST _ingest/pipeline/drop-events-too-old/_simulate
{
  "docs": [
    {
      "_index": "not-to-be-dropped",
      "_id": "1",
      "_source": {
        "@timestamp": "2024-11-21T11:17:45.000Z"
      }
    },
    {
      "_index": "dropped",
      "_id": "2",
      "_source": {
        "@timestamp": "2024-11-01T11:17:45.000Z"
      }
    }
  ]
}

GET /_nodes/stats/ingest?filter_path=nodes.*.ingest shows (I have 2 nodes)

{
  "nodes": {
    "Hq2-7GwTTn2IRKaTK-4hYw": {
      "ingest": {
        "total": {
          "count": 0,
          "time_in_millis": 0,
          "current": 0,
          "failed": 0
        },
        "pipelines": {
          "drop-events-too-old": {
            "count": 0,
            "time_in_millis": 0,
            "current": 0,
            "failed": 0,
            "processors": [
              {
                "drop": {
                  "type": "conditional",
                  "stats": {
                    "count": 0,
                    "time_in_millis": 0,
                    "current": 0,
                    "failed": 0
                  }
                }
              }
            ]
          }
        }
      }
    },
    "bvzCC5NRRSuFjHhh5h6GWA": {
      "ingest": {
        "total": {
          "count": 0,
          "time_in_millis": 0,
          "current": 0,
          "failed": 0
        },
        "pipelines": {
          "drop-events-too-old": {
            "count": 2,
            "time_in_millis": 61,
            "current": 0,
            "failed": 0,
            "processors": [
              {
                "drop": {
                  "type": "conditional",
                  "stats": {
                    "count": 1,
                    "time_in_millis": 0,
                    "current": 0,
                    "failed": 0
                  }
                }
              }
            ]
          }
        }
      }
    }
  }
}
2

I have just found this setting that I need to use:

index.default_pipeline (String): The default ingest node pipeline for the index. If the default pipeline is set and the pipeline does not exist, then index requests fail. The pipeline name _none specifies that the index does not have an ingest pipeline.

The ingest pipeline documentation does not seem to refer to this setting. It never specifies under which conditions a pipeline is run.

One can also specify a pipeline to run when we post a document using the rest API.