Error setting up filebeat pipeline

OpenSearch
1 
sudo /usr/bin/filebeat setup --pipelines --modules nginx,suricata,system
Exiting: 1 error: error loading pipeline for fileset system/auth: couldn't load pipeline: couldn't load json. Error: 400 Bad Request: {"error":{"root_cause":[{"type":"parse_exception","reason":"[value] required property is missing","processor_type":"set","property_name":"value"}],"type":"parse_exception","reason":"[value] required property is missing","processor_type":"set","property_name":"value"},"status":400}. Response body: {"error":{"root_cause":[{"type":"parse_exception","reason":"[value] required property is missing","processor_type":"set","property_name":"value"}],"type":"parse_exception","reason":"[value] required property is missing","processor_type":"set","property_name":"value"},"status":400}

Using filebeat 7.17.6 and opensearch 2.3.0. I have compatibility.override_main_response_version: true

2

Looks like from 7.17.5 to 7.17.6 we changed from:

processors:
- set:
    field: event.ingested
    value: '{{_ingest.timestamp}}'

to

processors:
  - set:
      field: event.ingested
      copy_from: _ingest.timestamp
3

Reverting the 7.17.6 ingest pipeline to the value form allowed me to set it up. So, looks like opensearch needs to accept the “copy_from” field?

4

Officially only filebeat 7.12.1 is the latest supported filebeat version unless you output from filebeat to logstash.

That might be part of the problem

5

We do ship via logstash, but we need to setup the ingest pipeline for that to work well. Looks like an incompatibility has crept in with the pipeline definition. Hopefully that can be resolved. Should I file a bug report for that?

6

Turns out i have the same exact problem on my filebeat setup…

i think it’s worth creating a bug report…

7

turns out that “copy_from” was added in elasticsearch 7.11 (i.e. the 1st releast after opensearch fork)

i am adding a bug/feature request asking to add support for this

8

here: [BUG] processors:set.copy_from: isn't understood breaking pipelines · Issue #4672 · opensearch-project/OpenSearch · GitHub