Data Prepper can not write to Opensearch Datastream [BUG] #2037

Data Prepper
1

Hello.
There is a bug report on github? And we get the same issue. Can not send logs to Opensearch Data stream:

Error “WARN com.amazon.dataprepper.plugins.sink.opensearch.OpenSearchSink - Document [org.opensearch.client.opensearch.core.bulk.BulkOperation@c4aecd] has failure: java.lang.RuntimeException: only write ops with an op_type of create are allowed in data streams”

Configuration:
data-prepper:latests, opensearch v 2.10.0,

kafka-sysmon_security_eventlog-pipeline:
  source:
    kafka:
      acknowledgments: true
      encryption:
          type: none
      bootstrap_servers:
        - xxxxxxxx:9094
        - xxxxxxxx:9095
        - xxxxxxxx:9096
      topics:
        - name: "sysmon_security_eventlog"
          group_id: "data_prepper"
          key_mode: "discard"
          serde_format: "json"
          auto_commit: true
  processor:
    - aggregate:
          identification_keys: ["event.provider","event.code","event.outcome","host.name","winlog.event_data.AuthenticationPackageName","winlog.event_data.TargetDomainName","winlog.event_data.TargetUserName","winlog.event_data.TargetUserSid","winlog.event_data.WorkstationName"]
          action:
            remove_duplicates:
          group_duration: 30s
  sink:
    - opensearch:
          hosts: ["https://xxxxxxxx:9200"]
          username: xxxxxxxx
          password: xxxxxxxx
          insecure: true
          connect_timeout: 60000
          index: logs-events-sysmon_security_eventlog
          index_type: management_disabled

Is any progress there ?
The bug has been open for almost a year and so far nothing has moved, whereas this feature would be very necessary.
Or is there some workaround?

2

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.