Opensearch sink with index template makes dataprepper fails dataprepper to start

piotrfrq · February 3, 2025, 1:45pm

I have the latest Opensearch/Dataprepper (2.18).

This pipeline works fine without this directives
template_type: index-template
template_content: >
…

sink:
    - opensearch:
        hosts: [ https://...... ]
        insecure: true
        username: "...."
        password: "....."
        index: syslog-standard
        dlq_file: "/usr/share/data-prepper/pipelines/dlq_syslog.err"
        template_type: index-template
        template_content: >
          {
            "template" : {
                  "mappings" : {
                          "properties": {
                                  "logsource": { "type" : "text" },
                                  "program": { "type" : "text" },
                                  "pid": { "type" : "text" },
                                  "timestampOriginalString": { "type" : "text" },
                                  "message": { "type" : "wildcard" }
                          }
                  }
            }
          }

But with them included I got errors in dataprepper logs:

2025-02-03T13:20:28,398 [syslog-pipeline-sink-worker-4-thread-1] WARN  org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Failed to initialize OpenSearch sink, retrying: Forbidden access
2025-02-03T13:20:28,602 [syslog-pipeline-sink-worker-4-thread-1] INFO  org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [syslog-pipeline] - sink is not ready for execution, retrying
2025-02-03T13:20:28,603 [syslog-pipeline-sink-worker-4-thread-1] INFO  org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Initializing OpenSearch sink
2025-02-03T13:20:28,603 [syslog-pipeline-sink-worker-4-thread-1] INFO  org.opensearch.dataprepper.plugins.sink.opensearch.ConnectionConfiguration - Using the username provided in the config.
2025-02-03T13:20:28,603 [syslog-pipeline-sink-worker-4-thread-1] INFO  org.opensearch.dataprepper.plugins.sink.opensearch.ConnectionConfiguration - Using the trust all strategy

Why?

Update: because of user permissions… The user I used (not admin) had enough rights to create index, but not for template creation. When I used admin user, it started to work.
P.

kris · February 26, 2025, 6:38pm

@dlv - do you have any idea why this would require admin permissions? Is this a setting that can be adjusted? thanks

nateynate · February 26, 2025, 8:00pm

It makes sense. Out of the box (see screenie below) it looks like only admin has access to create an index template (/index_template/put). You may want to create a role that has the permission to do that and then assign your user to that role.

dlv · February 27, 2025, 2:12pm

@piotrfrq,

Yes, you will need permissions to put an index template as @nateynate suggested.

Alternatively, you can choose not to create the index template in Data Prepper.

We do not currently have a specific role with scoped down Data Prepper permissions. We have an open issue to support it, but haven’t made any specific work towards it.

github.com/opensearch-project/security

[FEATURE] Provide built-in Data Prepper roles

opened 03:58PM - 01 Dec 23 UTC

dlvenable

enhancement triaged

**Is your feature request related to a problem?** Users of OpenSearch and Dat…a Prepper often have difficulty setting up the roles they use. Determining the permissions they need can be challenging. **What solution would you like?** To help users with these permissions, I'd like to provide useful predefined configurations in OpenSearch. In particular, I'm thinking of having existing OpenSearch roles tailored for Data Prepper. Users can choose to map the role or roles that fit their scenario to the user that they configure Data Prepper to use. Here are the roles I'm thinking of creating: * `data_prepper_write` - Can write to any index along with other roles for writing * `data_prepper_observability` - Similar to `data_prepper_write`, but can be scoped down to predefined indexes such as that for trace analytics and [Simple Schema for Observability](https://github.com/opensearch-project/opensearch-catalog/tree/main/docs/schema/observability). It would also need to be able to read from these indexes. * `data_prepper_read` - A role that can read from any index to support the `opensearch` source. **What alternatives have you considered?** One alternative is a single role with all permissions. But, this seems too broad. We could encourage users to combine the roles they need for their specific use-case. Say, for example, have a user which maps to `data_prepper_write` and `data_prepper_read`. **Do you have any additional context?** We could create a PR for this, but I want to get feedback first.

system · April 28, 2025, 2:12pm

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Failed Initialization of Opensearch sink with data prepper OpenSearch	1	742	December 8, 2023
I am getting error "Failed to initialize OpenSearch sink, retrying: opensearch-node2: System error" Data Prepper troubleshoot , configure	2	347	September 18, 2024
Data prepper - Workers did not terminate in PT30S, forcing termination of sink workers Data Prepper troubleshoot	1	27	January 27, 2025
How to use pipeline field vars to construct opensearch sink index param Data Prepper	4	808	July 4, 2024
Stuck on "Submitting request to initiate the pipeline processing" message Data Prepper	6	742	September 25, 2023

Opensearch sink with index template makes dataprepper fails dataprepper to start

Related topics