How to make post processing using Data Prepper

Data Prepper
1

Versions (relevant - OpenSearch):

Describe the issue:

After reading Data Prepper documentation, i saw all processor options - these options are providing pre processing on the data before indexing - if i got it right.
how can i make some post processing after all the index creates, for example if i want to change the alias of the index.
there is such option? using the pipeline configuration?

2

Hey @Offir

Is this what your referring to?

Index Management --> Templates

image
image1212×403 19.5 KB

Or

Index Management --> Indices

image
image1188×656 31.6 KB

3

Im not sure, what is this UI?
what i meant is if there is an option to add some line in the pipeline configuration in order to
change index alias. i define in the sink the index name - so to add an alias and remove the previous one if exist. Not using UI.

btw, there is post processing that can be apply on the index after it has been created using pipeline configuration? (similar to “processor” - but post processing ones).

4

Hey @Offir

Not that I know of. What I did was create a index template first, this would have my alias/shard/replica’s, etc… set. Then for example i set my sink like this.

sink:
    - opensearch:
        hosts: [ "https://opensearch.com:9200" ]        
        username: "admin"
        password: "changeit"        
        cert:  /opt/opensearch-data-prepper/config/root-ca.pem        
        index: apache_logs-%{YYYY.MM.dd}

That was it, as for adding something in the pipe, not sure.

5

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.