# Latest data-prepper not working

**URL:** <https://forum.opensearch.org/t/latest-data-prepper-not-working/11055>\
**Category:** Data Prepper\
**Created:** [September 26, 2022, 10:53pm UTC](https://forum.opensearch.org/t/latest-data-prepper-not-working/11055 "2022-09-26T22:53:44Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![kkondaka](https://avatars.discourse-cdn.com/v4/letter/k/71c47a/32.png) [@kkondaka](https://forum.opensearch.org/u/kkondaka)\
**Post date:** [September 26, 2022, 10:53pm UTC](https://forum.opensearch.org/t/latest-data-prepper-not-working/11055/1 "2022-09-26T22:53:44Z")

</div>

Hi,

I have been using data-prepper with older source code ( code from Jul 8, hash 59d43209a239b33a9089248e68e233472cce8ecc) and it works fine. But when I try to use latest data-prepper (just by doing `git clone https://github.com/opensearch-project/data-prepper.git`), I see many issues. Compilation goes fine but at runtime I see many issues.

Has anything (either command-line arguments or java env or configuration) changed since then? Some of the issues I am seeing are -

1. I am using file as source and stdout as sink, but the data prepper crashes while instantiating opensearch sink
2. If I comment out “initialize” in opensearch sink, it does not crash but still file source is not processed
3. Previously, doing “ctrl-C” kills the docker instance but it does not do that anymore
4. I tried with Java 11 and java 17 but in both cases, data-prepper prints the following. I do not see this when using older data-prepper

```auto
/opt/java/openjdk/bin/java
Found openjdk version of 17.0.4
(standard_in) 1: syntax error

```

Any idea on what I am doing wrong?

Thanks!

---

<div class="post-metadata">

**Author:** ![ddpowers](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/ddpowers/32/3675_2.png) [@ddpowers](https://forum.opensearch.org/u/ddpowers)\
**Post date:** [September 27, 2022, 4:32pm UTC](https://forum.opensearch.org/t/latest-data-prepper-not-working/11055/2 "2022-09-27T16:32:44Z")

</div>

Hi @kkondaka ,

Can you please post your pipeline.yaml file? There have been several changes lately and it could be that there was a breaking change. I’d like to confirm.

---

<div class="post-metadata">

**Author:** ![dlv](https://avatars.discourse-cdn.com/v4/letter/d/5f9b8f/32.png) [@dlv](https://forum.opensearch.org/u/dlv)\
**Post date:** [September 27, 2022, 5:01pm UTC](https://forum.opensearch.org/t/latest-data-prepper-not-working/11055/3 "2022-09-27T17:01:23Z")

</div>

Hello @kkondaka ,

First, we are preparing for Data Prepper 2.0 which does introduce some breaking changes. Those are reflected in `main` and thus some things will be different for you. As @ddpowers requested, please share your pipeline so that we can best help. I’ll also try to address a few points below.

> [@kkondaka](#):
>
> I am using file as source and stdout as sink, but the data prepper crashes while instantiating opensearch sink

There is a [known issue](https://github.com/opensearch-project/data-prepper/issues/936) where OpenSearch needs to be running. Feel free to share your logs so that we can help you resolve the issue.

> [@kkondaka](#):
>
> If I comment out “initialize” in opensearch sink, it does not crash but still file source is not processed

Yes, this is needed to run correctly. Please be sure that OpenSearch is running first.

> [@kkondaka](#):
>
> Previously, doing “ctrl-C” kills the docker instance but it does not do that anymore

Can you elaborate on this? How are you running the Docker image?

> [@kkondaka](#):
>
> I tried with Java 11 and java 17 but in both cases, data-prepper prints the following. I do not see this when using older data-prepper

The Docker image is based on an image with Java 17 installed. If you run from this image, you will get Java 17. If you want to run with a specific version of Java you should use the archives (`.tar.gz`). There is one with an embedded JDK 17 and one without a JDK.

---

<div class="post-metadata">

**Author:** ![dlv](https://avatars.discourse-cdn.com/v4/letter/d/5f9b8f/32.png) [@dlv](https://forum.opensearch.org/u/dlv)\
**Post date:** [September 27, 2022, 5:19pm UTC](https://forum.opensearch.org/t/latest-data-prepper-not-working/11055/4 "2022-09-27T17:19:00Z")

</div>

> [@kkondaka](#):
>
> Previously, doing “ctrl-C” kills the docker instance but it does not do that anymore

I was able to reproduce this and created [[BUG] Unable to shutdown Data Prepper Docker with Ctrl+C · Issue #1816 · opensearch-project/data-prepper · GitHub](https://github.com/opensearch-project/data-prepper/issues/1816) to track it.

---

<div class="post-metadata">

**Author:** ![kkondaka](https://avatars.discourse-cdn.com/v4/letter/k/71c47a/32.png) [@kkondaka](https://forum.opensearch.org/u/kkondaka)\
**Post date:** [September 27, 2022, 6:39pm UTC](https://forum.opensearch.org/t/latest-data-prepper-not-working/11055/5 "2022-09-27T18:39:35Z")

</div>

With opesearch initialize() commented out, data-prepper starts fine but it does not read from the file and does not print anything to stdout. Are you saying that even for file source to work, open search must be initialized? I tried with opensearch running and still seeing the same error.  
Run log (It looks like there is a limit on number of characters, so limiting to the first few lines of the log) -

```auto
dev-dsk-krishkdk-2b-9095a429 % docker run --rm --name data-prepper -p 4900:4900 --network=opensearch_opensearch_opensearch-net -v /tmp/input-file:/tmp/input-file -v 
~/pipelines/pipeline_file_stdout.yaml:/usr/share/data-prepper/pipelines.yaml -v ${PWD}/data-prepper-config.yaml:/usr/share/data-prepper/data-prepper-config.yaml opensea
rch-data-prepper:2.0.0-SNAPSHOT                                                                                                                                         
Reading pipelines and data-prepper configuration files from Data Prepper home directory.
/opt/java/openjdk/bin/java
Found openjdk version of 17.0.4
(standard_in) 1: syntax error
2022-09-27T18:18:59,451 [main] INFO org.opensearch.dataprepper.parser.config.DataPrepperAppConfiguration - Command line args: /usr/share/data-prepper/pipelines,/usr/sh
are/data-prepper/config/data-prepper-config.yaml
2022-09-27T18:18:59,453 [main] INFO org.opensearch.dataprepper.parser.config.DataPrepperArgs - Using /usr/share/data-prepper/pipelines configuration file
2022-09-27T18:18:59,856 [main] WARN org.opensearch.dataprepper.peerforwarder.server.PeerForwarderHttpServerProvider - Creating Peer Forwarder server without SSL/TLS. T
his is not secure.
2022-09-27T18:19:00,026 [main] INFO org.opensearch.dataprepper.parser.PipelineParser - Reading pipeline configuration from pipelines.yaml
2022-09-27T18:19:00,040 [main] INFO org.opensearch.dataprepper.parser.PipelineParser - Building pipeline [entry-pipeline] from provided configuration
2022-09-27T18:19:00,040 [main] INFO org.opensearch.dataprepper.parser.PipelineParser - Building [otel_trace_source] as source component for the pipeline [entry-pipelin
e]
2022-09-27T18:19:00,095 [main] WARN com.amazon.dataprepper.plugins.source.oteltrace.OTelTraceSource - Creating otel-trace-source without authentication. This is not se
cure.
2022-09-27T18:19:00,095 [main] WARN com.amazon.dataprepper.plugins.source.oteltrace.OTelTraceSource - In order to set up Http Basic authentication for the otel-trace-s
ource, go here: https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/otel-trace-source#authentication-configurations
2022-09-27T18:19:00,096 [main] INFO org.opensearch.dataprepper.parser.PipelineParser - Building buffer for the pipeline [entry-pipeline]
2022-09-27T18:19:00,097 [main] INFO org.opensearch.dataprepper.parser.PipelineParser - Building processors for the pipeline [entry-pipeline]
2022-09-27T18:19:00,097 [main] INFO org.opensearch.dataprepper.parser.PipelineParser - Building sinks for the pipeline [entry-pipeline]
2022-09-27T18:19:00,098 [main] INFO org.opensearch.dataprepper.parser.PipelineParser - Building [pipeline] as sink component
2022-09-27T18:19:00,098 [main] INFO org.opensearch.dataprepper.parser.PipelineParser - Building [pipeline] as sink component
2022-09-27T18:19:00,101 [main] INFO org.opensearch.dataprepper.parser.PipelineParser - Building pipeline [service-map-pipeline] from provided configuration
2022-09-27T18:19:00,101 [main] INFO org.opensearch.dataprepper.parser.PipelineParser - Building [pipeline] as source component for the pipeline [service-map-pipeline]
2022-09-27T18:19:00,101 [main] INFO org.opensearch.dataprepper.parser.PipelineParser - Building buffer for the pipeline [service-map-pipeline]
2022-09-27T18:19:00,102 [main] INFO org.opensearch.dataprepper.parser.PipelineParser - Building processors for the pipeline [service-map-pipeline]
2022-09-27T18:19:00,407 [main] INFO org.opensearch.dataprepper.parser.PipelineParser - Building sinks for the pipeline [service-map-pipeline]
2022-09-27T18:19:00,407 [main] INFO org.opensearch.dataprepper.parser.PipelineParser - Building [opensearch] as sink component
2022-09-27T18:19:00,422 [main] INFO org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Initializing OpenSearch sink
2022-09-27T18:19:00,428 [main] INFO org.opensearch.dataprepper.plugins.sink.opensearch.ConnectionConfiguration - aws_sigv4 is set, will sign requests using AWSRequestS
igningApacheInterceptor
2022-09-27T18:19:00,538 [main] INFO org.opensearch.dataprepper.plugins.sink.opensearch.ConnectionConfiguration - Using the trust all strategy
2022-09-27T18:19:00,655 [main] ERROR org.opensearch.dataprepper.plugin.PluginCreator - Encountered exception while instantiating the plugin OpenSearchSink
java.lang.reflect.InvocationTargetException: null
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
        at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
        at org.opensearch.dataprepper.plugin.PluginCreator.newPluginInstance(PluginCreator.java:40) ~[data-prepper-core-2.0.0-SNAPSHOT.jar:?]
        at org.opensearch.dataprepper.plugin.DefaultPluginFactory.loadPlugin(DefaultPluginFactory.java:61) ~[data-prepper-core-2.0.0-SNAPSHOT.jar:?]
        at org.opensearch.dataprepper.parser.PipelineParser.buildSinkOrConnector(PipelineParser.java:245) ~[data-prepper-core-2.0.0-SNAPSHOT.jar:?]
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?]
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
        at org.opensearch.dataprepper.parser.PipelineParser.buildPipelineFromConfiguration(PipelineParser.java:180) ~[data-prepper-core-2.0.0-SNAPSHOT.jar:?]
        at org.opensearch.dataprepper.parser.PipelineParser.parseConfiguration(PipelineParser.java:93) ~[data-prepper-core-2.0.0-SNAPSHOT.jar:?]
        at org.opensearch.dataprepper.DataPrepper.<init>(DataPrepper.java:60) ~[data-prepper-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
        at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
        at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:211) ~[spring-beans-5.3.22.jar:5.3.22]
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:117) ~[spring-beans-5.3.22.jar:5.3.22]
        at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:311) ~[spring-beans-5.3.22.jar:5.3.22]
        at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:296) ~[spring-beans-5.3.22.jar:5.3.22]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1372) ~[spring-beans
-5.3.22.jar:5.3.22]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1222) ~[spring-beans-
5.3.22.jar:5.3.22]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582) ~[spring-beans-5.3.22.
jar:5.3.22]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) ~[spring-beans-5.3.22.ja
r:5.3.22]
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) ~[spring-beans-5.3.22.jar:5.3.22]
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-5.3.22.jar:5.3.22]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) ~[spring-beans-5.3.22.jar:5.3.22]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) ~[spring-beans-5.3.22.jar:5.3.22]
        at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) ~[spring-beans-5.3.22.jar:5.3.22]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1391) ~[spring-beans-5.3.22.jar:5.3.
22]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1311) ~[spring-beans-5.3.22.jar:5.3.22
]
        at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:887) ~[spring-beans-5.3.22.jar:5.3.22]
        at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:791) ~[spring-beans-5.3.22.jar:5.3.22]
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:541) ~[spring-beans-5.3.22.jar:5.3.22]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1352) ~[sp
ring-beans-5.3.22.jar:5.3.22]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1195) ~[spring-beans-
5.3.22.jar:5.3.22]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582) ~[spring-beans-5.3.22.
jar:5.3.22]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) ~[spring-beans-5.3.22.ja
r:5.3.22]
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) ~[spring-beans-5.3.22.jar:5.3.22]
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-5.3.22.jar:5.3.22]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) ~[spring-beans-5.3.22.jar:5.3.22]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) ~[spring-beans-5.3.22.jar:5.3.22]
        at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) ~[spring-beans-5.3.22.jar:5.3.22]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1391) ~[spring-beans-5.3.22.jar:5.3.
22]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1311) ~[spring-beans-5.3.22.jar:5.3.22
]
        at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:887) ~[spring-beans-5.3.22.jar:5.3.22]
        at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:791) ~[spring-beans-5.3.22.jar:5.3.22]
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:541) ~[spring-beans-5.3.22.jar:5.3.22]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1352) ~[sp
ring-beans-5.3.22.jar:5.3.22]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1195) ~[spring-beans-
5.3.22.jar:5.3.22]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582) ~[spring-beans-5.3.22.
jar:5.3.22]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) ~[spring-beans-5.3.22.ja
r:5.3.22]
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) ~[spring-beans-5.3.22.jar:5.3.22]
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-5.3.22.jar:5.3.22]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) ~[spring-beans-5.3.22.jar:5.3.22]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) ~[spring-beans-5.3.22.jar:5.3.22]
        at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) ~[spring-beans-5.3.22.jar:5.3.22]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1391) ~[spring-beans-5.3.22.jar:5.3.
22]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1311) ~[spring-beans-5.3.22.jar:5.3.22
]
        at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:887) ~[spring-beans-5.3.22.jar:5.3.22]
        at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:791) ~[spring-beans-5.3.22.jar:5.3.22]
        at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:229) ~[spring-beans-5.3.22.jar:5.3.22]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1372) ~[spring-beans
-5.3.22.jar:5.3.22]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1222) ~[spring-beans-
5.3.22.jar:5.3.22]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582) ~[spring-beans-5.3.22.
jar:5.3.22]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) ~[spring-beans-5.3.22.ja
r:5.3.22]
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) ~[spring-beans-5.3.22.jar:5.3.22]
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-5.3.22.jar:5.3.22]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) ~[spring-beans-5.3.22.jar:5.3.22]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) ~[spring-beans-5.3.22.jar:5.3.22]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:955) ~[spring-beans-5.3.22.jar:
5.3.22]
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918) ~[spring-context-5.3.22.j
ar:5.3.22]
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583) ~[spring-context-5.3.22.jar:5.3.22]
        at org.opensearch.dataprepper.ContextManager.<init>(ContextManager.java:48) ~[data-prepper-main-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
        at org.opensearch.dataprepper.DataPrepperExecute.main(DataPrepperExecute.java:33) ~[data-prepper-main-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
Caused by: java.lang.RuntimeException: Invalid URI
        at org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink.<init>(OpenSearchSink.java:93) ~[opensearch-2.0.0-SNAPSHOT.jar:?]
        ... 82 more
Caused by: java.io.IOException: Invalid URI
        at org.opensearch.client.RestClient.extractAndWrapCause(RestClient.java:912) ~[opensearch-rest-client-1.3.5.jar:1.3.5]
        at org.opensearch.client.RestClient.performRequest(RestClient.java:301) ~[opensearch-rest-client-1.3.5.jar:1.3.5]
        at org.opensearch.client.RestClient.performRequest(RestClient.java:289) ~[opensearch-rest-client-1.3.5.jar:1.3.5]
        at org.opensearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1762) ~[opensearch-rest-high-level-client-1.3.5.jar:1.3.5]
        at org.opensearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1728) ~[opensearch-rest-high-level-client-1.3.5.jar:1.3.5]
        at org.opensearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1696) ~[opensearch-rest-high-level-client-1.3.5.jar:1.3.5]
        at org.opensearch.client.ClusterClient.getSettings(ClusterClient.java:119) ~[opensearch-rest-high-level-client-1.3.5.jar:1.3.5]
        at org.opensearch.dataprepper.plugins.sink.opensearch.index.IndexManager.checkISMEnabled(IndexManager.java:149) ~[opensearch-2.0.0-SNAPSHOT.jar:?]
        at org.opensearch.dataprepper.plugins.sink.opensearch.index.IndexManager.checkAndCreateIndexTemplate(IndexManager.java:165) ~[opensearch-2.0.0-SNAPSHOT.jar:?]
        at org.opensearch.dataprepper.plugins.sink.opensearch.index.IndexManager.setupIndex(IndexManager.java:160) ~[opensearch-2.0.0-SNAPSHOT.jar:?]
        at org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink.initialize(OpenSearchSink.java:106) ~[opensearch-2.0.0-SNAPSHOT.jar:?]
        at org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink.<init>(OpenSearchSink.java:90) ~[opensearch-2.0.0-SNAPSHOT.jar:?]
        ... 82 more
Caused by: java.io.IOException: Invalid URI
        at org.opensearch.dataprepper.plugins.sink.opensearch.AwsRequestSigningApacheInterceptor.buildUri(AwsRequestSigningApacheInterceptor.java:182) ~[opensearch-2.0.
0-SNAPSHOT.jar:?]
        at org.opensearch.dataprepper.plugins.sink.opensearch.AwsRequestSigningApacheInterceptor.process(AwsRequestSigningApacheInterceptor.java:134) ~[opensearch-2.0.0
-SNAPSHOT.jar:?]
        at org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:133) ~[httpcore-4.4.13.jar:4.4.13]
        at org.apache.http.impl.nio.client.MainClientExec.prepareRequest(MainClientExec.java:520) ~[httpasyncclient-4.1.4.jar:4.1.4]
        at org.apache.http.impl.nio.client.MainClientExec.prepare(MainClientExec.java:146) ~[httpasyncclient-4.1.4.jar:4.1.4]
        at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(DefaultClientExchangeHandlerImpl.java:128) ~[httpasyncclient-4.1.4.jar:4.1.4]
        at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:141) ~[httpasyncclient-4.1.4.jar:4.1.4]
        at org.opensearch.client.RestClient.performRequest(RestClient.java:297) ~[opensearch-rest-client-1.3.5.jar:1.3.5]
        at org.opensearch.client.RestClient.performRequest(RestClient.java:289) ~[opensearch-rest-client-1.3.5.jar:1.3.5]
        at org.opensearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1762) ~[opensearch-rest-high-level-client-1.3.5.jar:1.3.5]
        at org.opensearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1728) ~[opensearch-rest-high-level-client-1.3.5.jar:1.3.5]
        at org.opensearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1696) ~[opensearch-rest-high-level-client-1.3.5.jar:1.3.5]
        at org.opensearch.client.ClusterClient.getSettings(ClusterClient.java:119) ~[opensearch-rest-high-level-client-1.3.5.jar:1.3.5]
        at org.opensearch.dataprepper.plugins.sink.opensearch.index.IndexManager.checkISMEnabled(IndexManager.java:149) ~[opensearch-2.0.0-SNAPSHOT.jar:?]
        at org.opensearch.dataprepper.plugins.sink.opensearch.index.IndexManager.checkAndCreateIndexTemplate(IndexManager.java:165) ~[opensearch-2.0.0-SNAPSHOT.jar:?]
        at org.opensearch.dataprepper.plugins.sink.opensearch.index.IndexManager.setupIndex(IndexManager.java:160) ~[opensearch-2.0.0-SNAPSHOT.jar:?]
        at org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink.initialize(OpenSearchSink.java:106) ~[opensearch-2.0.0-SNAPSHOT.jar:?]
        at org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink.<init>(OpenSearchSink.java:90) ~[opensearch-2.0.0-SNAPSHOT.jar:?]
        ... 82 more
Caused by: java.net.URISyntaxException: Illegal character in authority at index 8: https://<your-domain>.<region>.es.amazonaws.com:443/_cluster/settings?master_timeout=
30s&include_defaults=true
        at java.net.URI$Parser.fail(URI.java:2974) ~[?:?]
        at java.net.URI$Parser.parseAuthority(URI.java:3308) ~[?:?]
        at java.net.URI$Parser.parseHierarchical(URI.java:3219) ~[?:?]
        at java.net.URI$Parser.parse(URI.java:3175) ~[?:?]
        at java.net.URI.<init>(URI.java:623) ~[?:?]
        at org.apache.http.client.utils.URIBuilder.build(URIBuilder.java:146) ~[httpclient-4.5.13.jar:4.5.13]
        at org.opensearch.dataprepper.plugins.sink.opensearch.AwsRequestSigningApacheInterceptor.buildUri(AwsRequestSigningApacheInterceptor.java:180) ~[opensearch-2.0.
0-SNAPSHOT.jar:?]
        at org.opensearch.dataprepper.plugins.sink.opensearch.AwsRequestSigningApacheInterceptor.process(AwsRequestSigningApacheInterceptor.java:134) ~[opensearch-2.0.0
-SNAPSHOT.jar:?]
        at org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:133) ~[httpcore-4.4.13.jar:4.4.13]
        at org.apache.http.impl.nio.client.MainClientExec.prepareRequest(MainClientExec.java:520) ~[httpasyncclient-4.1.4.jar:4.1.4]
        at org.apache.http.impl.nio.client.MainClientExec.prepare(MainClientExec.java:146) ~[httpasyncclient-4.1.4.jar:4.1.4]
        at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(DefaultClientExchangeHandlerImpl.java:128) ~[httpasyncclient-4.1.4.jar:4.1.4]
        at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:141) ~[httpasyncclient-4.1.4.jar:4.1.4]
        at org.opensearch.client.RestClient.performRequest(RestClient.java:297) ~[opensearch-rest-client-1.3.5.jar:1.3.5]
        at org.opensearch.client.RestClient.performRequest(RestClient.java:289) ~[opensearch-rest-client-1.3.5.jar:1.3.5]
        at org.opensearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1762) ~[opensearch-rest-high-level-client-1.3.5.jar:1.3.5]
        at org.opensearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1728) ~[opensearch-rest-high-level-client-1.3.5.jar:1.3.5]
        at org.opensearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1696) ~[opensearch-rest-high-level-client-1.3.5.jar:1.3.5]
        at org.opensearch.client.ClusterClient.getSettings(ClusterClient.java:119) ~[opensearch-rest-high-level-client-1.3.5.jar:1.3.5]
        at org.opensearch.dataprepper.plugins.sink.opensearch.index.IndexManager.checkISMEnabled(IndexManager.java:149) ~[opensearch-2.0.0-SNAPSHOT.jar:?]
        at org.opensearch.dataprepper.plugins.sink.opensearch.index.IndexManager.checkAndCreateIndexTemplate(IndexManager.java:165) ~[opensearch-2.0.0-SNAPSHOT.jar:?]
        at org.opensearch.dataprepper.plugins.sink.opensearch.index.IndexManager.setupIndex(IndexManager.java:160) ~[opensearch-2.0.0-SNAPSHOT.jar:?]
        at org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink.initialize(OpenSearchSink.java:106) ~[opensearch-2.0.0-SNAPSHOT.jar:?]
        at org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink.<init>(OpenSearchSink.java:90) ~[opensearch-2.0.0-SNAPSHOT.jar:?]
        ... 82 more
2022-09-27T18:19:00,661 [main] ERROR org.opensearch.dataprepper.parser.PipelineParser - Construction of pipeline components failed, skipping building of pipeline [servi
ce-map-pipeline] and its connected pipelines
com.amazon.dataprepper.model.plugin.PluginInvocationException: Exception throw from the plugin'OpenSearchSink'.
        at org.opensearch.dataprepper.plugin.PluginCreator.newPluginInstance(PluginCreator.java:47) ~[data-prepper-core-2.0.0-SNAPSHOT.jar:?]
        at org.opensearch.dataprepper.plugin.DefaultPluginFactory.loadPlugin(DefaultPluginFactory.java:61) ~[data-prepper-core-2.0.0-SNAPSHOT.jar:?]
        at org.opensearch.dataprepper.parser.PipelineParser.buildSinkOrConnector(PipelineParser.java:245) ~[data-prepper-core-2.0.0-SNAPSHOT.jar:?]
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?]
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
        at org.opensearch.dataprepper.parser.PipelineParser.buildPipelineFromConfiguration(PipelineParser.java:180) ~[data-prepper-core-2.0.0-SNAPSHOT.jar:?]
        at org.opensearch.dataprepper.parser.PipelineParser.parseConfiguration(PipelineParser.java:93) ~[data-prepper-core-2.0.0-SNAPSHOT.jar:?]
        at org.opensearch.dataprepper.DataPrepper.<init>(DataPrepper.java:60) ~[data-prepper-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
        at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
        at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:211) ~[spring-beans-5.3.22.jar:5.3.22]
   
 

```

---

<div class="post-metadata">

**Author:** ![kkondaka](https://avatars.discourse-cdn.com/v4/letter/k/71c47a/32.png) [@kkondaka](https://forum.opensearch.org/u/kkondaka)\
**Post date:** [September 27, 2022, 6:40pm UTC](https://forum.opensearch.org/t/latest-data-prepper-not-working/11055/6 "2022-09-27T18:40:32Z")

</div>

```auto

% cat ~/pipelines/pipeline_file_stdout.yaml
simple-sample-pipeline:
  workers: 2
  delay: "5000"
  source:
    file:
      path: /tmp/input-file
      record_type: event
  sink:
    - stdout:

```

---

<div class="post-metadata">

**Author:** ![kkondaka](https://avatars.discourse-cdn.com/v4/letter/k/71c47a/32.png) [@kkondaka](https://forum.opensearch.org/u/kkondaka)\
**Post date:** [September 27, 2022, 6:44pm UTC](https://forum.opensearch.org/t/latest-data-prepper-not-working/11055/7 "2022-09-27T18:44:17Z")

</div>

This exact same yaml file works with older data prepper

---

<div class="post-metadata">

**Author:** ![oeyh](https://avatars.discourse-cdn.com/v4/letter/o/8edcca/32.png) [@oeyh](https://forum.opensearch.org/u/oeyh)\
**Post date:** [September 27, 2022, 7:50pm UTC](https://forum.opensearch.org/t/latest-data-prepper-not-working/11055/8 "2022-09-27T19:50:55Z")

</div>

> [@kkondaka](#):
>
> Has anything (either command-line arguments or java env or configuration) changed since then?

Indeed there are changes to command line arguments for loading configuration files. Related to your case - running the Docker image, you can refer to this recently updated [section in Getting Started guide](https://github.com/opensearch-project/data-prepper/blob/main/docs/getting_started.md#running).

Here, I think you will need to mount your local pipeline config yaml to a yaml inside `/usr/share/data-prepper/pipelines/` directory. For example:`-v ~/pipelines/pipeline_file_stdout.yaml:/usr/share/data-prepper/pipelines/pipeline_file_stdout.yaml`. Otherwise, Data Prepper reads an example pipeline config which has some placeholders in it. That’s probably the reason for the errors you saw.

---

<div class="post-metadata">

**Author:** ![kkondaka](https://avatars.discourse-cdn.com/v4/letter/k/71c47a/32.png) [@kkondaka](https://forum.opensearch.org/u/kkondaka)\
**Post date:** [September 28, 2022, 6:00pm UTC](https://forum.opensearch.org/t/latest-data-prepper-not-working/11055/9 "2022-09-28T18:00:08Z")

</div>

Thanks @oeyh!

It looks like the change is to map `~/pipelines/pipeline_file_stdout.yaml` to `/usr/share/data-prepper/pipelines/pipelines.yaml`. Previously, I was mapping `~/pipelines/pipeline_file_stdout.yaml` to `/usr/share/data-prepper/pipelines.yaml`.

Also, if I use the way, you suggested, i.e., `-v ~/pipelines/pipeline_file_stdout.yaml:/usr/share/data-prepper/pipelines/pipeline_file_stdout.yaml`, it is still trying to initialize opensearch sink and hitting an exception while doing that. But still file source and stdout sink are being processed after printing that exception stack.

Basically, the change seems to be looking for pipeline yaml file in `/usr/share/data-prepper/pipelines/` instead of `/usr/share/data-prepper`. Was this done intentionally?

Thanks!

---

<div class="post-metadata">

**Author:** ![oeyh](https://avatars.discourse-cdn.com/v4/letter/o/8edcca/32.png) [@oeyh](https://forum.opensearch.org/u/oeyh)\
**Post date:** [September 28, 2022, 6:54pm UTC](https://forum.opensearch.org/t/latest-data-prepper-not-working/11055/10 "2022-09-28T18:54:39Z")

</div>

> [@kkondaka](#):
>
> Basically, the change seems to be looking for pipeline yaml file in `/usr/share/data-prepper/pipelines/` instead of `/usr/share/data-prepper`. Was this done intentionally?

Yes. The new behavior is to look for any yaml files in `pipelines/` directory to configure pipelines and you can put more than one pipeline yaml files in the directory.

> [@kkondaka](#):
>
> Also, if I use the way, you suggested, i.e., `-v ~/pipelines/pipeline_file_stdout.yaml:/usr/share/data-prepper/pipelines/pipeline_file_stdout.yaml`, it is still trying to initialize opensearch sink and hitting an exception while doing that. But still file source and stdout sink are being processed after printing that exception stack.

This makes sense. That’s probably because we have an example yaml called `pipelines.yaml` in the `/usr/share/data-prepper/pipelines/` directory in the first place and Data Prepper read that file as well, which causes the exception you saw. When you mount to `.../pipelines/piplelines.yaml` instead, it will overwrite the example yaml and build only the pipeline you defined.

Thank you for the feedback @kkondaka!

---

<div class="post-metadata">

**Author:** ![kkondaka](https://avatars.discourse-cdn.com/v4/letter/k/71c47a/32.png) [@kkondaka](https://forum.opensearch.org/u/kkondaka)\
**Post date:** [September 28, 2022, 8:28pm UTC](https://forum.opensearch.org/t/latest-data-prepper-not-working/11055/11 "2022-09-28T20:28:48Z")

</div>

That makes sense.

Thanks!  
Krishna