I have been using data-prepper with older source code ( code from Jul 8, hash 59d43209a239b33a9089248e68e233472cce8ecc) and it works fine. But when I try to use latest data-prepper (just by doing git clone https://github.com/opensearch-project/data-prepper.git), I see many issues. Compilation goes fine but at runtime I see many issues.
Has anything (either command-line arguments or java env or configuration) changed since then? Some of the issues I am seeing are -
I am using file as source and stdout as sink, but the data prepper crashes while instantiating opensearch sink
If I comment out “initialize” in opensearch sink, it does not crash but still file source is not processed
Previously, doing “ctrl-C” kills the docker instance but it does not do that anymore
I tried with Java 11 and java 17 but in both cases, data-prepper prints the following. I do not see this when using older data-prepper
/opt/java/openjdk/bin/java
Found openjdk version of 17.0.4
(standard_in) 1: syntax error
Can you please post your pipeline.yaml file? There have been several changes lately and it could be that there was a breaking change. I’d like to confirm.
First, we are preparing for Data Prepper 2.0 which does introduce some breaking changes. Those are reflected in main and thus some things will be different for you. As @ddpowers requested, please share your pipeline so that we can best help. I’ll also try to address a few points below.
There is a known issue where OpenSearch needs to be running. Feel free to share your logs so that we can help you resolve the issue.
Yes, this is needed to run correctly. Please be sure that OpenSearch is running first.
Can you elaborate on this? How are you running the Docker image?
The Docker image is based on an image with Java 17 installed. If you run from this image, you will get Java 17. If you want to run with a specific version of Java you should use the archives (.tar.gz). There is one with an embedded JDK 17 and one without a JDK.
With opesearch initialize() commented out, data-prepper starts fine but it does not read from the file and does not print anything to stdout. Are you saying that even for file source to work, open search must be initialized? I tried with opensearch running and still seeing the same error.
Run log (It looks like there is a limit on number of characters, so limiting to the first few lines of the log) -
dev-dsk-krishkdk-2b-9095a429 % docker run --rm --name data-prepper -p 4900:4900 --network=opensearch_opensearch_opensearch-net -v /tmp/input-file:/tmp/input-file -v
~/pipelines/pipeline_file_stdout.yaml:/usr/share/data-prepper/pipelines.yaml -v ${PWD}/data-prepper-config.yaml:/usr/share/data-prepper/data-prepper-config.yaml opensea
rch-data-prepper:2.0.0-SNAPSHOT
Reading pipelines and data-prepper configuration files from Data Prepper home directory.
/opt/java/openjdk/bin/java
Found openjdk version of 17.0.4
(standard_in) 1: syntax error
2022-09-27T18:18:59,451 [main] INFO org.opensearch.dataprepper.parser.config.DataPrepperAppConfiguration - Command line args: /usr/share/data-prepper/pipelines,/usr/sh
are/data-prepper/config/data-prepper-config.yaml
2022-09-27T18:18:59,453 [main] INFO org.opensearch.dataprepper.parser.config.DataPrepperArgs - Using /usr/share/data-prepper/pipelines configuration file
2022-09-27T18:18:59,856 [main] WARN org.opensearch.dataprepper.peerforwarder.server.PeerForwarderHttpServerProvider - Creating Peer Forwarder server without SSL/TLS. T
his is not secure.
2022-09-27T18:19:00,026 [main] INFO org.opensearch.dataprepper.parser.PipelineParser - Reading pipeline configuration from pipelines.yaml
2022-09-27T18:19:00,040 [main] INFO org.opensearch.dataprepper.parser.PipelineParser - Building pipeline [entry-pipeline] from provided configuration
2022-09-27T18:19:00,040 [main] INFO org.opensearch.dataprepper.parser.PipelineParser - Building [otel_trace_source] as source component for the pipeline [entry-pipelin
e]
2022-09-27T18:19:00,095 [main] WARN com.amazon.dataprepper.plugins.source.oteltrace.OTelTraceSource - Creating otel-trace-source without authentication. This is not se
cure.
2022-09-27T18:19:00,095 [main] WARN com.amazon.dataprepper.plugins.source.oteltrace.OTelTraceSource - In order to set up Http Basic authentication for the otel-trace-s
ource, go here: https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/otel-trace-source#authentication-configurations
2022-09-27T18:19:00,096 [main] INFO org.opensearch.dataprepper.parser.PipelineParser - Building buffer for the pipeline [entry-pipeline]
2022-09-27T18:19:00,097 [main] INFO org.opensearch.dataprepper.parser.PipelineParser - Building processors for the pipeline [entry-pipeline]
2022-09-27T18:19:00,097 [main] INFO org.opensearch.dataprepper.parser.PipelineParser - Building sinks for the pipeline [entry-pipeline]
2022-09-27T18:19:00,098 [main] INFO org.opensearch.dataprepper.parser.PipelineParser - Building [pipeline] as sink component
2022-09-27T18:19:00,098 [main] INFO org.opensearch.dataprepper.parser.PipelineParser - Building [pipeline] as sink component
2022-09-27T18:19:00,101 [main] INFO org.opensearch.dataprepper.parser.PipelineParser - Building pipeline [service-map-pipeline] from provided configuration
2022-09-27T18:19:00,101 [main] INFO org.opensearch.dataprepper.parser.PipelineParser - Building [pipeline] as source component for the pipeline [service-map-pipeline]
2022-09-27T18:19:00,101 [main] INFO org.opensearch.dataprepper.parser.PipelineParser - Building buffer for the pipeline [service-map-pipeline]
2022-09-27T18:19:00,102 [main] INFO org.opensearch.dataprepper.parser.PipelineParser - Building processors for the pipeline [service-map-pipeline]
2022-09-27T18:19:00,407 [main] INFO org.opensearch.dataprepper.parser.PipelineParser - Building sinks for the pipeline [service-map-pipeline]
2022-09-27T18:19:00,407 [main] INFO org.opensearch.dataprepper.parser.PipelineParser - Building [opensearch] as sink component
2022-09-27T18:19:00,422 [main] INFO org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Initializing OpenSearch sink
2022-09-27T18:19:00,428 [main] INFO org.opensearch.dataprepper.plugins.sink.opensearch.ConnectionConfiguration - aws_sigv4 is set, will sign requests using AWSRequestS
igningApacheInterceptor
2022-09-27T18:19:00,538 [main] INFO org.opensearch.dataprepper.plugins.sink.opensearch.ConnectionConfiguration - Using the trust all strategy
2022-09-27T18:19:00,655 [main] ERROR org.opensearch.dataprepper.plugin.PluginCreator - Encountered exception while instantiating the plugin OpenSearchSink
java.lang.reflect.InvocationTargetException: null
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
at org.opensearch.dataprepper.plugin.PluginCreator.newPluginInstance(PluginCreator.java:40) ~[data-prepper-core-2.0.0-SNAPSHOT.jar:?]
at org.opensearch.dataprepper.plugin.DefaultPluginFactory.loadPlugin(DefaultPluginFactory.java:61) ~[data-prepper-core-2.0.0-SNAPSHOT.jar:?]
at org.opensearch.dataprepper.parser.PipelineParser.buildSinkOrConnector(PipelineParser.java:245) ~[data-prepper-core-2.0.0-SNAPSHOT.jar:?]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
at org.opensearch.dataprepper.parser.PipelineParser.buildPipelineFromConfiguration(PipelineParser.java:180) ~[data-prepper-core-2.0.0-SNAPSHOT.jar:?]
at org.opensearch.dataprepper.parser.PipelineParser.parseConfiguration(PipelineParser.java:93) ~[data-prepper-core-2.0.0-SNAPSHOT.jar:?]
at org.opensearch.dataprepper.DataPrepper.<init>(DataPrepper.java:60) ~[data-prepper-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:211) ~[spring-beans-5.3.22.jar:5.3.22]
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:117) ~[spring-beans-5.3.22.jar:5.3.22]
at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:311) ~[spring-beans-5.3.22.jar:5.3.22]
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:296) ~[spring-beans-5.3.22.jar:5.3.22]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1372) ~[spring-beans
-5.3.22.jar:5.3.22]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1222) ~[spring-beans-
5.3.22.jar:5.3.22]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582) ~[spring-beans-5.3.22.
jar:5.3.22]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) ~[spring-beans-5.3.22.ja
r:5.3.22]
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) ~[spring-beans-5.3.22.jar:5.3.22]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-5.3.22.jar:5.3.22]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) ~[spring-beans-5.3.22.jar:5.3.22]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) ~[spring-beans-5.3.22.jar:5.3.22]
at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) ~[spring-beans-5.3.22.jar:5.3.22]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1391) ~[spring-beans-5.3.22.jar:5.3.
22]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1311) ~[spring-beans-5.3.22.jar:5.3.22
]
at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:887) ~[spring-beans-5.3.22.jar:5.3.22]
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:791) ~[spring-beans-5.3.22.jar:5.3.22]
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:541) ~[spring-beans-5.3.22.jar:5.3.22]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1352) ~[sp
ring-beans-5.3.22.jar:5.3.22]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1195) ~[spring-beans-
5.3.22.jar:5.3.22]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582) ~[spring-beans-5.3.22.
jar:5.3.22]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) ~[spring-beans-5.3.22.ja
r:5.3.22]
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) ~[spring-beans-5.3.22.jar:5.3.22]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-5.3.22.jar:5.3.22]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) ~[spring-beans-5.3.22.jar:5.3.22]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) ~[spring-beans-5.3.22.jar:5.3.22]
at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) ~[spring-beans-5.3.22.jar:5.3.22]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1391) ~[spring-beans-5.3.22.jar:5.3.
22]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1311) ~[spring-beans-5.3.22.jar:5.3.22
]
at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:887) ~[spring-beans-5.3.22.jar:5.3.22]
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:791) ~[spring-beans-5.3.22.jar:5.3.22]
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:541) ~[spring-beans-5.3.22.jar:5.3.22]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1352) ~[sp
ring-beans-5.3.22.jar:5.3.22]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1195) ~[spring-beans-
5.3.22.jar:5.3.22]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582) ~[spring-beans-5.3.22.
jar:5.3.22]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) ~[spring-beans-5.3.22.ja
r:5.3.22]
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) ~[spring-beans-5.3.22.jar:5.3.22]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-5.3.22.jar:5.3.22]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) ~[spring-beans-5.3.22.jar:5.3.22]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) ~[spring-beans-5.3.22.jar:5.3.22]
at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) ~[spring-beans-5.3.22.jar:5.3.22]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1391) ~[spring-beans-5.3.22.jar:5.3.
22]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1311) ~[spring-beans-5.3.22.jar:5.3.22
]
at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:887) ~[spring-beans-5.3.22.jar:5.3.22]
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:791) ~[spring-beans-5.3.22.jar:5.3.22]
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:229) ~[spring-beans-5.3.22.jar:5.3.22]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1372) ~[spring-beans
-5.3.22.jar:5.3.22]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1222) ~[spring-beans-
5.3.22.jar:5.3.22]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582) ~[spring-beans-5.3.22.
jar:5.3.22]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) ~[spring-beans-5.3.22.ja
r:5.3.22]
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) ~[spring-beans-5.3.22.jar:5.3.22]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-5.3.22.jar:5.3.22]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) ~[spring-beans-5.3.22.jar:5.3.22]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) ~[spring-beans-5.3.22.jar:5.3.22]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:955) ~[spring-beans-5.3.22.jar:
5.3.22]
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918) ~[spring-context-5.3.22.j
ar:5.3.22]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583) ~[spring-context-5.3.22.jar:5.3.22]
at org.opensearch.dataprepper.ContextManager.<init>(ContextManager.java:48) ~[data-prepper-main-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
at org.opensearch.dataprepper.DataPrepperExecute.main(DataPrepperExecute.java:33) ~[data-prepper-main-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
Caused by: java.lang.RuntimeException: Invalid URI
at org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink.<init>(OpenSearchSink.java:93) ~[opensearch-2.0.0-SNAPSHOT.jar:?]
... 82 more
Caused by: java.io.IOException: Invalid URI
at org.opensearch.client.RestClient.extractAndWrapCause(RestClient.java:912) ~[opensearch-rest-client-1.3.5.jar:1.3.5]
at org.opensearch.client.RestClient.performRequest(RestClient.java:301) ~[opensearch-rest-client-1.3.5.jar:1.3.5]
at org.opensearch.client.RestClient.performRequest(RestClient.java:289) ~[opensearch-rest-client-1.3.5.jar:1.3.5]
at org.opensearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1762) ~[opensearch-rest-high-level-client-1.3.5.jar:1.3.5]
at org.opensearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1728) ~[opensearch-rest-high-level-client-1.3.5.jar:1.3.5]
at org.opensearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1696) ~[opensearch-rest-high-level-client-1.3.5.jar:1.3.5]
at org.opensearch.client.ClusterClient.getSettings(ClusterClient.java:119) ~[opensearch-rest-high-level-client-1.3.5.jar:1.3.5]
at org.opensearch.dataprepper.plugins.sink.opensearch.index.IndexManager.checkISMEnabled(IndexManager.java:149) ~[opensearch-2.0.0-SNAPSHOT.jar:?]
at org.opensearch.dataprepper.plugins.sink.opensearch.index.IndexManager.checkAndCreateIndexTemplate(IndexManager.java:165) ~[opensearch-2.0.0-SNAPSHOT.jar:?]
at org.opensearch.dataprepper.plugins.sink.opensearch.index.IndexManager.setupIndex(IndexManager.java:160) ~[opensearch-2.0.0-SNAPSHOT.jar:?]
at org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink.initialize(OpenSearchSink.java:106) ~[opensearch-2.0.0-SNAPSHOT.jar:?]
at org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink.<init>(OpenSearchSink.java:90) ~[opensearch-2.0.0-SNAPSHOT.jar:?]
... 82 more
Caused by: java.io.IOException: Invalid URI
at org.opensearch.dataprepper.plugins.sink.opensearch.AwsRequestSigningApacheInterceptor.buildUri(AwsRequestSigningApacheInterceptor.java:182) ~[opensearch-2.0.
0-SNAPSHOT.jar:?]
at org.opensearch.dataprepper.plugins.sink.opensearch.AwsRequestSigningApacheInterceptor.process(AwsRequestSigningApacheInterceptor.java:134) ~[opensearch-2.0.0
-SNAPSHOT.jar:?]
at org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:133) ~[httpcore-4.4.13.jar:4.4.13]
at org.apache.http.impl.nio.client.MainClientExec.prepareRequest(MainClientExec.java:520) ~[httpasyncclient-4.1.4.jar:4.1.4]
at org.apache.http.impl.nio.client.MainClientExec.prepare(MainClientExec.java:146) ~[httpasyncclient-4.1.4.jar:4.1.4]
at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(DefaultClientExchangeHandlerImpl.java:128) ~[httpasyncclient-4.1.4.jar:4.1.4]
at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:141) ~[httpasyncclient-4.1.4.jar:4.1.4]
at org.opensearch.client.RestClient.performRequest(RestClient.java:297) ~[opensearch-rest-client-1.3.5.jar:1.3.5]
at org.opensearch.client.RestClient.performRequest(RestClient.java:289) ~[opensearch-rest-client-1.3.5.jar:1.3.5]
at org.opensearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1762) ~[opensearch-rest-high-level-client-1.3.5.jar:1.3.5]
at org.opensearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1728) ~[opensearch-rest-high-level-client-1.3.5.jar:1.3.5]
at org.opensearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1696) ~[opensearch-rest-high-level-client-1.3.5.jar:1.3.5]
at org.opensearch.client.ClusterClient.getSettings(ClusterClient.java:119) ~[opensearch-rest-high-level-client-1.3.5.jar:1.3.5]
at org.opensearch.dataprepper.plugins.sink.opensearch.index.IndexManager.checkISMEnabled(IndexManager.java:149) ~[opensearch-2.0.0-SNAPSHOT.jar:?]
at org.opensearch.dataprepper.plugins.sink.opensearch.index.IndexManager.checkAndCreateIndexTemplate(IndexManager.java:165) ~[opensearch-2.0.0-SNAPSHOT.jar:?]
at org.opensearch.dataprepper.plugins.sink.opensearch.index.IndexManager.setupIndex(IndexManager.java:160) ~[opensearch-2.0.0-SNAPSHOT.jar:?]
at org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink.initialize(OpenSearchSink.java:106) ~[opensearch-2.0.0-SNAPSHOT.jar:?]
at org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink.<init>(OpenSearchSink.java:90) ~[opensearch-2.0.0-SNAPSHOT.jar:?]
... 82 more
Caused by: java.net.URISyntaxException: Illegal character in authority at index 8: https://<your-domain>.<region>.es.amazonaws.com:443/_cluster/settings?master_timeout=
30s&include_defaults=true
at java.net.URI$Parser.fail(URI.java:2974) ~[?:?]
at java.net.URI$Parser.parseAuthority(URI.java:3308) ~[?:?]
at java.net.URI$Parser.parseHierarchical(URI.java:3219) ~[?:?]
at java.net.URI$Parser.parse(URI.java:3175) ~[?:?]
at java.net.URI.<init>(URI.java:623) ~[?:?]
at org.apache.http.client.utils.URIBuilder.build(URIBuilder.java:146) ~[httpclient-4.5.13.jar:4.5.13]
at org.opensearch.dataprepper.plugins.sink.opensearch.AwsRequestSigningApacheInterceptor.buildUri(AwsRequestSigningApacheInterceptor.java:180) ~[opensearch-2.0.
0-SNAPSHOT.jar:?]
at org.opensearch.dataprepper.plugins.sink.opensearch.AwsRequestSigningApacheInterceptor.process(AwsRequestSigningApacheInterceptor.java:134) ~[opensearch-2.0.0
-SNAPSHOT.jar:?]
at org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:133) ~[httpcore-4.4.13.jar:4.4.13]
at org.apache.http.impl.nio.client.MainClientExec.prepareRequest(MainClientExec.java:520) ~[httpasyncclient-4.1.4.jar:4.1.4]
at org.apache.http.impl.nio.client.MainClientExec.prepare(MainClientExec.java:146) ~[httpasyncclient-4.1.4.jar:4.1.4]
at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(DefaultClientExchangeHandlerImpl.java:128) ~[httpasyncclient-4.1.4.jar:4.1.4]
at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:141) ~[httpasyncclient-4.1.4.jar:4.1.4]
at org.opensearch.client.RestClient.performRequest(RestClient.java:297) ~[opensearch-rest-client-1.3.5.jar:1.3.5]
at org.opensearch.client.RestClient.performRequest(RestClient.java:289) ~[opensearch-rest-client-1.3.5.jar:1.3.5]
at org.opensearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1762) ~[opensearch-rest-high-level-client-1.3.5.jar:1.3.5]
at org.opensearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1728) ~[opensearch-rest-high-level-client-1.3.5.jar:1.3.5]
at org.opensearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1696) ~[opensearch-rest-high-level-client-1.3.5.jar:1.3.5]
at org.opensearch.client.ClusterClient.getSettings(ClusterClient.java:119) ~[opensearch-rest-high-level-client-1.3.5.jar:1.3.5]
at org.opensearch.dataprepper.plugins.sink.opensearch.index.IndexManager.checkISMEnabled(IndexManager.java:149) ~[opensearch-2.0.0-SNAPSHOT.jar:?]
at org.opensearch.dataprepper.plugins.sink.opensearch.index.IndexManager.checkAndCreateIndexTemplate(IndexManager.java:165) ~[opensearch-2.0.0-SNAPSHOT.jar:?]
at org.opensearch.dataprepper.plugins.sink.opensearch.index.IndexManager.setupIndex(IndexManager.java:160) ~[opensearch-2.0.0-SNAPSHOT.jar:?]
at org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink.initialize(OpenSearchSink.java:106) ~[opensearch-2.0.0-SNAPSHOT.jar:?]
at org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink.<init>(OpenSearchSink.java:90) ~[opensearch-2.0.0-SNAPSHOT.jar:?]
... 82 more
2022-09-27T18:19:00,661 [main] ERROR org.opensearch.dataprepper.parser.PipelineParser - Construction of pipeline components failed, skipping building of pipeline [servi
ce-map-pipeline] and its connected pipelines
com.amazon.dataprepper.model.plugin.PluginInvocationException: Exception throw from the plugin'OpenSearchSink'.
at org.opensearch.dataprepper.plugin.PluginCreator.newPluginInstance(PluginCreator.java:47) ~[data-prepper-core-2.0.0-SNAPSHOT.jar:?]
at org.opensearch.dataprepper.plugin.DefaultPluginFactory.loadPlugin(DefaultPluginFactory.java:61) ~[data-prepper-core-2.0.0-SNAPSHOT.jar:?]
at org.opensearch.dataprepper.parser.PipelineParser.buildSinkOrConnector(PipelineParser.java:245) ~[data-prepper-core-2.0.0-SNAPSHOT.jar:?]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
at org.opensearch.dataprepper.parser.PipelineParser.buildPipelineFromConfiguration(PipelineParser.java:180) ~[data-prepper-core-2.0.0-SNAPSHOT.jar:?]
at org.opensearch.dataprepper.parser.PipelineParser.parseConfiguration(PipelineParser.java:93) ~[data-prepper-core-2.0.0-SNAPSHOT.jar:?]
at org.opensearch.dataprepper.DataPrepper.<init>(DataPrepper.java:60) ~[data-prepper-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:211) ~[spring-beans-5.3.22.jar:5.3.22]
Indeed there are changes to command line arguments for loading configuration files. Related to your case - running the Docker image, you can refer to this recently updated section in Getting Started guide.
Here, I think you will need to mount your local pipeline config yaml to a yaml inside /usr/share/data-prepper/pipelines/ directory. For example:-v ~/pipelines/pipeline_file_stdout.yaml:/usr/share/data-prepper/pipelines/pipeline_file_stdout.yaml. Otherwise, Data Prepper reads an example pipeline config which has some placeholders in it. That’s probably the reason for the errors you saw.
It looks like the change is to map ~/pipelines/pipeline_file_stdout.yaml to /usr/share/data-prepper/pipelines/pipelines.yaml. Previously, I was mapping ~/pipelines/pipeline_file_stdout.yaml to /usr/share/data-prepper/pipelines.yaml.
Also, if I use the way, you suggested, i.e., -v ~/pipelines/pipeline_file_stdout.yaml:/usr/share/data-prepper/pipelines/pipeline_file_stdout.yaml, it is still trying to initialize opensearch sink and hitting an exception while doing that. But still file source and stdout sink are being processed after printing that exception stack.
Basically, the change seems to be looking for pipeline yaml file in /usr/share/data-prepper/pipelines/ instead of /usr/share/data-prepper. Was this done intentionally?
Yes. The new behavior is to look for any yaml files in pipelines/ directory to configure pipelines and you can put more than one pipeline yaml files in the directory.
This makes sense. That’s probably because we have an example yaml called pipelines.yaml in the /usr/share/data-prepper/pipelines/ directory in the first place and Data Prepper read that file as well, which causes the exception you saw. When you mount to .../pipelines/piplelines.yaml instead, it will overwrite the example yaml and build only the pipeline you defined.