Issue:
I am using “PutElasticSearchHttp” processor in Nifi which is not sending data in opensearch. It is throwing bad Request error.
I have checked my opensearch IP and ports are correct , cluster is also healthy but data is not being ingested.
Relevant Logs:
Following error is coming in logs of Nifi
WARN [Timer-Driven Process Thread-5] o.a.n.p.e.PutElasticsearchHttp PutElasticsearchHttp[id=f839f409-8844-3c80-91e9-166029e9c813] Elasticsearch returned code 400 with message Bad Request, transferring flow file to failure
@Iqra_shafiq I’m not familiar with Apache Nifi, however, did you check the OpenSearch’s logs against any connectivity errors?
What user did you use in Apache Nifi to connect with the OpenSearch cluster?
@Iqra_shafiq Disabling SSL doesn’t disable authentication/authorization but only secured communication on port 9200. You’re still required to authenticate with OpenSearch.
Only disabling the security plugin will disable authentication but I don’t recommend this approach.
@Iqra_shafiq Can you trace Apache Nifi’s APIs sent to the OpenSearch cluster?
If yes, have you tried to send the PUT API constructed by Apache Nifi manually?
Can someone please let me know if Nifi ingested data to the OpenSearch version 2.5 ? If so would you please comment on performance and ingestion rate using Nifi tool for OpenSearch ingestion ?
I have done this for Opensearch version 2.6.0.
I have done this ingestion in testing environment (not production). It works fine, however haven’t noticed ingestion rate.
Ingestion rate also depends on Nifi cluster and way the queues are configured in Nifi.
Thank you for the info. So you just used putelasticsearchjson processor and accomplished. What is the size of the data in sraging? Trying to evaluate the nifi ingestion to opensearch for production use