PutElasticSearchHttp Processorin Nifi is not ingesting data in Opensearch

OpenSearch
, ,
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Opensearch Version: 2.6.0
Nifi: 1.19.0
OS: Ubuntu Server 22.04

Issue:
I am using “PutElasticSearchHttp” processor in Nifi which is not sending data in opensearch. It is throwing bad Request error.
I have checked my opensearch IP and ports are correct , cluster is also healthy but data is not being ingested.

Relevant Logs:
Following error is coming in logs of Nifi
WARN [Timer-Driven Process Thread-5] o.a.n.p.e.PutElasticsearchHttp PutElasticsearchHttp[id=f839f409-8844-3c80-91e9-166029e9c813] Elasticsearch returned code 400 with message Bad Request, transferring flow file to failure

2

@Iqra_shafiq I’m not familiar with Apache Nifi, however, did you check the OpenSearch’s logs against any connectivity errors?
What user did you use in Apache Nifi to connect with the OpenSearch cluster?

3

There was no error in Opensearch logs. SSL was disabled in opensearch therefore no username/password was used.

4

@Iqra_shafiq Disabling SSL doesn’t disable authentication/authorization but only secured communication on port 9200. You’re still required to authenticate with OpenSearch.

Only disabling the security plugin will disable authentication but I don’t recommend this approach.

5

@Iqra_shafiq Try running the following commands from the Apache Nifi VM.

curl http://<OpenSearch_node_IP_or_FQDN>:9200

curl -u admin:<password> http://<OpenSearch_node_IP_or_FQDN>:9200
6 
curl http://<OpenSearch_node_IP_or_FQDN>:9200

This command response shows that cluster is accessible without authentication and authorization.

7

@Iqra_shafiq Can you trace Apache Nifi’s APIs sent to the OpenSearch cluster?
If yes, have you tried to send the PUT API constructed by Apache Nifi manually?

8

I was successful in ingesting data to Opensearch using PutElasticSearchJson processor which uses elasticsearch API.

9

@Iqra_shafiq According to Apache Nifi documentation PutElasticSearchHttp has been deprecated and PutElasticSearchJson should be used instead.

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-elasticsearch-nar/1.21.0/org.apache.nifi.processors.elasticsearch.PutElasticsearchHttp/index.html

10

I used PutElasticSearchJson processor instead of PutElasticSearchHttp and it resolved my issue.

11

Can someone please let me know if Nifi ingested data to the OpenSearch version 2.5 ? If so would you please comment on performance and ingestion rate using Nifi tool for OpenSearch ingestion ?

12

I have done this for Opensearch version 2.6.0.
I have done this ingestion in testing environment (not production). It works fine, however haven’t noticed ingestion rate.
Ingestion rate also depends on Nifi cluster and way the queues are configured in Nifi.

13

Thank you for the info. So you just used putelasticsearchjson processor and accomplished. What is the size of the data in sraging? Trying to evaluate the nifi ingestion to opensearch for production use

14

I did it few months back, I don’t remember exactly.