PutElasticSearchHttp Processorin Nifi is not ingesting data in Opensearch

OpenSearch
, ,
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Opensearch Version: 2.6.0
Nifi: 1.19.0
OS: Ubuntu Server 22.04

Issue:
I am using “PutElasticSearchHttp” processor in Nifi which is not sending data in opensearch. It is throwing bad Request error.
I have checked my opensearch IP and ports are correct , cluster is also healthy but data is not being ingested.

Relevant Logs:
Following error is coming in logs of Nifi
WARN [Timer-Driven Process Thread-5] o.a.n.p.e.PutElasticsearchHttp PutElasticsearchHttp[id=f839f409-8844-3c80-91e9-166029e9c813] Elasticsearch returned code 400 with message Bad Request, transferring flow file to failure

2

@Iqra_shafiq I’m not familiar with Apache Nifi, however, did you check the OpenSearch’s logs against any connectivity errors?
What user did you use in Apache Nifi to connect with the OpenSearch cluster?

3

There was no error in Opensearch logs. SSL was disabled in opensearch therefore no username/password was used.

4

@Iqra_shafiq Disabling SSL doesn’t disable authentication/authorization but only secured communication on port 9200. You’re still required to authenticate with OpenSearch.

Only disabling the security plugin will disable authentication but I don’t recommend this approach.

5

@Iqra_shafiq Try running the following commands from the Apache Nifi VM.

curl http://<OpenSearch_node_IP_or_FQDN>:9200

curl -u admin:<password> http://<OpenSearch_node_IP_or_FQDN>:9200
6 
curl http://<OpenSearch_node_IP_or_FQDN>:9200

This command response shows that cluster is accessible without authentication and authorization.

7

@Iqra_shafiq Can you trace Apache Nifi’s APIs sent to the OpenSearch cluster?
If yes, have you tried to send the PUT API constructed by Apache Nifi manually?

8

I was successful in ingesting data to Opensearch using PutElasticSearchJson processor which uses elasticsearch API.

9

@Iqra_shafiq According to Apache Nifi documentation PutElasticSearchHttp has been deprecated and PutElasticSearchJson should be used instead.

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-elasticsearch-nar/1.21.0/org.apache.nifi.processors.elasticsearch.PutElasticsearchHttp/index.html

10

I used PutElasticSearchJson processor instead of PutElasticSearchHttp and it resolved my issue.