Versions (relevant - OpenSearch/Dashboard/Server OS/Browser): Latest
Describe the issue:
- opensearch_api is not explained in the document, although it was mentioned in some 2 years old blog as an available feature.
- traffic doesn’t flow when we use kafka buffer with opensearch_api. it works fine with bounded blocking.
Configuration: (see snippet below, refer to screenshot as well for formatted yaml)
version: “2”
dual-write-pipeline:
source:
opensearch_api:
port: 9200
buffer:
kafka:
bootstrap_servers: [“kafka:9092”]
authentication:
sasl:
plaintext:
username: “kafka-user”
password: “kafka-pass”
encryption:
type: ssl
topics:
- name: “opensearch-writes”
group_id: “dp-dual-write”
create_topic: true
sink:
- opensearch:
hosts: [“``https://cluster-a-direct:9200``”]
username: “admin”
password: “admin”
index: “${getMetadata("opensearch_index")}”
action: “${getMetadata("opensearch_action")}”
document_id: “${getMetadata("opensearch_id")}”
routing: “${getMetadata("opensearch_routing")}”
- opensearch:
hosts: [“``https://cluster-b:9200``”]
username: “admin”
password: “admin”
index: “${getMetadata("opensearch_index")}”
action: “${getMetadata("opensearch_action")}”
document_id: “${getMetadata("opensearch_id")}”
document_version_type: “external”
document_version: “${getMetadata("opensearch_document_version")}”
routing: "${getMetadata("opensearch_routing")}
Relevant Logs or Screenshots: