Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpenSearch node v2
DataPrepper v2.7.0
Describe the issue:
The error occurs when the OpenSearch client makes a GET request to the endpoint / _cluster/settings?include_defaults=true on the OpenSearch server at opensearch-node1:9200. This issue happens rarely and results in the following error:
ERROR org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - There was an exception when constructing the index name.
For that reason incoming data could not indexed and displayed in the dashboard.
Configuration:
version: "2"
event-pipeline:
workers: 1
delay: 100
source:
kafka:
bootstrap_servers:
- kafka:9092
topics:
- name: opensearch-topic
group_id: opensearch-data-prepper
serde_format: json
encryption:
type: none
insecure: true
buffer:
bounded_blocking:
buffer_size: 10240
batch_size: 1024
processor:
- date:
from_time_received: true
- lowercase_string:
with_keys: ["eventName"]
- flatten:
source: typedValueMap
target: ""
- delete_entries:
with_keys: ["typedValueMap"]
sink:
- stdout:
- opensearch:
hosts: ["opensearch-node1:9200"]
cert: /usr/share/data-prepper/certs/opensearch.pem
key: /usr/share/data-prepper/certs/opensearch-key.pem
trust: /usr/share/data-prepper/certs/root-ca.pem
username: "username_test"
password: "password_test"
aws_sigv4: false
insecure: false
index: 'events-${/eventName}-%{yyyy.MM.dd}'
dlq_file: /usr/share/data-prepper/log/event-dlq-file
Relevant Logs or Screenshots:
opensearch-node1:9200
2025-01-16T14:55:10,108 [event-pipeline-sink-worker-2-thread-1] DEBUG org.apache.http.impl.nio.client.AbstractClientExchangeHandler - [exchange: 406] Request connection for {s}->opensearch-node1:9200
2025-01-16T14:55:10,108 [event-pipeline-sink-worker-2-thread-1] DEBUG org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager - Connection request: [route: {s}->opensearch-node1:9200][total kept alive: 1; route allocated: 1 of 10; total allocated: 1 of 30]
2025-01-16T14:55:10,109 [event-pipeline-sink-worker-2-thread-1] DEBUG org.apache.http.impl.nio.conn.LoggingIOSession - http-outgoing-10 10.0.2.87:37982<->10.0.2.53:9200[ACTIVE][r:r][ACTIVE][r][NOT_HANDSHAKING][0][0][0]: Set timeout 0
2025-01-16T14:55:10,109 [event-pipeline-sink-worker-2-thread-1] DEBUG org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager$1 - Connection leased: [id: http-outgoing-10][route: {s}->opensearch-node1:9200][total kept alive: 0; route allocated: 1 of 10; total allocated: 1 of 30]
2025-01-16T14:55:10,109 [event-pipeline-sink-worker-2-thread-1] DEBUG org.apache.http.impl.nio.client.AbstractClientExchangeHandler - [exchange: 406] Connection allocated: CPoolProxy{http-outgoing-10 [ACTIVE]}
2025-01-16T14:55:10,109 [event-pipeline-sink-worker-2-thread-1] DEBUG org.apache.http.impl.nio.conn.LoggingIOSession - http-outgoing-10 10.0.2.87:37982<->10.0.2.53:9200[ACTIVE][r:r][ACTIVE][r][NOT_HANDSHAKING][0][0][0]: Set attribute http.nio.exchange-handler
2025-01-16T14:55:10,109 [event-pipeline-sink-worker-2-thread-1] DEBUG org.apache.http.impl.nio.conn.LoggingIOSession - http-outgoing-10 10.0.2.87:37982<->10.0.2.53:9200[ACTIVE][rw:r][ACTIVE][rw][NOT_HANDSHAKING][0][0][0]: Event set [w]
2025-01-16T14:55:10,109 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.client.InternalRequestExecutor - http-outgoing-10 [ACTIVE] Request ready
2025-01-16T14:55:10,109 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.client.MainClientExec - [exchange: 406] Attempt 1 to execute request
2025-01-16T14:55:10,109 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.client.MainClientExec - [exchange: 406] Target auth state: UNCHALLENGED
2025-01-16T14:55:10,109 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.client.MainClientExec - [exchange: 406] Proxy auth state: UNCHALLENGED
2025-01-16T14:55:10,109 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.conn.LoggingIOSession - http-outgoing-10 10.0.2.87:37982<->10.0.2.53:9200[ACTIVE][rw:w][ACTIVE][rw][NOT_HANDSHAKING][0][0][0]: Set timeout 30000
2025-01-16T14:55:10,109 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionImpl - http-outgoing-10 >> GET /_cluster/settings?include_defaults=true HTTP/1.1
2025-01-16T14:55:10,109 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionImpl - http-outgoing-10 >> User-Agent: opensearch-java/2.8.1 (Java/17.0.10)
2025-01-16T14:55:10,109 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionImpl - http-outgoing-10 >> Accept: application/json; charset=UTF-8
2025-01-16T14:55:10,109 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionImpl - http-outgoing-10 >> Content-Length: 0
2025-01-16T14:55:10,109 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionImpl - http-outgoing-10 >> Host: opensearch-node1:9200
2025-01-16T14:55:10,109 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionImpl - http-outgoing-10 >> Connection: Keep-Alive
2025-01-16T14:55:10,109 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionImpl - http-outgoing-10 >> Authorization: Basic xxx
2025-01-16T14:55:10,110 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.conn.LoggingIOSession - http-outgoing-10 10.0.2.87:37982<->10.0.2.53:9200[ACTIVE][rw:w][ACTIVE][rw][NOT_HANDSHAKING][0][0][0]: Event set [w]
2025-01-16T14:55:10,110 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.client.MainClientExec - [exchange: 406] Request completed
2025-01-16T14:55:10,110 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.conn.LoggingIOSession$LoggingByteChannel - http-outgoing-10 10.0.2.87:37982<->10.0.2.53:9200[ACTIVE][rw:w][ACTIVE][rw][NOT_HANDSHAKING][0][0][321]: 283 bytes written
2025-01-16T14:55:10,110 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.conn.Wire - http-outgoing-10 >> "GET /_cluster/settings?include_defaults=true HTTP/1.1[\r][\n]"
2025-01-16T14:55:10,110 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.conn.Wire - http-outgoing-10 >> "User-Agent: opensearch-java/2.8.1 (Java/17.0.10)[\r][\n]"
2025-01-16T14:55:10,110 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.conn.Wire - http-outgoing-10 >> "Accept: application/json; charset=UTF-8[\r][\n]"
2025-01-16T14:55:10,110 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.conn.Wire - http-outgoing-10 >> "Content-Length: 0[\r][\n]"
2025-01-16T14:55:10,110 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.conn.Wire - http-outgoing-10 >> "Host: opensearch-node1:9200[\r][\n]"
2025-01-16T14:55:10,110 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.conn.Wire - http-outgoing-10 >> "Connection: Keep-Alive[\r][\n]"
2025-01-16T14:55:10,110 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.conn.Wire - http-outgoing-10 >> "Authorization: Basic xxx[\r][\n]"
2025-01-16T14:55:10,110 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.conn.Wire - http-outgoing-10 >> "[\r][\n]"
2025-01-16T14:55:10,110 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.client.InternalRequestExecutor - http-outgoing-10 [ACTIVE] Exception
2025-01-16T14:55:10,111 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.conn.LoggingIOSession - http-outgoing-10 10.0.2.87:37982<->10.0.2.53:9200[ACTIVE][rw:rw][ACTIVE][rw][NOT_HANDSHAKING][0][0][0]: Shutdown
2025-01-16T14:55:10,111 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.conn.LoggingIOSession - http-outgoing-10 10.0.2.87:37982<->10.0.2.53:9200[CLOSED][][CLOSED][rw][NOT_HANDSHAKING][0][0][0]: Shutdownindex name
2025-01-16T14:55:10,111 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.client.AbstractClientExchangeHandler - [exchange: 406] connection aborted
2025-01-16T14:55:10,111 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager - Releasing connection: [id: http-outgoing-10][route: {s}->opensearch-node1:9200][total kept alive: 0; route allocated: 1 of 10; total allocated: 1 of 30]
2025-01-16T14:55:10,111 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager - Connection released: [id: http-outgoing-10][route: {s}->opensearch-node1:9200][total kept alive: 0; route allocated: 0 of 10; total allocated: 0 of 30]
2025-01-16T14:55:10,111 [event-pipeline-sink-worker-2-thread-1] DEBUG org.opensearch.client.RequestLogger - request [GET opensearch-node1:9200/_cluster/settings?include_defaults=true] failed
2025-01-16T14:55:10,112 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.client.InternalRequestExecutor - http-outgoing-10 [CLOSED]: Disconnected
2025-01-16T14:55:10,112 [event-pipeline-sink-worker-2-thread-1] DEBUG org.opensearch.client.RestClient - added [[host=opensearch-node1:9200]] to denylist
2025-01-16T14:55:10,112 [event-pipeline-sink-worker-2-thread-1] ERROR org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - There was an exception when constructing the index name. Check the dlq if configured to see details about the affected Event: Connection reset