Exception when constructing the index name

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpenSearch node v2
DataPrepper v2.7.0

Describe the issue:
The error occurs when the OpenSearch client makes a GET request to the endpoint / _cluster/settings?include_defaults=true on the OpenSearch server at opensearch-node1:9200. This issue happens rarely and results in the following error:

ERROR org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - There was an exception when constructing the index name.

For that reason incoming data could not indexed and displayed in the dashboard.

Configuration:

version: "2"
event-pipeline:
  workers: 1
  delay: 100
  source:
    kafka:
      bootstrap_servers:
        - kafka:9092
      topics:
        - name: opensearch-topic
          group_id: opensearch-data-prepper
          serde_format: json
      encryption:
        type: none
        insecure: true
  buffer:
    bounded_blocking:
      buffer_size: 10240
      batch_size: 1024
  processor:
    - date:
        from_time_received: true
    - lowercase_string:
        with_keys: ["eventName"]
    - flatten:
        source: typedValueMap
        target: ""
    - delete_entries:
        with_keys: ["typedValueMap"]
  sink:
    - stdout:
    - opensearch:
        hosts: ["opensearch-node1:9200"]
        cert: /usr/share/data-prepper/certs/opensearch.pem
        key: /usr/share/data-prepper/certs/opensearch-key.pem
        trust: /usr/share/data-prepper/certs/root-ca.pem
        username: "username_test"
        password: "password_test"
        aws_sigv4: false
        insecure: false
        index: 'events-${/eventName}-%{yyyy.MM.dd}'
        dlq_file: /usr/share/data-prepper/log/event-dlq-file

Relevant Logs or Screenshots:
opensearch-node1:9200

2025-01-16T14:55:10,108 [event-pipeline-sink-worker-2-thread-1] DEBUG org.apache.http.impl.nio.client.AbstractClientExchangeHandler - [exchange: 406] Request connection for {s}->opensearch-node1:9200
2025-01-16T14:55:10,108 [event-pipeline-sink-worker-2-thread-1] DEBUG org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager - Connection request: [route: {s}->opensearch-node1:9200][total kept alive: 1; route allocated: 1 of 10; total allocated: 1 of 30]
2025-01-16T14:55:10,109 [event-pipeline-sink-worker-2-thread-1] DEBUG org.apache.http.impl.nio.conn.LoggingIOSession - http-outgoing-10 10.0.2.87:37982<->10.0.2.53:9200[ACTIVE][r:r][ACTIVE][r][NOT_HANDSHAKING][0][0][0]: Set timeout 0
2025-01-16T14:55:10,109 [event-pipeline-sink-worker-2-thread-1] DEBUG org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager$1 - Connection leased: [id: http-outgoing-10][route: {s}->opensearch-node1:9200][total kept alive: 0; route allocated: 1 of 10; total allocated: 1 of 30]
2025-01-16T14:55:10,109 [event-pipeline-sink-worker-2-thread-1] DEBUG org.apache.http.impl.nio.client.AbstractClientExchangeHandler - [exchange: 406] Connection allocated: CPoolProxy{http-outgoing-10 [ACTIVE]}
2025-01-16T14:55:10,109 [event-pipeline-sink-worker-2-thread-1] DEBUG org.apache.http.impl.nio.conn.LoggingIOSession - http-outgoing-10 10.0.2.87:37982<->10.0.2.53:9200[ACTIVE][r:r][ACTIVE][r][NOT_HANDSHAKING][0][0][0]: Set attribute http.nio.exchange-handler
2025-01-16T14:55:10,109 [event-pipeline-sink-worker-2-thread-1] DEBUG org.apache.http.impl.nio.conn.LoggingIOSession - http-outgoing-10 10.0.2.87:37982<->10.0.2.53:9200[ACTIVE][rw:r][ACTIVE][rw][NOT_HANDSHAKING][0][0][0]: Event set [w]
2025-01-16T14:55:10,109 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.client.InternalRequestExecutor - http-outgoing-10 [ACTIVE] Request ready
2025-01-16T14:55:10,109 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.client.MainClientExec - [exchange: 406] Attempt 1 to execute request
2025-01-16T14:55:10,109 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.client.MainClientExec - [exchange: 406] Target auth state: UNCHALLENGED
2025-01-16T14:55:10,109 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.client.MainClientExec - [exchange: 406] Proxy auth state: UNCHALLENGED
2025-01-16T14:55:10,109 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.conn.LoggingIOSession - http-outgoing-10 10.0.2.87:37982<->10.0.2.53:9200[ACTIVE][rw:w][ACTIVE][rw][NOT_HANDSHAKING][0][0][0]: Set timeout 30000
2025-01-16T14:55:10,109 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionImpl - http-outgoing-10 >> GET /_cluster/settings?include_defaults=true HTTP/1.1
2025-01-16T14:55:10,109 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionImpl - http-outgoing-10 >> User-Agent: opensearch-java/2.8.1 (Java/17.0.10)
2025-01-16T14:55:10,109 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionImpl - http-outgoing-10 >> Accept: application/json; charset=UTF-8
2025-01-16T14:55:10,109 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionImpl - http-outgoing-10 >> Content-Length: 0
2025-01-16T14:55:10,109 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionImpl - http-outgoing-10 >> Host: opensearch-node1:9200
2025-01-16T14:55:10,109 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionImpl - http-outgoing-10 >> Connection: Keep-Alive
2025-01-16T14:55:10,109 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionImpl - http-outgoing-10 >> Authorization: Basic xxx
2025-01-16T14:55:10,110 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.conn.LoggingIOSession - http-outgoing-10 10.0.2.87:37982<->10.0.2.53:9200[ACTIVE][rw:w][ACTIVE][rw][NOT_HANDSHAKING][0][0][0]: Event set [w]
2025-01-16T14:55:10,110 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.client.MainClientExec - [exchange: 406] Request completed
2025-01-16T14:55:10,110 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.conn.LoggingIOSession$LoggingByteChannel - http-outgoing-10 10.0.2.87:37982<->10.0.2.53:9200[ACTIVE][rw:w][ACTIVE][rw][NOT_HANDSHAKING][0][0][321]: 283 bytes written
2025-01-16T14:55:10,110 [I/O dispatcher 2] DEBUG       org.apache.http.impl.nio.conn.Wire - http-outgoing-10 >> "GET /_cluster/settings?include_defaults=true HTTP/1.1[\r][\n]"
2025-01-16T14:55:10,110 [I/O dispatcher 2] DEBUG       org.apache.http.impl.nio.conn.Wire - http-outgoing-10 >> "User-Agent: opensearch-java/2.8.1 (Java/17.0.10)[\r][\n]"
2025-01-16T14:55:10,110 [I/O dispatcher 2] DEBUG       org.apache.http.impl.nio.conn.Wire - http-outgoing-10 >> "Accept: application/json; charset=UTF-8[\r][\n]"
2025-01-16T14:55:10,110 [I/O dispatcher 2] DEBUG       org.apache.http.impl.nio.conn.Wire - http-outgoing-10 >> "Content-Length: 0[\r][\n]"
2025-01-16T14:55:10,110 [I/O dispatcher 2] DEBUG       org.apache.http.impl.nio.conn.Wire - http-outgoing-10 >> "Host: opensearch-node1:9200[\r][\n]"
2025-01-16T14:55:10,110 [I/O dispatcher 2] DEBUG       org.apache.http.impl.nio.conn.Wire - http-outgoing-10 >> "Connection: Keep-Alive[\r][\n]"
2025-01-16T14:55:10,110 [I/O dispatcher 2] DEBUG       org.apache.http.impl.nio.conn.Wire - http-outgoing-10 >> "Authorization: Basic xxx[\r][\n]"
2025-01-16T14:55:10,110 [I/O dispatcher 2] DEBUG       org.apache.http.impl.nio.conn.Wire - http-outgoing-10 >> "[\r][\n]"
2025-01-16T14:55:10,110 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.client.InternalRequestExecutor - http-outgoing-10 [ACTIVE] Exception
2025-01-16T14:55:10,111 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.conn.LoggingIOSession - http-outgoing-10 10.0.2.87:37982<->10.0.2.53:9200[ACTIVE][rw:rw][ACTIVE][rw][NOT_HANDSHAKING][0][0][0]: Shutdown
2025-01-16T14:55:10,111 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.conn.LoggingIOSession - http-outgoing-10 10.0.2.87:37982<->10.0.2.53:9200[CLOSED][][CLOSED][rw][NOT_HANDSHAKING][0][0][0]: Shutdownindex name
2025-01-16T14:55:10,111 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.client.AbstractClientExchangeHandler - [exchange: 406] connection aborted
2025-01-16T14:55:10,111 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager - Releasing connection: [id: http-outgoing-10][route: {s}->opensearch-node1:9200][total kept alive: 0; route allocated: 1 of 10; total allocated: 1 of 30]
2025-01-16T14:55:10,111 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager - Connection released: [id: http-outgoing-10][route: {s}->opensearch-node1:9200][total kept alive: 0; route allocated: 0 of 10; total allocated: 0 of 30]
2025-01-16T14:55:10,111 [event-pipeline-sink-worker-2-thread-1] DEBUG      org.opensearch.client.RequestLogger - request [GET opensearch-node1:9200/_cluster/settings?include_defaults=true] failed
2025-01-16T14:55:10,112 [I/O dispatcher 2] DEBUG org.apache.http.impl.nio.client.InternalRequestExecutor - http-outgoing-10 [CLOSED]: Disconnected
2025-01-16T14:55:10,112 [event-pipeline-sink-worker-2-thread-1] DEBUG         org.opensearch.client.RestClient - added [[host=opensearch-node1:9200]] to denylist
2025-01-16T14:55:10,112 [event-pipeline-sink-worker-2-thread-1] ERROR org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - There was an exception when constructing the index name. Check the dlq if configured to see details about the affected Event: Connection reset

Hi @ilkererikli,

Could it be that your “${/eventName}” occasionally has an Uppercase letter(s) or special characters?

See here for more on Index name restrictions:

best,
mj

Hello Mantas,

Thanks for asking! The first thing I checked was the eventName, and it’s always lowercase.

Best,
Ilker