io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2Exception

SlavaUtesinov · October 28, 2026, 10:58am

Versions
OpenSearch 3.9.0
OpenSearch operator 2.8.0

Describe the issue:
I switched on grpc transport

but after I had made the first grpc request SearchService/Search (request completed successfully):
{
"index": [
"test_index1"
],
"search_request_body": {
"query": {
"match_all": {}
}
},
"x_source": {
"fetch": true
}
}
I noticed in logs of pod suspicious messages. They keep appear even if I don’t execute grpc requests.

Configuration:

apiVersion: opensearch.opster.io/v1
kind: OpenSearchCluster
status:
  availableNodes: 3
  componentsStatus:
    - component: Restarter
      status: Finished
  health: green
  initialized: true
  phase: RUNNING
  version: 3.9.0
spec:
  bootstrap:
    resources: {}
  confMgmt: {}
  dashboards:
    affinity:
      podAntiAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchExpressions:
                  - key: opensearch.cluster.dashboards
                    operator: In
                    values:
                      - cluster
              topologyKey: kubernetes.io/hostname
            weight: 100
    enable: true
    image: >-
      opensearch-dashboards:3.9.0
    opensearchCredentialsSecret:
      name: custom-admin-credentials
    replicas: 1
    resources:
      limits:
        cpu: 503m
        memory: 504M
      requests:
        cpu: 503m
        memory: 504M
    service:
      type: ClusterIP
    version: 3.9.0
  general:
    additionalVolumes:
      - configMap:
          name: log4j2.properties
        name: log
        path: /usr/share/opensearch/config/log4j2.properties
        restartPods: true
        subPath: log4j2.properties
      - name: snapshots0
        path: /usr/share/opensearch/config/snapshots0.crt
        restartPods: true
        secret:
          secretName: trusted-certificates
        subPath: snapshots0.crt
      - name: restore0
        path: /usr/share/opensearch/config/restore0.crt
        restartPods: true
        secret:
          secretName: trusted-certificates
        subPath: restore0.crt
      - name: plugins0
        path: /usr/share/opensearch/config/plugins0.crt
        restartPods: true
        secret:
          secretName: trusted-certificates
        subPath: plugins0.crt
    command: >
      ~/jdk/bin/keytool -importcert -storepass changeit -noprompt -alias
      snapshots0 -keystore /usr/share/opensearch/jdk/lib/security/cacerts -file
      /usr/share/opensearch/config/snapshots0.crt &&

      ~/jdk/bin/keytool -importcert -storepass changeit -noprompt -alias
      restore0 -keystore /usr/share/opensearch/jdk/lib/security/cacerts -file
      /usr/share/opensearch/config/restore0.crt &&

      ~/jdk/bin/keytool -importcert -storepass changeit -noprompt -alias
      plugins0 -keystore /usr/share/opensearch/jdk/lib/security/cacerts -file
      /usr/share/opensearch/config/plugins0.crt &&

      ./bin/opensearch-plugin install --batch
      'https://prometheus-exporter/prometheus-exporter-3.9.0.0.zip'
      'https://repository-s3/repository-s3-3.9.0.zip'
      &&

      ./opensearch-docker-entrypoint.sh
    httpPort: 9200
    image: opensearchproject/opensearch:3.9.0
    keystore:
      - secret:
          name: snapshot-repositories
    monitoring: {}
    serviceAccount: disabled-automount-service-account-token
    serviceName: cluster
    setVMMaxMapCount: true
    snapshotRepositories:
      - name: snapshots
        settings:
          base_path: snapshots32
          bucket: snapshots
          client: snapshots
          readonly: 'false'
          region: us-east-1
        type: s3
      - name: restore
        settings:
          base_path: snapshots30
          bucket: restore
          client: restore
          readonly: 'true'
          region: us-east-1
        type: s3
    vendor: opensearch
    version: 3.9.0
  initHelper:
    image: >-
      busybox:1.36.1
    resources: {}
  nodePools:
    - additionalConfig:
        aux.transport.transport-grpc.port: 9400-9400
        aux.transport.types: '["transport-grpc"]'
        s3.client.restore.endpoint: <endpoint>
        s3.client.restore.max_retries: '3'
        s3.client.restore.path_style_access: 'true'
        s3.client.restore.protocol: https
        s3.client.restore.read_timeout: 60s
        s3.client.snapshots.endpoint: <endpoint>
        s3.client.snapshots.max_retries: '3'
        s3.client.snapshots.path_style_access: 'true'
        s3.client.snapshots.protocol: http
        s3.client.snapshots.read_timeout: 60s
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                labelSelector:
                  matchExpressions:
                    - key: opster.io/opensearch-nodepool
                      operator: In
                      values:
                        - data
                topologyKey: kubernetes.io/hostname
              weight: 100
      annotations:
        prometheus.io/path: /_prometheus/metrics
        prometheus.io/port: '9200'
        prometheus.io/scheme: https
        prometheus.io/scrape: 'true'
      component: data
      diskSize: 2Gi
      env:
        - name: AWS_EC2_METADATA_DISABLED
          value: 'true'
      pdb:
        enable: true
        maxUnavailable: 1
      persistence:
        pvc:
          accessModes:
            - ReadWriteOnce
          storageClass: <custom>
      replicas: 3
      resources:
        limits:
          cpu: 2500m
          memory: 10G
        requests:
          cpu: 2500m
          memory: 10G
      roles:
        - cluster_manager
        - data
        - ingest
  security:
    config:
      adminCredentialsSecret:
        name: custom-admin-credentials
      adminSecret:
        name: custom-admin-cert
      securityConfigSecret:
        name: custom-security-config
      updateJob:
        resources: {}
    tls:
      http:
        caSecret:
          name: ''
        secret:
          name: custom-http-cert
      transport:
        adminDn:
          - <custom>
        caSecret:
          name: ''
        nodesDn:
          - <custom>
        secret:
          name: custom-transport-cert

Relevant Logs or Screenshots:

{"type": "server", "timestamp": "2026-10-28T10:52:02,822+0000", "level": "INFO", "component": "i.g.n.s.i.g.n.N.connections", "cluster.name": "cluster", "node.name": "cluster-data-0", "message": "Transport failed", "cluster.uuid": "3YsliDkFRQWJ1unjPy6KWQ", "node.id": "Nlcc3FeST8qa5CwpIPHrPw" ,
"stacktrace": ["io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2Exception: HTTP/2 client preface string missing or corrupt. Hex dump for received bytes: 16030100fc010000f8030362236b3138a7b269b4e83e6c1b",
"at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2Exception.connectionError(Http2Exception.java:107) ~[grpc-netty-shaded-1.75.0.jar:1.75.0]",
"at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.readClientPrefaceString(Http2ConnectionHandler.java:319) ~[grpc-netty-shaded-1.75.0.jar:1.75.0]",
"at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.decode(Http2ConnectionHandler.java:245) ~[grpc-netty-shaded-1.75.0.jar:1.75.0]",
"at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:451) ~[grpc-netty-shaded-1.75.0.jar:1.75.0]",
"at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530) ~[grpc-netty-shaded-1.75.0.jar:1.75.0]",
"at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469) ~[grpc-netty-shaded-1.75.0.jar:1.75.0]",
"at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[grpc-netty-shaded-1.75.0.jar:1.75.0]",
"at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[grpc-netty-shaded-1.75.0.jar:1.75.0]",
"at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[grpc-netty-shaded-1.75.0.jar:1.75.0]",
"at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[grpc-netty-shaded-1.75.0.jar:1.75.0]",
"at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357) ~[grpc-netty-shaded-1.75.0.jar:1.75.0]",
"at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[grpc-netty-shaded-1.75.0.jar:1.75.0]",
"at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[grpc-netty-shaded-1.75.0.jar:1.75.0]",
"at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868) ~[grpc-netty-shaded-1.75.0.jar:1.75.0]",
"at io.grpc.netty.shaded.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[grpc-netty-shaded-1.75.0.jar:1.75.0]",
"at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:796) [grpc-netty-shaded-1.75.0.jar:1.75.0]",
"at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:732) [grpc-netty-shaded-1.75.0.jar:1.75.0]",
"at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:658) [grpc-netty-shaded-1.75.0.jar:1.75.0]",
"at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [grpc-netty-shaded-1.75.0.jar:1.75.0]",
"at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998) [grpc-netty-shaded-1.75.0.jar:1.75.0]",
"at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [grpc-netty-shaded-1.75.0.jar:1.75.0]",
"at java.base/java.lang.Thread.run(Thread.java:1474) [?:?]"] }

pablo · October 29, 2026, 12:15am

@SlavaUtesinov, how many nodes are in your cluster?
I’ve tested with OpenSearch Operator 2.8.0, but I couldn’t reproduce your messages
Did you try OpenSearch Operator 3.0.0?

Could you share your full OpenSearchCluster manifest?

SlavaUtesinov · October 29, 2026, 3:46am

Hi, @pablo, I added full manifest in the post

pablo · October 29, 2026, 2:07pm

@SlavaUtesinov, thanks for sharing. I’ve checked your deployment, and I couldn’t find anything that would cause the reported message.

My logs are silent when I run a gRPC search query. I used a Python client to test it.

import grpc

from opensearch.protobufs.schemas import *
from opensearch.protobufs.services import SearchServiceStub


channel = grpc.insecure_channel(
    target="localhost:30753",
)

search_stub = SearchServiceStub(channel)

# Create a term query
term_query = TermQuery(
    field="field",
    value=FieldValue(string="value")
)
query_container = QueryContainer(term=term_query)

# Create a search request
request = SearchRequest(
    search_request_body=SearchRequestBody(query=query_container),
    index=["my-index"]
)

# Send request and handle response
try:
    response = search_stub.Search(request=request)
    if response.hits:
        print("Found {} hits".format(response.hits.total))
        print(response.hits)
    elif response.timed_out or response.terminated_early:
        print("Request timed out or terminated early")
    elif response.x_shards.failed:
        print("Some shards failed to execute the search")
        print(response.x_shards.failures)
except grpc.RpcError as e:
    if e.code() == StatusCode.UNAVAILABLE:
        print("Failed to reach server: {}".format(e))
    elif e.code() == StatusCode.PERMISSION_DENIED:
        print("Permission denied: {}".format(e))
    elif e.code() == StatusCode.INVALID_ARGUMENT:
        print("Invalid argument: {}".format(e))
finally:
    channel.close()

I’ve also noticed that you’re mapping log4j2.properties. Did you enable any additional logging for gRPC? Could you share that file?

pablo · October 29, 2026, 2:49pm

@SlavaUtesinov, how do you run your gRPC queries? Could you share your client?
Also, is there anything else running gRPC queries at the same time?

Did you check the timestamps of those messages? Is there any pattern?

SlavaUtesinov · October 30, 2026, 3:52am

@pablo It seems I found the problem - I used Postman and tried to invoke grpc request with and without TLS. TLS caused the first log then It seems Postman stuck sending request silently that is why logs keep appear. Thank you for help.

Topic		Replies	Views
OpenSearchException: Transport client authentication no longer supported Security troubleshoot , configure	2	2768	November 9, 2022
"Transport client authentication no longer supported" in between-node transport layer communication Security troubleshoot , configure	6	1695	December 13, 2023
Opensearch Dashboards - Getting connection error with opensearch message":"[ConnectionError]: socket hang up"} after enabling SSL Security	8	2119	June 21, 2024
OpenSearch 2.01 => Docker Shard failed after some days OpenSearch troubleshoot	2	493	July 21, 2022
SSL Error unable to configure Opensearch Cluster in 6 VMs Security troubleshoot , configure	14	1823	March 8, 2023

io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2Exception

Related topics