Describe the issue:Recently, we upgraded our OpenSearch version to 3.3.1.
During the upgrade from 2.11.0 to 3.3.1, the upgrade process completes successfully, but we are observing pod restarts caused by the following error:
Error opening zip file or JAR manifest missing: agent/opensearch-agent.jar
The existing ingest pod (version 2.11.1) is already running in a 3/3 Ready state.
When we trigger the upgrade, a new ingest pod (version 3.1.1) is created.
It usually takes around 3–4 minutes for the new pod to reach the 3/3 Ready state.
During this time, the old 2.11.1 ingest pod continues running in 3/3 Ready state, so there is no service impact.
Once the new 3.1.1 pod becomes healthy, the old 2.11.1 pod is terminated, traffic switches seamlessly to the new pod, and the upgrade completes normally.
However, in the problematic case:
While the new ingest 3.1.1 pod is still initializing (still in its 3–4 mins to start),
The old ingest 2.11.1 pod unexpectedly restarts.
This results in no active ingest pod during that period(Zero downtime exists).
Because of this, there is traffic loss, and we observed the mentioned error in the logs of the restarted pod.
My question is: why are the older version pods restarting in the first place?
Configuration: provided 3 ingest 3 master and 3 data pods
Hi @pablo , Actually, regarding the same query — the failing agent JAR is only a suspected cause. The main point is that the OpenSearch upgrade from 2.11.0 to 3.1.1 is not a supported upgrade path. Because of this, during the upgrade the cluster tries to resync, which leads to downtime.
To avoid this downtime, do we have any possible workaround during the upgrade process?
Hi @pablo , as @AshokPonna mentioned earlier, this is only a suspicion — the main focus is the OpenSearch upgrade from 2.11.0 to 3.3.1.
To answer your question: we extract the OpenSearch tar file and run it as a binary inside our Kubernetes pods. During the upgrade, we replace the existing pod with a new Kubernetes pod that contains the updated OpenSearch folder (version 3.3.1).
@Badamkoushil@AshokPonna What do you mean that 2.11.0 to 3.3.1 is not supported?
According to OpenSearch documentation, upgrade between major versions i.e. 2.x and 3.x or 1.x and 2.x is supported.
Regarding the upgrade process. @Badamkoushil This looks very manual and I understand that you don’t use official OpenSearch Helm Charts or OpenSearch Operator.
Could you describe your deployment in more details? How many nodes do you have?
Could you share output of this command?
GET _cat/nodes
OpenSearch pods only
kubectl get pods
Did you use deployment or stateful sets to create OpenSearch cluster?
Hi @pablo , Thanks for your response, but when we are trying to upgrade from 2.11.1 to 3.3.1, upgrade is working but with some downtime, between this time we could see error logs as below. {“extra_data”:{“class”:“c.e.b.a.s.o.p.s.t.TlsTransport”},“message”:“Peer not connected [channel: Netty4TcpChannel{localAddress=/192.168.161.67:9300, remoteAddress=/192.168.52.103:50358}] Cause: Received message from unsupported version: [2.11.1] minimal compatible version is: [2.19.0]”,“metadata”:{“container_name”:“master”,“namespace”:“ashok”,“pod_name”:“opensearch-master-2”},“service_id”:“opensearch”,“severity”:“warning”,“timestamp”:“2025-11-20T12:15:45.180+00:00”,“version”:“1.2.0”} {“extra_data”:{“class”:“c.e.b.a.s.o.p.s.t.TlsTransport”},“message”:“Exception caught onException TlsTransport cause : Peer not connected [channel: Netty4TcpChannel{localAddress=/192.168.161.67:9300, remoteAddress=/192.168.52.103:50358}] Cause: Received message from unsupported version: [2.11.1] minimal compatible version is: [2.19.0]”,“metadata”:{“container_name”:“master”,“namespace”:“ashok”,“pod_name”:“opensearch-master-2”},“service_id”:“opensearch”,“severity”:“warning”,“timestamp”:“2025-11-20T12:15:45.180+00:00”,“version”:“1.2.0”} {“extra_data”:{“class”:“c.e.b.a.s.o.p.s.t.TlsTransport”},“message”:“Peer not connected [channel: Netty4TcpChannel{localAddress=/[fc00:1000:0:0:0:0:0:a144]:9300, remoteAddress=/[fc00:1000:0:0:0:0:0:3444]:48842}] Cause: Received message from unsupported version: [2.11.1] minimal compatible version is: [2.19.0]”,“metadata”:{“container_name”:“master”,“namespace”:“ashok”,“pod_name”:“opensearch-master-2”},“service_id”:“opensearch”,“severity”:“warning”,“timestamp”:“2025-11-20T12:15:45.181+00:00”,“version”:“1.2.0”} {“extra_data”:{“class”:“c.e.b.a.s.o.p.s.t.TlsTransport”},“message”:“Exception caught onException TlsTransport cause : Peer not connected [channel: Netty4TcpChannel{localAddress=/[fc00:1000:0:0:0:0:0:a144]:9300, remoteAddress=/[fc00:1000:0:0:0:0:0:3444]:48842}] Cause: Received message from unsupported version: [2.11.1] minimal compatible version is: [2.19.0]”,“metadata”:{“container_name”:“master”,“namespace”:“ashok”,“pod_name”:“opensearch-master-2”},“service_id”:“opensearch”,“severity”:“warning”,“timestamp”:“2025-11-20T12:15:45.182+00:00”,“version”:“1.2.0”}