# Upgrade from 2.11.0 to 3.3.1 failing agent/opensearch-agent.jar **URL:** \ **Category:** OpenSearch\ **Created:** [November 20, 2025, 6:53am UTC](https://forum.opensearch.org/t/upgrade-from-2-11-0-to-3-3-1-failing-agent-opensearch-agent-jar/27515 "2025-11-20T06:53:50Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Badamkoushil](https://avatars.discourse-cdn.com/v4/letter/b/eb9ed0/32.png) [@Badamkoushil](https://forum.opensearch.org/u/Badamkoushil)\ **Post date:** [November 20, 2025, 6:53am UTC](https://forum.opensearch.org/t/upgrade-from-2-11-0-to-3-3-1-failing-agent-opensearch-agent-jar/27515/1 "2025-11-20T06:53:51Z")
**Versions** (relevant - OpenSearch/Dashboard/Server OS/Browser):3.3.1 **Describe the issue** :Recently, we upgraded our OpenSearch version to **3.3.1**. During the upgrade from **2.11.0 to 3.3.1** , the upgrade process completes successfully, but we are observing **pod restarts** caused by the following error: ```auto Error opening zip file or JAR manifest missing: agent/opensearch-agent.jar ``` - The existing ingest pod (version **2.11.1** ) is already running in a **3/3 Ready** state. - When we trigger the upgrade, a new ingest pod (version **3.1.1** ) is created. - It usually takes around **3–4 minutes** for the new pod to reach the **3/3 Ready** state. - During this time, the old **2.11.1** ingest pod continues running in **3/3 Ready** state, so there is no service impact. - Once the new **3.1.1** pod becomes healthy, the old **2.11.1** pod is terminated, traffic switches seamlessly to the new pod, and the upgrade completes normally. However, in the problematic case: - While the new **ingest 3.1.1** pod is still initializing (still in its 3–4 mins to start), - The old **ingest 2.11.1** pod unexpectedly restarts. - This results in **no active ingest pod** during that period(Zero downtime exists). - Because of this, there is **traffic loss** , and we observed the mentioned error in the logs of the restarted pod. My question is: **why are the older version pods restarting in the first place?** **Configuration** : provided 3 ingest 3 master and 3 data pods **Relevant Logs or Screenshots** : ---
**Author:** ![pablo](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/pablo/32/10363_2.png) [@pablo](https://forum.opensearch.org/u/pablo)\ **Post date:** [November 20, 2025, 1:33pm UTC](https://forum.opensearch.org/t/upgrade-from-2-11-0-to-3-3-1-failing-agent-opensearch-agent-jar/27515/2 "2025-11-20T13:33:16Z")
@Badamkoushil How did you deploy/upgraded the cluster? Did you use helm charts? If so, could you share your values.yml file? ---
**Author:** ![AshokPonna](https://avatars.discourse-cdn.com/v4/letter/a/59ef9b/32.png) [@AshokPonna](https://forum.opensearch.org/u/AshokPonna)\ **Post date:** [November 21, 2025, 4:33am UTC](https://forum.opensearch.org/t/upgrade-from-2-11-0-to-3-3-1-failing-agent-opensearch-agent-jar/27515/3 "2025-11-21T04:33:27Z")
Hi @pablo , Actually, regarding the same query — the failing agent JAR is only a suspected cause. The main point is that the OpenSearch upgrade from **2.11.0 to 3.1.1 is not a supported upgrade path**. Because of this, during the upgrade the cluster tries to resync, which leads to downtime. To avoid this downtime, do we have any possible workaround during the upgrade process? TIA/Ashok ---
**Author:** ![Badamkoushil](https://avatars.discourse-cdn.com/v4/letter/b/eb9ed0/32.png) [@Badamkoushil](https://forum.opensearch.org/u/Badamkoushil)\ **Post date:** [November 24, 2025, 1:18pm UTC](https://forum.opensearch.org/t/upgrade-from-2-11-0-to-3-3-1-failing-agent-opensearch-agent-jar/27515/4 "2025-11-24T13:18:11Z")
Hi @pablo , as @AshokPonna mentioned earlier, this is only a suspicion — the main focus is the OpenSearch upgrade from **2.11.0 to 3.3.1**. To answer your question: we extract the OpenSearch tar file and run it as a binary inside our Kubernetes pods. During the upgrade, we replace the existing pod with a new Kubernetes pod that contains the updated OpenSearch folder (version **3.3.1** ). Regards, Koushil ---
**Author:** ![pablo](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/pablo/32/10363_2.png) [@pablo](https://forum.opensearch.org/u/pablo)\ **Post date:** [November 24, 2025, 5:27pm UTC](https://forum.opensearch.org/t/upgrade-from-2-11-0-to-3-3-1-failing-agent-opensearch-agent-jar/27515/5 "2025-11-24T17:27:11Z")
@Badamkoushil @AshokPonna What do you mean that 2.11.0 to 3.3.1 is not supported? According to OpenSearch documentation, upgrade between major versions i.e. 2.x and 3.x or 1.x and 2.x is supported. > **[Rolling upgrade](https://docs.opensearch.org/latest/migrate-or-upgrade/rolling-upgrade/#preparing-to-upgrade)** > > Rolling upgrade Regarding the upgrade process. @Badamkoushil This looks very manual and I understand that you don’t use official OpenSearch Helm Charts or OpenSearch Operator. Could you describe your deployment in more details? How many nodes do you have? Could you share output of this command? ```auto GET _cat/nodes ``` OpenSearch pods only ```auto kubectl get pods ``` Did you use deployment or stateful sets to create OpenSearch cluster? ---
**Author:** ![AshokPonna](https://avatars.discourse-cdn.com/v4/letter/a/59ef9b/32.png) [@AshokPonna](https://forum.opensearch.org/u/AshokPonna)\ **Post date:** [November 25, 2025, 5:44am UTC](https://forum.opensearch.org/t/upgrade-from-2-11-0-to-3-3-1-failing-agent-opensearch-agent-jar/27515/6 "2025-11-25T05:44:18Z")
Hi @pablo , Thanks for your response, but when we are trying to upgrade from 2.11.1 to 3.3.1, upgrade is working but with some downtime, between this time we could see error logs as below. `{“extra_data”:{“class”:“c.e.b.a.s.o.p.s.t.TlsTransport”},“message”:“Peer not connected [channel: Netty4TcpChannel{localAddress=/192.168.161.67:9300, remoteAddress=/192.168.52.103:50358}] Cause: Received message from unsupported version: [2.11.1] minimal compatible version is: [2.19.0]”,“metadata”:{“container_name”:“master”,“namespace”:“ashok”,“pod_name”:“opensearch-master-2”},“service_id”:“opensearch”,“severity”:“warning”,“timestamp”:“2025-11-20T12:15:45.180+00:00”,“version”:“1.2.0”}` `{“extra_data”:{“class”:“c.e.b.a.s.o.p.s.t.TlsTransport”},“message”:“Exception caught onException TlsTransport cause : Peer not connected [channel: Netty4TcpChannel{localAddress=/192.168.161.67:9300, remoteAddress=/192.168.52.103:50358}] Cause: Received message from unsupported version: [2.11.1] minimal compatible version is: [2.19.0]”,“metadata”:{“container_name”:“master”,“namespace”:“ashok”,“pod_name”:“opensearch-master-2”},“service_id”:“opensearch”,“severity”:“warning”,“timestamp”:“2025-11-20T12:15:45.180+00:00”,“version”:“1.2.0”}` `{“extra_data”:{“class”:“c.e.b.a.s.o.p.s.t.TlsTransport”},“message”:“Peer not connected [channel: Netty4TcpChannel{localAddress=/[fc00:1000:0:0:0:0:0:a144]:9300, remoteAddress=/[fc00:1000:0:0:0:0:0:3444]:48842}] Cause: Received message from unsupported version: [2.11.1] minimal compatible version is: [2.19.0]”,“metadata”:{“container_name”:“master”,“namespace”:“ashok”,“pod_name”:“opensearch-master-2”},“service_id”:“opensearch”,“severity”:“warning”,“timestamp”:“2025-11-20T12:15:45.181+00:00”,“version”:“1.2.0”}` `{“extra_data”:{“class”:“c.e.b.a.s.o.p.s.t.TlsTransport”},“message”:“Exception caught onException TlsTransport cause : Peer not connected [channel: Netty4TcpChannel{localAddress=/[fc00:1000:0:0:0:0:0:a144]:9300, remoteAddress=/[fc00:1000:0:0:0:0:0:3444]:48842}] Cause: Received message from unsupported version: [2.11.1] minimal compatible version is: [2.19.0]”,“metadata”:{“container_name”:“master”,“namespace”:“ashok”,“pod_name”:“opensearch-master-2”},“service_id”:“opensearch”,“severity”:“warning”,“timestamp”:“2025-11-20T12:15:45.182+00:00”,“version”:“1.2.0”}` BR/Ashok ---
**Author:** ![pablo](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/pablo/32/10363_2.png) [@pablo](https://forum.opensearch.org/u/pablo)\ **Post date:** [November 25, 2025, 10:15am UTC](https://forum.opensearch.org/t/upgrade-from-2-11-0-to-3-3-1-failing-agent-opensearch-agent-jar/27515/7 "2025-11-25T10:15:51Z")
@AshokPonna This is a statement from the AWS OpenSearch managed service > If you’re running OpenSearch 1.3 or 2.x, you must first upgrade to OpenSearch 2.19 _before_ upgrading to OpenSearch 3.1. > **[Upgrading Amazon OpenSearch Service domains - Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/version-migration.html)** > > Learn how to upgrade an Amazon OpenSearch Service domain to a newer OpenSearch or Elasticsearch version. This confirms your finding. I’ve raised a GitHub issue to update the documentation. > > > Added note about upgrading to OpenSearch 2.19 before 3.x. > > \### Description > Cu…rrent OpenSearch documentation doesn't specify a minimum 2.x version for a successful upgrade to 3.x. > > Upgrade from pre-2.19 to 3.x will fail with the following error. > > \`\`\` > {“extra\_data”:{“class”:“c.e.b.a.s.o.p.s.t.TlsTransport”},“message”:“Peer not connected \[channel: Netty4TcpChannel{localAddress=/192.168.161.67:9300, remoteAddress=/192.168.52.103:50358}\] Cause: Received message from unsupported version: \[2.11.1\] minimal compatible version is: \[2.19.0\]”,“metadata”:{“container\_name”:“master”,“namespace”:“ashok”,“pod\_name”:“opensearch-master-2”},“service\_id”:“opensearch”,“severity”:“warning”,“timestamp”:“2025-11-20T12:15:45.180+00:00”,“version”:“1.2.0”} > \`\`\` > > \### Version > OpenSearch 3.x > > \### Checklist > \- \[\] By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the \[Developers Certificate of Origin\](https://github.com/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#developer-certificate-of-origin). > For more information on following Developer Certificate of Origin and signing off your commits, please check \[here\](https://github.com/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#developer-certificate-of-origin).