Shards appear to have negative size, and block listing them

Camusensei · September 16, 2026, 9:10am

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser): 3.8.0

Describe the issue: Some shards appear to report a negative size. It’s impossible to list the shards because of this.

Configuration: Read/write separated (remote storage is minio), currently 20 data nodes and 5 search nodes. 5 data nodes also are cluster_managers. The index has ZERO REPLICAS.

Relevant Logs or Screenshots:

The problem is that I cannot list shards for the affected index:

$ curl -ksSu "$creds" https://localhost:9200/_list/shards/big_index
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Values less than -1 bytes are not supported: -505064b"}],"type":"illegal_argument_exception","reason":"Values less than -1 bytes are not supported: -505064b"},"status":400}

There is more than one shard with this issue because I also saw these values in the error message while I was adding data nodes:
-943336b
-1001992b
-493792b

From what I can see, everything is working except listing the shards.

How do I fix this? Is there a way to still get the shards listed despite this bug?
I know about cloning my index and deleting the old one (what’s happening now but it will a full day), is there any other way to fix this?

I don’t know when the problem happened, but since the cluster and the index were created, all the operations that were done were:

restart one data node
cut internet for 6 minutes on one data node
restart search nodes
transform 10 search nodes (from 15) to data nodes, one by one

I’ll have to redo all this if you need me to try to reproduce.

pablo · September 24, 2026, 1:21pm

@Camusensei Can you list shards related to the affected indices? If yes, are they all located on the same node?
Do you see any errors regarding affected indices or it’s shards or disk space watermarks?
Do you monitor disk space on all data nodes?

Camusensei · September 25, 2026, 9:22am

The error only comes if I try to list all shards or just the affected index’s shards (that’s how I found which index was affected, by trial and error on all indexes until I got the error)
I do monitor disk space on all data nodes, I had no space issue <25% for all data nodes

When I wrote the ticket, my situation cluster looked like this:

20 data (contain the indexes)
5 search (contain search replicas)

After writing this ticket, I excluded the first 10 data nodes, effectively moving all the shards from 20 nodes to the second set of 10 nodes:

10 data (empty)
10 data (contain the indexes)
5 search (contain search replicas)

I reinstalled those 10 empty nodes (that was the operation I needed to do). Then I moved the data back to the first 10 nodes by excluding the second set of 10 nodes:

10 data (contain the indexes)
10 data (empty)
5 search (contain search replicas)

Finally I changed the roles of the empty data nodes back to search nodes:

10 data (contain the indexes)
10 search (empty)
5 search (contain search replicas)

Now I can see that the issue is gone (I am able to list the shards again).
(I also see that I forgot to remove the exclusion which explains why 10 of my search nodes were empty)
Unfortunately I do not know at which step the problem went away as I completely forgot about the issue while I was performing my operation.

But if it happens again, it would be nice to have the code still list the shards that have a positive size despite showing the error… Which would help troubleshoot which shards have a negative size and why.

pablo · September 28, 2026, 7:11pm

@Camusensei, I’m not aware of a script or option that would list indices with negative values.
However, if this happens again, I suggest finding the affected nodes first.
Try these commands:

Get IDs of all nodes

GET /_cat/nodes?v&full_id=true&h=id,name,ip

Get the shards routing table of the affected indices.

GET /_cluster/state/routing_table/security-auditlog-2026.09.28

{
  "cluster_name": "my-first-cluster",
  "cluster_uuid": "Zq0-w99WQ6CqfN1DU4Nfzg",
  "routing_table": {
    "indices": {
      "security-auditlog-2026.09.28": {
        "shards": {
          "0": [
            {
              "state": "STARTED",
              "primary": true,
              "node": "G0WlcPSoTPuBRMfWg0wORg",
              "relocating_node": null,
              "shard": 0,
              "index": "security-auditlog-2026.09.28",
              "allocation_id": {
                "id": "MQNOdUrfSI2fW8We-f5ydA"
              }
            },
            {
              "state": "STARTED",
              "primary": false,
              "node": "RP3CRvDCTMug685-XGFVgQ",
              "relocating_node": null,
              "shard": 0,
              "index": "security-auditlog-2026.09.28",
              "allocation_id": {
                "id": "hpGPuWPuTzWgc5Dnv0qxGQ"
              }
            }
          ]
        }
      }
    }
  }
}

Check the logs of the affected nodes for any related issues.

Did you have any power outages or hard restarts on any of the nodes? How did you deploy your OS cluster - public or private cloud, Docker, Kubernetes or Helm charts?

Camusensei · October 14, 2026, 10:38am

Sorry for the delay.
Ok, I’ll see if I can get the routing table next time, thank you.

Regarding power outages or hard restarts, yes, I did, as I explained:
1 service restart on a data node, 1 network cut for 6 minutes on a data node.

My cluster is on 25 private bare metal servers. Opensearch is installed as a service on each machine through chef.

Camusensei · October 15, 2026, 6:34am

Well, it happened again on another cluster, this time after just reducing the replication for one index through opensearch-dashboard… Sporadic issue for sure.

And now even _cat/nodes returns the error from some machines!
Elected master (node-5) is replying with an error all the time. Other nodes sometimes reply with errors. (nodes-2) and node-6 almost always replies with errors.
When I _cat/nodes from node-1, I can see the nodes 5 and 6 without any metrics and sometimes node-2 as well.

`{“error”:{“root_cause”:[{“type”:“illegal_argument_exception”,“reason”:“Values less than -1 bytes are not supported: -61864b”}],“type”:“illegal_argument_exception”,“reason”:“Values less than -1 bytes are not supported: -61864b”},“status”:400}`

(The value is changing almost on every call: -126312b -61240b -224b -126568b -65000b, …)

The impacted index is still my big index, and according to cluster routing, it has one shard per node (3 primaries and 3 replicas)

Camusensei · October 15, 2026, 6:58am

This is the kind of error logs I get on the nodes:

[2026-10-15T06:45:39,000][WARN ][o.o.t.OutboundHandler ] [node-6] send message failed [channel: Netty4TcpChannel{localAddress=/node-6-ip:9300, remoteAddress=/node-1-ip:35150}] java.lang.IllegalStateException: Negative longs unsupported, use writeLong or writeZLong for negative numbers [-3528]

And on other (working) nodes:

[2026-10-15T06:45:39,005][WARN ][r.suppressed             ] [node-1] path: /_prometheus/metrics, params: {} java.lang.NullPointerException: Cannot invoke "org.opensearch.index.shard.DocsStats.getCount()" because the return value of "org.opensearch.action.admin.indices.stats.CommonStats.getDocs()" is null

Restarting the affected nodes (the ones that were sporadically “blank” in _cat/nodes) fixed the negative number issue on them.
Note that while they were blank, they themselves show the error on some endpoints (e.g. _cat/nodes and _list/shards) and while they are back in the list, they work again:

On the other hand, some other endpoints like _cluster/state/routing_table always work.

Oh, full trace in case it helps:

[2026-10-15T06:56:43,607][WARN ][o.o.t.OutboundHandler    ] [node-6] send message failed [channel: Netty4TcpChannel{localAddress=/node-6-ip:9300, remoteAddress=/node-3-ip:57956}]
java.lang.IllegalStateException: Negative longs unsupported, use writeLong or writeZLong for negative numbers [-56184]
        at org.opensearch.core.common.io.stream.StreamOutput.writeVLong(StreamOutput.java:310)
        at org.opensearch.index.fielddata.FieldDataStats.writeTo(FieldDataStats.java:167)
        at org.opensearch.core.common.io.stream.StreamOutput.lambda$writeOptionalWriteable$0(StreamOutput.java:974)
        at org.opensearch.core.common.io.stream.StreamOutput.writeOptionalWriteable(StreamOutput.java:980)
        at org.opensearch.core.common.io.stream.StreamOutput.writeOptionalWriteable(StreamOutput.java:974)
        at org.opensearch.action.admin.indices.stats.CommonStats.writeTo(CommonStats.java:281)
        at org.opensearch.action.admin.indices.stats.ShardStats.writeTo(ShardStats.java:242)
        at org.opensearch.core.common.io.stream.StreamOutput.lambda$writeOptionalWriteable$0(StreamOutput.java:974)
        at org.opensearch.core.common.io.stream.StreamOutput.writeOptionalWriteable(StreamOutput.java:980)
        at org.opensearch.core.common.io.stream.StreamOutput.writeOptionalWriteable(StreamOutput.java:974)
        at org.opensearch.action.support.broadcast.node.TransportBroadcastByNodeAction$NodeResponse.writeTo(TransportBroadcastByNodeAction.java:670)
        at org.opensearch.transport.nativeprotocol.NativeOutboundMessage.writeMessage(NativeOutboundMessage.java:106)
        at org.opensearch.transport.nativeprotocol.NativeOutboundMessage.serialize(NativeOutboundMessage.java:84)
        at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:218)
        at org.opensearch.transport.nativeprotocol.NativeOutboundHandler$MessageSerializer.get(NativeOutboundHandler.java:204)
        at org.opensearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:139)
        at org.opensearch.transport.OutboundHandler.sendBytes(OutboundHandler.java:80)
        at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendMessage(NativeOutboundHandler.java:187)
        at org.opensearch.transport.nativeprotocol.NativeOutboundHandler.sendResponse(NativeOutboundHandler.java:152)
        at org.opensearch.transport.TcpTransportChannel.sendResponse(TcpTransportChannel.java:96)
        at org.opensearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:72)
        at org.opensearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:501)
        at org.opensearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:473)
        at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceivedDecorate(SecuritySSLRequestHandler.java:186)
        at org.opensearch.security.transport.SecurityRequestHandler.messageReceivedDecorate(SecurityRequestHandler.java:340)
        at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceived(SecuritySSLRequestHandler.java:148)
        at org.opensearch.security.OpenSearchSecurityPlugin$4$1.messageReceived(OpenSearchSecurityPlugin.java:943)
        at org.opensearch.indexmanagement.rollup.interceptor.RollupInterceptor$interceptHandler$1.messageReceived(RollupInterceptor.kt:189)
        at org.opensearch.performanceanalyzer.transport.PerformanceAnalyzerTransportRequestHandler.messageReceived(PerformanceAnalyzerTransportRequestHandler.java:44)
        at org.opensearch.performanceanalyzer.transport.RTFPerformanceAnalyzerTransportRequestHandler.messageReceived(RTFPerformanceAnalyzerTransportRequestHandler.java:92)
        at org.opensearch.wlm.WorkloadManagementTransportInterceptor$RequestHandler.messageReceived(WorkloadManagementTransportInterceptor.java:63)
        at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:108)
        at org.opensearch.transport.NativeMessageHandler$RequestHandler.doRun(NativeMessageHandler.java:528)
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1011)
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
        at java.base/java.lang.Thread.run(Thread.java:1474)

Camusensei · October 15, 2026, 8:16am

https://github.com/opensearch-project/OpenSearch/issues/20363 seems to be related, I’m reading it now

Topic		Replies	Views
Shard allocation failure due to negative free space OpenSearch	3	553	October 9, 2024
Shrink action says "There are no available nodes to move to to execute a shrink" but there are qualifying nodes Index Management	6	915	September 8, 2022
Error: Shards Failed OpenSearch Dashboards	3	799	August 5, 2025
Multi index issue with shards Open Source Elasticsearch and Kibana	5	1075	September 10, 2021
Shard node limit OpenSearch	2	553	February 3, 2025

Shards appear to have negative size, and block listing them

Related topics