System.OutOfMemoryException

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
v 2.17.0

Describe the issue:
I am trying to read the documents from opensearch, where i am keep getting Out Of Memory Exception.

Here is my code, where i am trying to read 1 million record at a time, but it’s supporting upto 200konly.

public async Task<IEnumerable<_doc>> ReadAllDocumentsFromElasticSearch(int periodId, CancellationToken cancellationToken = default)
{
    var elasticIndex = new ElasticIndex(periodId);
    var scrollTimeout = TimeSpan.FromSeconds(await _settingRepository.GetValueOrDefaultAsync(
           SettingTypes.ScrollKeepAliveInSeconds, ElasticConstants.DefaultScrollKeepAliveInSeconds));
    var batchSize = DocumentStoreConstants.MaxSearchSize;

    var searchRequest = new SearchRequest(elasticIndex.DataIndex)
    {
        Scroll = scrollTimeout,
        Size = batchSize,
        Query = new MatchAllQuery()
    };

    var documents = new List<_doc>();
    var response = await _elasticClient.SearchAsync<_doc>(searchRequest);
    var scrollId = response.ScrollId;

    while (response.Documents.Any())
    {
        documents.AddRange(response.Documents);

        var scrollRequest = new ScrollRequest(scrollId, scrollTimeout);
        response = await _elasticClient.ScrollAsync<_doc>(scrollRequest);
        scrollId = response.ScrollId;
    }

    var clearScrollRequest = new ClearScrollRequest(scrollId);
    await _elasticClient.ClearScrollAsync(clearScrollRequest);

    return documents;
}

Configuration:

var connectionSettings = new ConnectionSettings(pool, connection,
sourceSerializer: (builtin, settings) =>
new JsonNetSerializer(builtin, settings, () =>
    new JsonSerializerSettings { NullValueHandling = NullValueHandling.Include },
        resolver => resolver.NamingStrategy = new CamelCaseNamingStrategy()));

connectionSettings.RequestTimeout(new TimeSpan(1, 0, 0));
connectionSettings.ThrowExceptions();

Relevant Logs or Screenshots:

OpenSearch.Net.UnexpectedOpenSearchClientException: Exception of type 'System.OutOfMemoryException' was thrown.
 ---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Array.Resize[T](T[]& array, Int32 newSize)   at OpenSearch.Net.Utf8Json.Internal.ArrayBuffer`1.Add(T value)   at OpenSearch.Net.Utf8Json.Formatters.CollectionFormatterBase`4.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver)   at Deserialize(Object[], JsonReader&, IJsonFormatterResolver)   at OpenSearch.Client.ReadAsFormatter`2.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver)   at Deserialize(Object[], JsonReader&, IJsonFormatterResolver)   at OpenSearch.Net.Utf8Json.JsonSerializer.Deserialize[T](Byte[] bytes, Int32 offset, IJsonFormatterResolver resolver)   at OpenSearch.Net.Utf8Json.JsonSerializer.DeserializeAsync[T](Stream stream, IJsonFormatterResolver resolver)   at OpenSearch.Net.ResponseBuilder.SetBodyAsync[TResponse](ApiCallDetails details, RequestData requestData, Stream responseStream, String mimeType, CancellationToken cancellationToken)   at OpenSearch.Net.ResponseBuilder.ToResponseAsync[TResponse](RequestData requestData, Exception ex, Nullable`1 statusCode, IEnumerable`1 warnings, Stream responseStream, String mimeType, CancellationToken cancellationToken)   at OpenSearch.Net.HttpConnection.RequestAsync[TResponse](RequestData requestData, CancellationToken cancellationToken)   at OpenSearch.Net.RequestPipeline.CallOpenSearchAsync[TResponse](RequestData requestData, CancellationToken cancellationToken)   at OpenSearch.Net.Transport`1.RequestAsync[TResponse](HttpMethod method, String path, CancellationToken cancellationToken, PostData data, IRequestParameters requestParameters)
   --- End of inner exception stack trace ---

Any update on this?

Hey @prakashapr27

What is your spec’s on the server? I Know awhile back there was a issue of this /bug but I think they resolved it not 100% sure. Also what have you tried to resolve this issue?

Hi @Gsmitt,

We are trying to find the contra between the documents, say for eg

If document A is 10 and B is -10 then this should be considered as contra document. Previously we were doing this with open search aggregation. But for larger files 1M and above documents it’s taking longer than expected. So we rewritten entire logic by c# code. But for this we need entire documents to be read and process through loops.

This is our case for reading all the records it’s throwing out of memory exception.

@prakashapr27

Ah ok, I assume you do have enough memory on the node. Have you tried to increase you heap size?

@Gsmitt

I have 2 nodes with 16 GB of heap size

Please see the below details

{
  "_nodes" : {
    "total" : 2,
    "successful" : 2,
    "failed" : 0
  },
  "cluster_name" : "035512119119:k-analyser-dev-es",
  "nodes" : {
    "7DVCU6X_QzCaX4EV5e-6TA" : {
      "timestamp" : 1744636955405,
      "name" : "719b6f63a75e3b0a1cab14551a315609",
      "roles" : [ "data", "ingest", "master", "remote_cluster_client" ],
      "jvm" : {
        "timestamp" : 1744636955405,
        "uptime_in_millis" : 1118603896,
        "mem" : {
          "heap_used_in_bytes" : 2261157536,
          "heap_used_percent" : 26,
          "heap_committed_in_bytes" : 8589934592,
          "heap_max_in_bytes" : 8589934592,
          "non_heap_used_in_bytes" : 453835064,
          "non_heap_committed_in_bytes" : 501022720,
          "pools" : {
            "young" : {
              "used_in_bytes" : 1509949440,
              "max_in_bytes" : 0,
              "peak_used_in_bytes" : 5146411008,
              "peak_max_in_bytes" : 0,
              "last_gc_stats" : {
                "used_in_bytes" : 0,
                "max_in_bytes" : 0,
                "usage_percent" : -1
              }
            },
            "old" : {
              "used_in_bytes" : 747796928,
              "max_in_bytes" : 8589934592,
              "peak_used_in_bytes" : 6095274648,
              "peak_max_in_bytes" : 8589934592,
              "last_gc_stats" : {
                "used_in_bytes" : 678429584,
                "max_in_bytes" : 8589934592,
                "usage_percent" : 7
              }
            },
            "survivor" : {
              "used_in_bytes" : 3411168,
              "max_in_bytes" : 0,
              "peak_used_in_bytes" : 144703488,
              "peak_max_in_bytes" : 0,
              "last_gc_stats" : {
                "used_in_bytes" : 3411168,
                "max_in_bytes" : 0,
                "usage_percent" : -1
              }
            }
          }
        },
        "threads" : {
          "count" : 543,
          "peak_count" : 543
        },
        "gc" : {
          "collectors" : {
            "young" : {
              "collection_count" : 1955,
              "collection_time_in_millis" : 44026
            },
            "G1 Concurrent GC" : {
              "collection_count" : 1036,
              "collection_time_in_millis" : 21383
            },
            "old" : {
              "collection_count" : 0,
              "collection_time_in_millis" : 0
            }
          }
        },
        "buffer_pools" : {
          "mapped" : {
            "count" : 2029,
            "used_in_bytes" : 9131618118,
            "total_capacity_in_bytes" : 9131618118
          },
          "direct" : {
            "count" : 449,
            "used_in_bytes" : 10596477,
            "total_capacity_in_bytes" : 10596474
          },
          "mapped - 'non-volatile memory'" : {
            "count" : 0,
            "used_in_bytes" : 0,
            "total_capacity_in_bytes" : 0
          }
        },
        "classes" : {
          "current_loaded_count" : 55894,
          "total_loaded_count" : 57926,
          "total_unloaded_count" : 2032
        }
      }
    },
    "8Rd2izDOQ2-HtwMJQeHEzw" : {
      "timestamp" : 1744636955406,
      "name" : "e075b4a9f19e2602df6a0734fcac2ee3",
      "roles" : [ "data", "ingest", "master", "remote_cluster_client" ],
      "jvm" : {
        "timestamp" : 1744636955406,
        "uptime_in_millis" : 1118643005,
        "mem" : {
          "heap_used_in_bytes" : 3565309256,
          "heap_used_percent" : 41,
          "heap_committed_in_bytes" : 8589934592,
          "heap_max_in_bytes" : 8589934592,
          "non_heap_used_in_bytes" : 477237584,
          "non_heap_committed_in_bytes" : 529268736,
          "pools" : {
            "young" : {
              "used_in_bytes" : 2793406464,
              "max_in_bytes" : 0,
              "peak_used_in_bytes" : 5146411000,
              "peak_max_in_bytes" : 0,
              "last_gc_stats" : {
                "used_in_bytes" : 0,
                "max_in_bytes" : 0,
                "usage_percent" : -1
              }
            },
            "old" : {
              "used_in_bytes" : 765002240,
              "max_in_bytes" : 8589934592,
              "peak_used_in_bytes" : 5971782552,
              "peak_max_in_bytes" : 8589934592,
              "last_gc_stats" : {
                "used_in_bytes" : 723059200,
                "max_in_bytes" : 8589934592,
                "usage_percent" : 8
              }
            },
            "survivor" : {
              "used_in_bytes" : 6900552,
              "max_in_bytes" : 0,
              "peak_used_in_bytes" : 148897792,
              "peak_max_in_bytes" : 0,
              "last_gc_stats" : {
                "used_in_bytes" : 6900552,
                "max_in_bytes" : 0,
                "usage_percent" : -1
              }
            }
          }
        },
        "threads" : {
          "count" : 561,
          "peak_count" : 561
        },
        "gc" : {
          "collectors" : {
            "young" : {
              "collection_count" : 2291,
              "collection_time_in_millis" : 59008
            },
            "G1 Concurrent GC" : {
              "collection_count" : 1564,
              "collection_time_in_millis" : 38592
            },
            "old" : {
              "collection_count" : 0,
              "collection_time_in_millis" : 0
            }
          }
        },
        "buffer_pools" : {
          "mapped" : {
            "count" : 2321,
            "used_in_bytes" : 6456674065,
            "total_capacity_in_bytes" : 6456674065
          },
          "direct" : {
            "count" : 464,
            "used_in_bytes" : 10640045,
            "total_capacity_in_bytes" : 10640041
          },
          "mapped - 'non-volatile memory'" : {
            "count" : 0,
            "used_in_bytes" : 0,
            "total_capacity_in_bytes" : 0
          }
        },
        "classes" : {
          "current_loaded_count" : 57941,
          "total_loaded_count" : 60094,
          "total_unloaded_count" : 2153
        }
      }
    }
  }
}

@prakashapr27 It doesn’t look like OpenSearch complaining about high heap size memory usage.
The error comes from your script as you’re probably loading everything to the memory in one go.
Have you consider dividing it? Did you monitor memory of your client machine?