Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
v 2.17.0
Describe the issue:
I am trying to read the documents from opensearch, where i am keep getting Out Of Memory Exception.
Here is my code, where i am trying to read 1 million record at a time, but it’s supporting upto 200konly.
public async Task<IEnumerable<_doc>> ReadAllDocumentsFromElasticSearch(int periodId, CancellationToken cancellationToken = default)
{
var elasticIndex = new ElasticIndex(periodId);
var scrollTimeout = TimeSpan.FromSeconds(await _settingRepository.GetValueOrDefaultAsync(
SettingTypes.ScrollKeepAliveInSeconds, ElasticConstants.DefaultScrollKeepAliveInSeconds));
var batchSize = DocumentStoreConstants.MaxSearchSize;
var searchRequest = new SearchRequest(elasticIndex.DataIndex)
{
Scroll = scrollTimeout,
Size = batchSize,
Query = new MatchAllQuery()
};
var documents = new List<_doc>();
var response = await _elasticClient.SearchAsync<_doc>(searchRequest);
var scrollId = response.ScrollId;
while (response.Documents.Any())
{
documents.AddRange(response.Documents);
var scrollRequest = new ScrollRequest(scrollId, scrollTimeout);
response = await _elasticClient.ScrollAsync<_doc>(scrollRequest);
scrollId = response.ScrollId;
}
var clearScrollRequest = new ClearScrollRequest(scrollId);
await _elasticClient.ClearScrollAsync(clearScrollRequest);
return documents;
}
Configuration:
var connectionSettings = new ConnectionSettings(pool, connection,
sourceSerializer: (builtin, settings) =>
new JsonNetSerializer(builtin, settings, () =>
new JsonSerializerSettings { NullValueHandling = NullValueHandling.Include },
resolver => resolver.NamingStrategy = new CamelCaseNamingStrategy()));
connectionSettings.RequestTimeout(new TimeSpan(1, 0, 0));
connectionSettings.ThrowExceptions();
Relevant Logs or Screenshots:
OpenSearch.Net.UnexpectedOpenSearchClientException: Exception of type 'System.OutOfMemoryException' was thrown.
---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at System.Array.Resize[T](T[]& array, Int32 newSize) at OpenSearch.Net.Utf8Json.Internal.ArrayBuffer`1.Add(T value) at OpenSearch.Net.Utf8Json.Formatters.CollectionFormatterBase`4.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver) at Deserialize(Object[], JsonReader&, IJsonFormatterResolver) at OpenSearch.Client.ReadAsFormatter`2.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver) at Deserialize(Object[], JsonReader&, IJsonFormatterResolver) at OpenSearch.Net.Utf8Json.JsonSerializer.Deserialize[T](Byte[] bytes, Int32 offset, IJsonFormatterResolver resolver) at OpenSearch.Net.Utf8Json.JsonSerializer.DeserializeAsync[T](Stream stream, IJsonFormatterResolver resolver) at OpenSearch.Net.ResponseBuilder.SetBodyAsync[TResponse](ApiCallDetails details, RequestData requestData, Stream responseStream, String mimeType, CancellationToken cancellationToken) at OpenSearch.Net.ResponseBuilder.ToResponseAsync[TResponse](RequestData requestData, Exception ex, Nullable`1 statusCode, IEnumerable`1 warnings, Stream responseStream, String mimeType, CancellationToken cancellationToken) at OpenSearch.Net.HttpConnection.RequestAsync[TResponse](RequestData requestData, CancellationToken cancellationToken) at OpenSearch.Net.RequestPipeline.CallOpenSearchAsync[TResponse](RequestData requestData, CancellationToken cancellationToken) at OpenSearch.Net.Transport`1.RequestAsync[TResponse](HttpMethod method, String path, CancellationToken cancellationToken, PostData data, IRequestParameters requestParameters)
--- End of inner exception stack trace ---