Memory issue - java client 2.14.0

OpenSearch Client Libraries
1

java client 2.14

we are using open search java client 2.14.0 , we are observing that in heap memory the response are getting retained -org.opensearch.client.transport.httpclient5.internal.HeapBufferedAsyncEntityConsumer . Earlier after every request the memory used to come down but now it is gradually climbing and finally it forces microservices to restart

image
image1492×613 60.1 KB

2

This is very weird but it looks like the response is never being consumed? Could you please share the snippet(s) how the client is being used? Thank you

3

List<Hit<Map<String,Object>>> hits = response.hits().hits(); return hits.stream() .map(hit → { Map<String, Object> data = Optional.ofNullable(hit.source()).orElse(new HashMap<>()); data.put(“_id”, hit.id()); return data; }) .map(map → replaceKey(map, request)).collect(Collectors.toList());
This is how we are reading the response

4

Thanks @vgjk , could you please share how the client is being used? Basically, how the response is being received.

5

public SearchResponse search(SearchRequest searchRequest) throws IOException {
return openSearchClient.search(searchRequest, Map.class);
}

this is the common function we call to get the response please find below the triggering code

SearchResponse<Map<String,Object>> searchResponse = elasticSearchClient.search(searchRequest);

6

public SearchResponse search(SearchRequest searchRequest) throws IOException {
return openSearchClient.search(searchRequest, Map.class);
}

this is the common function we call to get the response please find below the triggering code

SearchResponse<Map<String,Object>> searchResponse = elasticSearchClient.search(searchRequest);
after consuming the data we are directly freeing the memory by setting searchResponse as null
searchResponse = null;

7

as per our use case we can’t a predefined dto as we have 10 different index with grouping response so that is why we are taking it to map . we are converting it to map but we are freeing it up after the useage

8

reta is there an explicit way to free up memory held by the response , we searched the documentation couldn’t find anyting . client.close will free it up but we would like to get something which would explictly clear the memory after every search

image
image1695×655 76.7 KB

9

This is interesting, few things:

  • the response entity processing lifecycle does resources cleanup as part of the lifecycle, no other ways to enforce this process from the outside
  • you are using java agent (new relic) which seems to be holding references (weak ones)

Could you please try to evaluate the application without the java agent? (new relic).
Thank you.

10

Thanks reta we can try that option couple of querries

  1. is there any GC specific options need to be set for better performance
  2. java client currently it is singleton it would be closed only when the microservice is pull down . we are thinking about testing the behavior by opening new client per request and after processing the responsewe are thinking about closing it would it creates any performance issue
11

Thank you @vgjk , regarding your questions:

  1. There are tons of options available for GC of your choice, please refer to Java Platform, Standard Edition HotSpot Virtual Machine Garbage Collection Tuning Guide, Release 21
  2. Client is thread safe and could be shared, there should be no issues with that (to keep it as a singleton)
12

Thanks @reta , the newrelic seems to be the culprit

13

@vgjk did you guys get to the root of what the issue was? We are facing exactly similar problem and service is getting OOMed on production.