Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpenSearch: 2.5
Knn plugin version: 2.5.0.0
Describe the issue:
I am currently experiencing a performance issue related to the KNN plugin in my OpenSearch cluster. The cluster has been triggering the circuit breaker, and I’m encountering high graph memory usage. Here are the details of the issue:
Relevant Logs or Screenshots:
{‘_nodes’: {‘total’: 1, ‘successful’: 1, ‘failed’: 0}, ‘cluster_name’: ‘<cluster_id>:<domain_name>’, ‘circuit_breaker_triggered’: True, ‘model_index_status’: None, ‘nodes’: {‘<node_id>’: {‘graph_memory_usage_percentage’: 99.11605, ‘graph_query_requests’: 30480, ‘graph_memory_usage’: 7457111, ‘cache_capacity_reached’: True, ‘load_success_count’: 1721, ‘training_memory_usage’: 0, ‘indices_in_cache’: {‘index_0’: {‘graph_memory_usage_percentage’: 89.40047, ‘graph_memory_usage’: 6726148, ‘graph_count’: 122}, ‘index_1’: {‘graph_memory_usage_percentage’: 9.715582, ‘graph_memory_usage’: 730963, ‘graph_count’: 24}}, ‘script_query_errors’: 0, ‘hit_count’: 28759, ‘knn_query_requests’: 487, ‘total_load_time’: 666478748813, ‘miss_count’: 1721, ‘knn_query_with_filter_requests’: 0, ‘training_memory_usage_percentage’: 0.0, ‘lucene_initialized’: False, ‘graph_index_requests’: 0, ‘faiss_initialized’: False, ‘load_exception_count’: 0, ‘training_errors’: 0, ‘eviction_count’: 1575, ‘nmslib_initialized’: True, ‘script_compilations’: 0, ‘script_query_requests’: 0, ‘graph_query_errors’: 0, ‘indexing_from_model_degraded’: False, ‘graph_index_errors’: 0, ‘training_requests’: 0, ‘script_compilation_errors’: 0}}
Given this problem, i can’t index more documents. What would be the best solution in this case ? Increase the cache size, how ? How can i remove the indices from the plugin cache ? Thank you very much.
Note: I am using the opensearch aws service.