Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
- Opensearch - 2.17.1
- Lucene - 9.11.1
Describe the issue:
By calling _settings?include_defaults=true API, we know “defaults.index.store” in OpenSearch is “hybrid” which uses both mmapfs and niofs.
As I know, mmapfs is recommended in general unless the index is too large to be in-memory. Lucene indeed decides to open segment files whether JRE_IS_64BIT
is true or false.
public static FSDirectory open(Path path, LockFactory lockFactory) throws IOException {
if (Constants.JRE_IS_64BIT) {
return new MMapDirectory(path, lockFactory);
} else {
return new NIOFSDirectory(path, lockFactory);
}
}
Built with k8s-operator, java process in containers is based on OpenJDK 64-Bit so I figure OpenSearch can only open MmapDirectory.
[opensearch@test-opensearch-cluster-data-0 0]$ java --version
openjdk 21.0.4 2024-07-16 LTS
OpenJDK Runtime Environment Temurin-21.0.4+7 (build 21.0.4+7-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.4+7 (build 21.0.4+7-LTS, mixed mode, sharing)
Despite the above information, in volume(/usr/share/opensearch/data/nodes/0/indices/{uuid}/0
), there are compound files which is a combination of Mmap and Nio FS.
[opensearch@test-opensearch-cluster-data-0 0]$ ls index -lahSS
total 2.8G
-rw-rw-r-- 1 opensearch opensearch 259M Nov 25 05:44 _kh.cfs
-rw-rw-r-- 1 opensearch opensearch 258M Nov 25 06:49 _pl.fdt
-rw-rw-r-- 1 opensearch opensearch 190M Nov 25 06:16 _nx.cfs
-rw-rw-r-- 1 opensearch opensearch 162M Nov 25 06:49 _pl_Lucene99_0.pos
-rw-rw-r-- 1 opensearch opensearch 140M Nov 25 05:58 _kl.cfs
-rw-rw-r-- 1 opensearch opensearch 102M Dec 10 06:09 _1dl.fdt
-rw-rw-r-- 1 opensearch opensearch 100M Nov 25 05:44 _ki.cfs
-rw-rw-r-- 1 opensearch opensearch 97M Dec 10 06:09 _1dl_Lucene99_0.pos
-rw-rw-r-- 1 opensearch opensearch 89M Nov 25 01:47 _ix.fdt
-rw-rw-r-- 1 opensearch opensearch 71M Dec 18 12:13 _1r7.cfs
-rw-rw-r-- 1 opensearch opensearch 58M Nov 25 06:51 _r2.cfs
-rw-rw-r-- 1 opensearch opensearch 55M Dec 4 01:40 _13y.cfs
-rw-rw-r-- 1 opensearch opensearch 47M Nov 25 06:49 _pl_165_AR_TITLE_VECTOR_768.faiss
-rw-rw-r-- 1 opensearch opensearch 44M Nov 25 06:49 _pl_NativeEngines990KnnVectorsFormat_1.vec
-rw-rw-r-- 1 opensearch opensearch 40M Nov 25 06:49 _pl_Lucene99_0.doc
...
I wonder why OpenSearch stores segments by both method though they have to be opened as MMapDirectory class.