Reindexing the same index again is giving different counts for neural search

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

OpenSearch 3.2.0

Describe the issue:

If we reindex a document with vector fields with no change in data or the embedding model.
Then use same search query for hybrid search.
I am getting different count results.

Is there any explanation for this?

For a 8k docs index each time i rebuild and search using the radial search max_distance of 1.4 i am getting counts from 78, 413 and 493 changing variably each time.
Same is happening even for min_score

Please give a feedback or if i am missing any configuration

Configuration:
OpenSearch 3.2.0 with embedding model and vectors indexed

Any feedback on this would be helpful
Please let know if any data is required

it seems like Fix postion overflow in hybrid bulk scorer by vibrantvarun · Pull Request #1706 · opensearch-project/neural-search · GitHub check with 3.5.0

I tried with Opensearch 3.9.0 and still faced the same issue

with a small subset of 1203 docs
First time index and search i am getting the count for results is 18
next time delete , reindex and the same query search i am getting the count as 16

It’s tough. It might be benign tie or so. Try to drop replicas, request full results.
Enable explain pipeline for hybrid; request explanations. Then feed it into AI/chat/agent along side with the query.
They might be good in digging into causes.

for 10k docs i am seeing the counts varying very huge from 1k to 5k
So what is the explanation for same dataset , same pipeline same query , how can the counts be varied so much?
Is there an issue here or an explanation for this?

Please try with SEGMENT type replication in your index settings.

Also, try connecting to master service while querying, if you have master and data nodes as seperate.

To add to what @mkhl pointed out: that bug (neural-search #1706, fixed in 3.5.0) fits your symptoms well, and 3.2 still has the old code. In the hybrid query’s bulk scorer, after a scoring window it jumps ahead to the highest next doc ID across the subqueries, so k-NN hits with lower doc IDs in that segment get skipped. Doc IDs depend on indexing order and segment merges, so every rebuild skips a different set of k-NN hits, and the counts jump around.

Two quick checks before you upgrade:

1. Run the radial neural query on its own (outside hybrid) against each rebuilt index. If the count is stable there but varies inside hybrid, it’s the bulk scorer bug, and 3.5+ should fix it.

2. If the standalone count also varies, that part comes from the HNSW graph: radial search walks the graph, so it’s approximate like top-k, and the graph comes out a bit different on every build (insertion order, concurrent indexing, merges). I’d expect small differences from that, not 78 vs. 493. An exact script_score query (knn_score) with the same threshold gives you the true count to compare against, and force-merging to one segment (_forcemerge?max_num_segments=1) makes results more repeatable, since each segment has its own graph.

If you can share the hybrid query and the mapping of the vector field (engine, space_type), it’s easier to say which of the two it is.

Hi ,

Please find the details below:

index name: semantic-repeat-test

Model used for embedding:
huggingface/sentence-transformers/all-MiniLM-L12-v2

Mapping:

{
“settings”: {
“index.knn”: true,
“number_of_shards”: 1,
“number_of_replicas”: 0
},
“mappings”: {
“properties”: {
“title”: {
“type”: “text”
},
“description”: {
“type”: “semantic”,
“model_id”: “GjOB3poBn6Lh5apOZkru”,
“raw_field_type”: “text”,
“chunking”: false,
“dense_embedding_config”: {
“method”: {
“engine”: “lucene”,
“name”: “hnsw”,
“parameters”: {}
}
}
}
}
}
}

Search query is:
POST /semantic-repeat-test/_search?search_pipeline=_none&allow_partial_search_results=false
{
“from”: 0,
“size”: 10,
“track_total_hits”: true,
“_source”: [
“title”,
“description”
],
“query”: {
“neural”: {
“description”: {
“query_text”: “Information Systems”,
“max_distance”: 1.4
}
}
}
}

Documents used 5k able to paste only 2.5k
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“1”}}
{“title”:“System Administrator-10046,San Francisco, San Francisco, CA”,“description”:“Information Systems.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“2”}}
{“title”:“HRIS Specialist,Corporation Headquarters, New York, NY”,“description”:“Human Resources.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“3”}}
{“title”:“Foreign Affairs Officer,Swish Fnd. North Cal Fed Rec, Pleasanton, CA”,“description”:“Information Systems.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“4”}}
{“title”:“Telecommunications Specialist,”,“description”:“Midwest Region.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“5”}}
{“title”:“Computer Programmer,”,“description”:“Midwest Region.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“6”}}
{“title”:“Administrative Officer,”,“description”:“Midwest Region.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“7”}}
{“title”:“Administrator,”,“description”:“Northwestern Region.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“8”}}
{“title”:“Criminal Investigator,”,“description”:“Baltimore Office.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“9”}}
{“title”:“Nuclear Engineer,”,“description”:“Customer Services.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“10”}}
{“title”:“Medical Officer,”,“description”:“Northwestern Region.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“11”}}
{“title”:“Web Designer,San Francisco, San Francisco, CA”,“description”:“Human Resources.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“12”}}
{“title”:“Representative-Customer Assoc,Oklahoma, Oklahoma City, OK”,“description”:“Customer Service.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“13”}}
{“title”:“Criminal Investigator,”,“description”:“Northwestern Region.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“14”}}
{“title”:“Self Service JobCode,”,“description”:“HRMS Division.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“15”}}
{“title”:“Museum curator,”,“description”:“Maintenance Services.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“16”}}
{“title”:“Guest Services Manager,Hong Kong, Hong Kong”,“description”:“Hospitality Services.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“17”}}
{“title”:“Attorney,”,“description”:“Northwestern Region.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“18”}}
{“title”:“Logistics Manager,Swish Fnd. North Cal Fed Rec, Pleasanton, CA”,“description”:“Information Systems.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“19”}}
{“title”:“Customer Representative,”,“description”:“Customer Service.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“20”}}
{“title”:“Data Entry Clerk Specialist - Multiple Jobs,San Francisco, San Francisco, CA”,“description”:“Information Systems.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“21”}}
{“title”:“Position Classification Specia,”,“description”:“Midwest Region.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“22”}}
{“title”:“Developer/Analyst,”,“description”:“Information Systems.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“23”}}
{“title”:“Administrator,”,“description”:“Midwest Region.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“24”}}
{“title”:“Administrator,”,“description”:“Northwestern Region.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“25”}}
{“title”:“Logistics Manager,Swish Fnd. North Cal Fed Rec, Pleasanton, CA”,“description”:“Information Systems.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“26”}}
{“title”:“Nuclear Engineer,”,“description”:“Customer Services.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“27”}}
{“title”:“Program Analyst,”,“description”:“Southern Region.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“28”}}
{“title”:“Web Developer,New Orleans, New Orleans, LA”,“description”:“Information Systems.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“29”}}
{“title”:“Secretary,”,“description”:“Baltimore Office.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“30”}}
{“title”:“Data Entry Clerk,San Francisco, San Francisco, CA”,“description”:“Human Resources.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“31”}}
{“title”:“Bellhop,Hong Kong, Hong Kong”,“description”:“Customer Services.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“32”}}
{“title”:“Data Warehouse Manager,London, London”,“description”:“Information Systems.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“33”}}
{“title”:“Test 19.1,”,“description”:“HRMS Division.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“34”}}
{“title”:“Mail Clerk,”,“description”:“Northwestern Region.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“35”}}
{“title”:“Budget Analyst,”,“description”:“Northwestern Region.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“36”}}
{“title”:“Program Analyst,”,“description”:“Western Region.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“37”}}
{“title”:“Program Analyst,”,“description”:“Southern Region.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“38”}}
{“title”:“Computer Programmer,”,“description”:“Baltimore Office.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“39”}}
{“title”:“Personnel Clerk,”,“description”:“Western Region.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“40”}}
{“title”:“Computer Specialist,”,“description”:“Southern Region.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“41”}}
{“title”:“Labor Relations Specialist,”,“description”:“Northwestern Region.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“42”}}
{“title”:“Sous Chef,Vancouver, Vancouver, BC”,“description”:“Hospitality Services.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“43”}}
{“title”:“Student Trainee,”,“description”:“Business Services.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“44”}}
{“title”:“Criminal Investigator,”,“description”:“Midwest Region.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“45”}}
{“title”:“Analyst-FinancialSr,”,“description”:“Corporate Headquarters.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“46”}}
{“title”:“Web Designer,San Francisco, San Francisco, CA”,“description”:“Human Resources.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“47”}}
{“title”:“Personnel Clerk,”,“description”:“Baltimore Office.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“48”}}
{“title”:“Criminal Investigator,”,“description”:“Midwest Region.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“49”}}
{“title”:“Criminal Investigator,”,“description”:“Baltimore Office.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“50”}}
{“title”:“Secretary,”,“description”:“Midwest Region.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“51”}}
{“title”:“Hospitality Manager,”,“description”:“Hospitality Services.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“52”}}
{“title”:“Clerk-Typist,”,“description”:“Southern Region.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“53”}}
{“title”:“Position Classification Specia,”,“description”:“Baltimore Office.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“54”}}
{“title”:“Employee Relations Specialist,”,“description”:“Baltimore Office.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“55”}}
{“title”:“Dental Officer,”,“description”:“Western Region.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“56”}}
{“title”:“Secretary,”,“description”:“Baltimore Office.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“57”}}
{“title”:“Clerk-Typist,”,“description”:“Baltimore Office.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“58”}}
{“title”:“RS JPM Job Code 25,”,“description”:“Hospitality Services.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“59”}}
{“title”:“General Manager,San Francisco, San Francisco, CA”,“description”:“Human Resources.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“60”}}
{“title”:“Human Resource Specialist,San Francisco, San Francisco, CA”,“description”:“Human Resources.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“61”}}
{“title”:“Logistics Manager,Swish Fnd. North Cal Fed Rec, Pleasanton, CA”,“description”:“Information Systems.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“62”}}
{“title”:“Computer Specialist,”,“description”:“Southern Region.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“63”}}
{“title”:“Director-Customer Services,”,“description”:“Customer Service.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“64”}}
{“title”:“Budget Analyst,”,“description”:“Western Region.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“65”}}
{“title”:“Web Developer,San Francisco, San Francisco, CA”,“description”:“Information Systems.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“66”}}
{“title”:“Guest Services Manager,San Francisco, San Francisco, CA”,“description”:“Hotel Management.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“67”}}
{“title”:“test position management,”,“description”:“HRMS Division.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“68”}}
{“title”:“Criminal Investigator,”,“description”:“Baltimore Office.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“69”}}
{“title”:“President \u0026 CEO,”,“description”:“All Departments.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“70”}}
{“title”:“Supervisory Employee Relations,”,“description”:“Southern Region.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“71”}}
{“title”:“Employee Relations Specialist,”,“description”:“Midwest Region.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“72”}}
{“title”:“Personnel Clerk,”,“description”:“Midwest Region.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“73”}}
{“title”:“Practical Nurse,”,“description”:“Southern Region.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“74”}}
{“title”:“General Manager,New York, New York, NY”,“description”:“Customer Services.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“75”}}
{“title”:“Chief Personnel Operations,”,“description”:“Southern Region.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“76”}}
{“title”:“JPM9 Position,”,“description”:“JPM2 Department.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“77”}}
{“title”:“Employment Specialist,Corporation Headquarters, New York, NY”,“description”:“Human Resources.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“78”}}
{“title”:“Specialist - Customer Services,”,“description”:“Customer Services.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“79”}}
{“title”:“RS JPM Job Code 7,”,“description”:“Hotel Management.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“80”}}
{“title”:“Sales for VC 004,”,“description”:“HRMS Division.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“81”}}
{“title”:“Secretary,”,“description”:“Southern Region.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“82”}}
{“title”:“Bellhop,San Francisco, San Francisco, CA”,“description”:“Hotel Services.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“83”}}
{“title”:“Medical Officer,”,“description”:“Baltimore Office.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“84”}}
{“title”:“Employee Relations Specialist,”,“description”:“Northwestern Region.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“85”}}
{“title”:“General Manager - Sales,”,“description”:“Sales Division.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“86”}}
{“title”:“Guest Services Manager,Hong Kong, Hong Kong”,“description”:“Hospitality Services.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“87”}}
{“title”:“Bellhop,Christchurch, Christchurch”,“description”:“Customer Services.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“88”}}
{“title”:“Investigator,”,“description”:“Northwestern Region.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“89”}}
{“title”:“Program Analyst,”,“description”:“Northwestern Region.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“90”}}
{“title”:“Criminal Investigator,”,“description”:“Northwestern Region.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“91”}}
{“title”:“Manager-Employment/Recruitment,Corporation Headquarters, New York, NY”,“description”:“Human Resources.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“92”}}
{“title”:“Head Chef,”,“description”:“Hotel Services.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“93”}}
{“title”:“Customer Srvc Representative,Vancouver, Vancouver, BC”,“description”:“Customer Services.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“94”}}
{“title”:“Administrator,”,“description”:“Midwest Region.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“95”}}
{“title”:“Specialist-HRIS,Corporation Headquarters, New York, NY”,“description”:“Human Resources.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“96”}}
{“title”:“Supervisory Personnel Manageme,Swish Foundation National Ofc, Oakland, CA”,“description”:“Human Resources.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“97”}}
{“title”:“Position Classification Specia,”,“description”:“Northwestern Region.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“98”}}
{“title”:“Operating Accountant,”,“description”:“Midwest Region.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“99”}}
{“title”:“Supervisory Computer Specialist,Swish Foundation National Ofc, Oakland, CA”,“description”:“Network Support.”}
{“index”:{“_index”:“semantic-repeat-test”,“_id”:“100”}}
{“title”:“Computer Specialist,”,“description”:“Baltimore Office.”}

was able to upload only 100 docs.

each time i rebuild the neural score is giving different.
I am using the latest opensearch 3.9.0

any feedback / suggestion if i am missing something would be helpful.

Thanks,
Tejas

Thanks for the mapping and query, that changes my earlier answer. You’re on the Lucene engine with a plain neural query (no hybrid), so the hybrid scorer bug I mentioned isn’t involved here.

In 3.9, a neural query with max_distance or min_score on a Lucene-engine field becomes Lucene’s FloatVectorSimilarityQuery, and the k-NN plugin creates it with a fixed decay of 0.95. Without a filter, Lucene then walks the HNSW graph and stops exploring once the neighbours it sees drop below an adaptive threshold. Only a decay of 1.0 makes Lucene fall back to an exact scan, and that value isn’t exposed as a query option. So a radial query returns the documents within your distance that the walk happens to reach, not all of them. Every reindex builds a new graph (and often a different segment layout), so the walk reaches a different subset and the count changes.

A quick way to confirm it with your data: document 1’s description is exactly your query text, so its stored vector (description_semantic_info.embedding, GET semantic-repeat-test/_doc/1) should be the same as the query embedding. Use it in an exact query and rebuild a few times:

POST /semantic-repeat-test/_search
{
"track_total_hits": true,
"min_score": <same min_score as in your radial
query>, "query": {
"script_score": {
"query": { "match_all": {} },
"script": {
"source": "knn_score",
"lang": "knn",
"params": {
"field": "description_semantic_info.embedding",
"query_value": [ ...the 384 values from doc 1... ],
"space_type": "l2"
}
}
}
}
}

If the exact count stays the same on every rebuild while the radial count moves, that’s the cause. For a few thousand documents the exact query is cheap, so if you need complete and stable results for a threshold, that’s the safer tool. If you only need the best matches, a top-k neural query (k instead of max_distance) is much less sensitive to this. A feature request on the k-NN GitHub repo to make the radial decay configurable would also be reasonable.

with k and min score outside gave me consistent results across builds
but only for radial search max_distance and min_score I am getting different results
Still doing some testing
will let know if I get more findings