In Opensearch Dashboards, hits.total is always a maximum value of 10000. This means anything using a Count metric in a Visualization, or in Alerting, ends up with a maximum value of 10000. This also impacts the “Hits (total)” count in Discover. Dashboards is effectively a broken calculator since you cannot work with values over 10000.
We touched on this during the last Community Meeting (Jan 18 2022). It was suggested I post about this again to bring more current visibility to the issue. This is a high-impact bug since it renders most computational features of Dashboards useless.
This bug is present in at least Opensearch Dashboards 1.2.0, and I assume all other versions since I don’t see anything about this in any release notes. I believe the bug was present in Kibana 7.10.2 and was forked over from that project.
Here is a forum post about this same issue from April 2020:
This is the breaking changes documentation for elasticsearch 7.x, which discusses the various bits which are the underlying caused of this bug:
track_total_hits is now false by default, which causes the backend to return a hit.total maximum value of 10000. A search must include the parameter track_total_hits: true in order for that search to receive accurate hits.total.
hits.total is returned in a different part of the response object than it was before.
I’m not sure this is an issue in Kibana 7.10.2 as well. At least not the simplest instance, which we run into constantly, which is just seeing how many results matched a query. Kibana 7.10.2 will show us that 34,123,456 results matched, where OpenSearch dashboards will never show more than 10,000. OpenSearch via the REST API will return the total number of results as expected when the track_total_hits param is set to true.
Indeed. I can’t get internal customers to adopt Opensearch Dashboards because of this. It is literally a showstopper bug.
The question is, what’s involved in setting track_total_hits to true in Dashboards. Is all the supporting code for this already in Dashboards and it’s just a matter of toggling it on somewhere?
Is there a temporary workaround or patch that could be used until a fix makes it into a release?
I’ve tried grep’ing through the code and experimenting, myself, trying to find if there’s some single place it can be toggled, but I’ve been unsuccessful.
The best I can find is that trackTotalHits is in fact already set to true in get_default_search_params.ts - but evidently is not being honored for an unknown reason.
I believe I might have a change that can fix it [here] but for some reason I can’t recreate the 10,000 count only. Like for me I pulled the latest OpenSearch 1.2.4 executable and latest OpenSearch Dashboards 1.2.0 executable and ran both of them. I created a small document and added to my index.
So my cluster is configured only return back the default amount but the Discover page and others still shows the total hits.
Are you able to share how you are running your stacks or potentially creating a BUG in the repo with recreation steps. If you can’t share an example doc then the mappings would help. Also how your index pattern looks like will also help like (timestamp field and how many indices does it match).
Thanks for this! However, I’m finding my installation of OpenSearch Dashboards (1.2.0) seems to be different than the one you have forked. The files you’ve modified are not similar to the ones I have on disk in my installation. Also, the file suffixes I have are .js, and yours are .ts
I’m going to do a fresh install using your fork and see what happens.
I suspect the only thing special about my setup is that the clusters I’m searching are remote cross-search clusters. I don’t have any data on the cluster local to Dashboards. But that shouldn’t matter, right? I’ll experiment with local data and confirm.
I think you are comparing source code and a compiled / deployed package. Those two are different things. You would need to go through the build process to produce a package that can be installed similar to the downloads you would get from the website.
REVELATION: When searching data on the local cluster, the hits.total count is accurate above 10k. When searching data on remote cross-search configured clusters, the hits.total is always 10000.
This problem appears isolated to remote cross-search clusters? At least, it appears that way for me.
I’ll still try a build of @kavilla 's fork to see if forcing the track_total_hits into the body of the request will fix the hit.total for remote cross-search clusters. Stay tuned…
I’ll still try a build of @kavilla 's fork to see if forcing the track_total_hits into the body of the request will fix the hit.total for remote cross-search clusters
Let me know if you would like me to create a 1.2.0 OpenSearch Dashboards distribution with these changes!
I’m looking at an example of the problem right now while doing some other analysis of data in the indexes.
Its may be tough to see, but those bars are over 20k results. However the total results provided in the discover tab is just 10k, when clearly its much more than that, even as shown in the results count histogram.
I might take you up on that offer of a build, @kavilla. That would accelerate my effort.
(My build environment I’m working with is only 20GB total, and I am running out of disk space as I’m attempting to create the Dashboards build. I’m working on getting more space to build this in…)
Still working on the build. The resulting snapshot doesn’t run, apparently because it doesn’t build/install any of the plugins. Still working on figuring that out.
sudo -u opensearch /usr/share/opensearch-dashboards/bin/opensearch-dashboards-plugin list
Thanks, @kavilla . I was able to pull your distribution from your link above, and it does start up and is usable.
Unfortunately, the 10000 limit persists, and when I inspect the request body, it does not contain track_total_hits: true.
Also, just to sanity check, I did some grep’ing of your distribution to see if the specific lines in your MR were present in the release. I did not find them (I would expect to find them in /src?).
So it seems maybe we’re somehow not getting your code change in the build, still.
When I try to build from your branch ’ avillk/track_total_hits , the resulting snapshot I get is of the 2.0.0 release with no plugins. The snapshot you’re providing is 1.2.0 with plugins. So I must be doing something wrong during my build process. I am following the build instructions from DEVELOPER_GUIDE.md. Specifically, here are the steps I am using:
git clone https://github.com/kavilla/OpenSearch-Dashboards-1.git
cd OpenSearch-Dashboards-1
git checkout avillk/track_total_hits
nvm use
npm i -g yarn
yarn osd bootstrap
yarn build --skip-os-packages
Am I missing a step that would base me on the 1.2.0 release? Instead of the 2.0.0 release?
Thanks again, I will keep trudging at it on my end.
Unfortunately, the 10000 limit persists, and when I inspect the request body, it does not contain track_total_hits: true.
I see, can you go to the dev tools page and hit your index pattern like the following: GET /opensearch_dashboards_sample_data_flights*/_search?track_total_hits=true
Do you get the correct amount?
Also, just to sanity check, I did some grep’ing of your distribution to see if the specific lines in your MR were present in the release. I did not find them (I would expect to find them in /src?).
The lines add were compiled down to JS so I would search for the strings only since the source will look a little different.
When I try to build from your branch ’ avillk/track_total_hits , the resulting snapshot I get is of the 2.0.0 release with no plugins. The snapshot you’re providing is 1.2.0 with plugins. So I must be doing something wrong during my build process.
# defaults to linux x64 but you can build arm64 by passing `-a arm64`
./build.sh manifests/1.2.0/opensearch-dashboards-1.2.0.yml
# then after it builds run the following
./assemble.sh builds/opensearch-dashboards/manifest.yml
I added another change but it explicitly passes the trackTotalHits in the Discover page. I just checked the legacy app for 7.10.2 and I don’t actually see them passing trackTotalHits on the Discover page as well. But just verify you said this was working for the legacy app 7.10.2? So even with the previous changes visualizations was still failing? (Alerting might need to explicitly pass this param if something in OpenSearch defaults track_total_hits to false).
I will have a distribution with the discover page in a few minutes for you.
Thanks @kavilla . I’m running the new distribution you just posted ~10min ago, and I’m getting the same behavior, maximum hits.total displayed in Discover and in Visualize is 10000. When inspecting the Request body, track_total_hits: true is still not present.
And when I used the Dev Tools page like you suggested, the response there does contain an accurate hits.total
Note that this 10000 limit only appears for remote cross-search clusters. Note my index family I’m searching is prefixed with a cluster name. es1:blahblah-*
When I search local index families, the hits.total is accurate.
When I search remote cross-search clusters, the hits.total is always maximum of 10000.
If this was only the result of the patch level change from .0 to .4, that would be a huge red flag for OpenSearch release management. The screenshot I provided was from a cluster running OpenSearch 1.2.3 and Dashboards 1.2.0. I will try to check out the provided dashboards build in the coming days as time allows.
