# When performing bulk requests we see performance differences in speed between "\_search?pretty&preference=\_replica" & "\_search?pretty&preference=\_primary" **URL:** \ **Category:** OpenSearch\ **Tags:** discuss\ **Created:** [November 3, 2025, 9:55am UTC](https://forum.opensearch.org/t/when-performing-bulk-requests-we-see-performance-differences-in-speed-between-search-pretty-preference-replica-search-pretty-preference-primary/27419 "2025-11-03T09:55:49Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![SearchBoy](https://avatars.discourse-cdn.com/v4/letter/s/838e76/32.png) [@SearchBoy](https://forum.opensearch.org/u/SearchBoy)\ **Post date:** [November 3, 2025, 9:55am UTC](https://forum.opensearch.org/t/when-performing-bulk-requests-we-see-performance-differences-in-speed-between-search-pretty-preference-replica-search-pretty-preference-primary/27419/1 "2025-11-03T09:55:49Z")
opensearch 2.19.3 1. if setting preference=\_primary\_first, search operations are always fast costing 40ms 2. if preference=replica，the first query will be slow spending 2-5s after a few time silence 3. this happens when there are some bulking operations 4. set request\_cache=false does not help 5. “first query is slow, then fast, if stop query for 1 minute, then first query will be slow” sounds like cache-miss question, but making datanode from 32G to 256G does not help and closing query-cache, request-cache, fielddata-cache are all useless ![image](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/e/e15f3e0d119ac618747073c5920b4ef08764316d.png) this query works on oss based cluster to make this more obvious，while on SSD, slow query spend 3-5s; More confusing, After refresh\_interval of target index was set to 1s explicitly which is exactly equal to the default value, no slow query appears ???!!! Is this about [Skip shard refreshes if shard is `search idle` by s1monw · Pull Request #27500 · elastic/elasticsearch · GitHub](https://github.com/elastic/elasticsearch/pull/27500) , search Idle ??? search idle !!! One Week debug !!! ---
**Author:** ![discourse\_ai\_spam](https://avatars.discourse-cdn.com/v4/letter/d/c68b51/32.png) [@discourse\_ai\_spam](https://forum.opensearch.org/u/discourse_ai_spam)\ **Post date:** [November 4, 2025, 5:14am UTC](https://forum.opensearch.org/t/when-performing-bulk-requests-we-see-performance-differences-in-speed-between-search-pretty-preference-replica-search-pretty-preference-primary/27419/2 "2025-11-04T05:14:18Z")
---
**Author:** ![Leeroy](https://avatars.discourse-cdn.com/v4/letter/l/eb9ed0/32.png) [@Leeroy](https://forum.opensearch.org/u/Leeroy)\ **Post date:** [November 4, 2025, 12:25pm UTC](https://forum.opensearch.org/t/when-performing-bulk-requests-we-see-performance-differences-in-speed-between-search-pretty-preference-replica-search-pretty-preference-primary/27419/3 "2025-11-04T12:25:08Z")
Hey @SearchBoy , Could you provide a sample of your bulk request used when noticing the speed differences. Leeroy. ---
**Author:** ![Leeroy](https://avatars.discourse-cdn.com/v4/letter/l/eb9ed0/32.png) [@Leeroy](https://forum.opensearch.org/u/Leeroy)\ **Post date:** [November 4, 2025, 12:40pm UTC](https://forum.opensearch.org/t/when-performing-bulk-requests-we-see-performance-differences-in-speed-between-search-pretty-preference-replica-search-pretty-preference-primary/27419/4 "2025-11-04T12:40:42Z")
Hey @SearchBoy , From initial testing I don’t see any consistent difference, if you could provide a sample of the bulk and size of the index, total shards and replicas. Leeroy. ---
**Author:** ![system](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/3/33547ea01a5b12dcca2958411d3edd97ae2ea8c1.png) [@system](https://forum.opensearch.org/u/system)\ **Post date:** [November 4, 2025, 3:06pm UTC](https://forum.opensearch.org/t/when-performing-bulk-requests-we-see-performance-differences-in-speed-between-search-pretty-preference-replica-search-pretty-preference-primary/27419/5 "2025-11-04T15:06:20Z")
---
**Author:** ![SearchBoy](https://avatars.discourse-cdn.com/v4/letter/s/838e76/32.png) [@SearchBoy](https://forum.opensearch.org/u/SearchBoy)\ **Post date:** [November 6, 2025, 6:53am UTC](https://forum.opensearch.org/t/when-performing-bulk-requests-we-see-performance-differences-in-speed-between-search-pretty-preference-replica-search-pretty-preference-primary/27419/6 "2025-11-06T06:53:01Z")
Details 1. two datanode(8c32G) and one clustermanager node(8c32G) 2. 6 primary shard with 6 replica shards 3. every shard: “p STARTED 1553477 1.6gb” 4. query is just like : “from”:0,“size”:40,“query”:{“bool”:{“must”:[{“term”:{“commodity\_status”:{“value”:20,“boost”:1.0}}},{“term”:{“commodity\_show\_status”:{“value”:0,“boost”:1.0}}},{“terms”:{“template\_id”:… “adjust\_pure\_negative”:true,“boost”:1.0}},“sort”:[{“commodity\_update\_time”:{“order”:“desc”}}],“track\_total\_hits”:false 5. bulk is just update these existed doc with 500 TPS（translog: request） Besides 1. I think this is relevent to IO, because in my mirro cluster which is based on “oss file system“, searching with _replica spend 30-50s while_ \_primary 40ms 2. what make me confused are : 1. I can not grap a waiting stack with jstack, it seems like there is no any long waiting action 2. I print on- cpu and wall time flamegraph every 5s repeating 6 times constantly (30s totally repeat 3 times)(on oss fs cluster to make things easier for jstack or asprof), however for ((i=0;i\<6;i++));do date; #./async-profiler/build/bin/asprof --wall 1ms -d 5 -f perf\_wall$i.html 1 ./async-profiler/build/bin/asprof -e cpu -i 1ms -d 5 -f perf\_cpu$i.html 1 done I am really confused about on which opensearch function the search query wait…. ---
**Author:** ![SearchBoy](https://avatars.discourse-cdn.com/v4/letter/s/838e76/32.png) [@SearchBoy](https://forum.opensearch.org/u/SearchBoy)\ **Post date:** [November 6, 2025, 6:53am UTC](https://forum.opensearch.org/t/when-performing-bulk-requests-we-see-performance-differences-in-speed-between-search-pretty-preference-replica-search-pretty-preference-primary/27419/7 "2025-11-06T06:53:37Z")
![image](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/b/b557adf281ed1efa6416f3ea199dd4a42e0bb2ac.jpeg) 2 or 3 constantly-printed graph show same stack ---
**Author:** ![SearchBoy](https://avatars.discourse-cdn.com/v4/letter/s/838e76/32.png) [@SearchBoy](https://forum.opensearch.org/u/SearchBoy)\ **Post date:** [November 6, 2025, 6:55am UTC](https://forum.opensearch.org/t/when-performing-bulk-requests-we-see-performance-differences-in-speed-between-search-pretty-preference-replica-search-pretty-preference-primary/27419/8 "2025-11-06T06:55:35Z")
![image](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/3/3fcf05ee7ba60e2c5900a5f11b618e577dce8c1f.jpeg) 2 or 3 constantly-printed graph show same stack ---
**Author:** ![SearchBoy](https://avatars.discourse-cdn.com/v4/letter/s/838e76/32.png) [@SearchBoy](https://forum.opensearch.org/u/SearchBoy)\ **Post date:** [November 6, 2025, 6:57am UTC](https://forum.opensearch.org/t/when-performing-bulk-requests-we-see-performance-differences-in-speed-between-search-pretty-preference-replica-search-pretty-preference-primary/27419/9 "2025-11-06T06:57:04Z")
![image](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/e/ee91cc5e02d2bf8485387795c55f9b0966670cc5.jpeg) 5 constantly-printed graph show same stack sorry，last 3 graph has search releted stack, but its rate is lower than 0.1% ---
**Author:** ![SearchBoy](https://avatars.discourse-cdn.com/v4/letter/s/838e76/32.png) [@SearchBoy](https://forum.opensearch.org/u/SearchBoy)\ **Post date:** [November 6, 2025, 6:58am UTC](https://forum.opensearch.org/t/when-performing-bulk-requests-we-see-performance-differences-in-speed-between-search-pretty-preference-replica-search-pretty-preference-primary/27419/10 "2025-11-06T06:58:09Z")
really appreciate you answering my question ---
**Author:** ![SearchBoy](https://avatars.discourse-cdn.com/v4/letter/s/838e76/32.png) [@SearchBoy](https://forum.opensearch.org/u/SearchBoy)\ **Post date:** [November 6, 2025, 7:06am UTC](https://forum.opensearch.org/t/when-performing-bulk-requests-we-see-performance-differences-in-speed-between-search-pretty-preference-replica-search-pretty-preference-primary/27419/11 "2025-11-06T07:06:30Z")
![image](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/5/56d4e10be590c7450bde3a3284cef7d4aced94a5.jpeg) ---
**Author:** ![SearchBoy](https://avatars.discourse-cdn.com/v4/letter/s/838e76/32.png) [@SearchBoy](https://forum.opensearch.org/u/SearchBoy)\ **Post date:** [November 9, 2025, 2:48pm UTC](https://forum.opensearch.org/t/when-performing-bulk-requests-we-see-performance-differences-in-speed-between-search-pretty-preference-replica-search-pretty-preference-primary/27419/12 "2025-11-09T14:48:19Z")
It’s all about search idle, thanks