Opensearch aggregations are slow as compared to Elasticsearch 7.10.2

OpenSearch
1

I am trying to compare the search performance of Opensearch 1.2.4 with Elasticsearch 7.10.2. I observed that performance of aggregations is slow as compared to Elasticsearch.

I have ingested the same data into Elasticsearch 7.10.2 and Opensearch 1.2.4 nodes and performed cardinality search aggregation with single thread. Here are the numbers.

Elasticsearch: Processed 219 requests in one minute
Opensearch: Procesesd 125 requests in one minute

Here is the program that I ran to compare the performance:

import org.elasticsearch.action.search.SearchRequestBuilder;
import org.elasticsearch.action.search.SearchResponse;
import org.elasticsearch.client.transport.TransportClient;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.common.transport.TransportAddress;
import org.elasticsearch.index.query.QueryBuilder;
import org.elasticsearch.index.query.QueryBuilders;
import org.elasticsearch.search.aggregations.AggregationBuilders;
import org.elasticsearch.search.aggregations.metrics.CardinalityAggregationBuilder;
import org.elasticsearch.transport.client.PreBuiltTransportClient;

import java.net.InetAddress;
import java.util.Random;
import java.util.concurrent.atomic.AtomicLong;

public class EsSearchBenchmark {

    public static void main(String[] args) throws Exception {
        Settings settings = Settings.builder().put("cluster.name", "elasticsearch").put("client.transport.ignore_cluster_name", "true").build();
        TransportAddress inetAddress = new TransportAddress(InetAddress.getByName("localhost"), 9300);
        TransportClient client = new PreBuiltTransportClient(settings).addTransportAddress(inetAddress);

        final Random rand = new Random();
        AtomicLong counter = new AtomicLong();
        long start = System.currentTimeMillis();
        while (true) {
            SearchRequestBuilder searchRequest = getSearchRequest(client, "index1", rand.nextInt(10000) + "");
            SearchResponse resp2 = searchRequest.execute().get();
            if (resp2.getFailedShards() != 0) {
                throw new RuntimeException("failed shards > 0");
            }
            long end = System.currentTimeMillis();
            if (end-start > 60*1000) {
                break;
            }
            counter.incrementAndGet();
        }

        int numRequests = counter.intValue();

        System.out.println("Num requests processed: " + numRequests);
    }

    public static SearchRequestBuilder getSearchRequest(TransportClient client, String index, String randomValue) {
        QueryBuilder qb = QueryBuilders.boolQuery().mustNot(QueryBuilders.termQuery("__id.keyword", randomValue));
        CardinalityAggregationBuilder agg = AggregationBuilders
                .cardinality("somename")
                .field("__id.keyword");
        return client.prepareSearch(index).setQuery(qb).addAggregation(agg);
    }
}

Our migration from Elasticsearch to Opensearch is stuck due to this performance issue. Please help in troubleshooting this issue.

Thanks
Rajesh

2

I have identified the commit which caused the slowness.
Here is the commit id:
https://github.com/opensearch-project/OpenSearch/commit/e153629871f9eaa1ba6c0e4bc9143d27ec4ef96c

Upgrading lucene from 8.8.2 to 8.9.0 has caused the slowness.

3

Lucene developer is saying that this issue should be fixed in OpenSearch.

Link: [LUCENE-10509] Performance degraded after upgrade from 8.8.2 to 8.9.0 - ASF JIRA

This is the comment from Lucene developer.

I suspect that this is due to [<del>LUCENE-9663</del>](https://issues.apache.org/jira/browse/LUCENE-9663), which improved compression of the terms dictionary. This affected OpenSearch because the cardinality aggregation performs value lookups on each document. You should open an issue against OpenSearch to change the way cardinality aggregations run to collect matching ordinals into a bitset first, and only look up values once the entire segment has been collected, this should address the performance problem and will likely make the cardinality aggregation faster than it was before Lucene 8.9.

1 Like