# 2 data prepper node work slowly than 1

**URL:** <https://forum.opensearch.org/t/2-data-prepper-node-work-slowly-than-1/23796>\
**Category:** Data Prepper\
**Created:** [March 17, 2025, 6:18am UTC](https://forum.opensearch.org/t/2-data-prepper-node-work-slowly-than-1/23796 "2025-03-17T06:18:08Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Nash](https://avatars.discourse-cdn.com/v4/letter/n/91b2a8/32.png) [@Nash](https://forum.opensearch.org/u/Nash)\
**Post date:** [March 17, 2025, 6:18am UTC](https://forum.opensearch.org/t/2-data-prepper-node-work-slowly-than-1/23796/1 "2025-03-17T06:18:08Z")

</div>

**Versions**  
opensearch-2.18.0-1.x86\_64  
opensearch-data-prepper-jdk-2.10.2-linux-x64

**Describe the issue** :  
2 Ingest nodes with data prepper, read a Kafka topic of simple log data (dummy log with a @timestamp).

I run only _ingest\_1_, all is ok, I’m limited a 30K log/sec.

When I add the second ingest node, _ingest\_2_, performance drop down.

I need 2 ingest for disponibility requirement.

Why adding a second ingest prepper node drop down the speed?

How could I fix it?

**Configuration** :  
kafka: 10.10.10.100:9092  
cluster: 6 data, 2 master & 2 ingest nodes.  
2 ingest nodes:  
_ingest\_1_ : 4 vCPU 10Go RAM 10.10.10.1  
_ingest\_2_: 2 vCPU 10Go RAM 10.10.10.2

**Relevant configuration** :  
Same configuration for ingest\_1 & ingest\_2, except _workers_.

```auto
pipelines.yaml

```

```auto
---
kafka-pipeline:
  workers: 8 # 4 for ingest_2
  delay: "50"
  source:
    kafka:
      bootstrap_servers:
        - 10.10.10.100:9092
      topics:
        - name: pipeline-fluentbit
          group_id: opensearch-data-prepper
          auto_offset_reset: "earliest"
          auto_commit: true
      encryption:
        type: "none"
        insecure: true

  processor:
    - parse_json:
        source: message
        delete_source: true
    - date:
        from_time_received: true
        destination: "@timestamp"

  sink:
    - opensearch:
        hosts:
          - "https://10.10.10.1:9200"
          - "https://10.10.10.2:9200"
        index: "pipeline-fluentbit"
        username: "admin"
        password: ""
        insecure: true
        batch_size: 5000
        flush_interval: "50"
---

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/3/33547ea01a5b12dcca2958411d3edd97ae2ea8c1.png) [@system](https://forum.opensearch.org/u/system)\
**Post date:** [May 16, 2025, 6:18am UTC](https://forum.opensearch.org/t/2-data-prepper-node-work-slowly-than-1/23796/2 "2025-05-16T06:18:41Z")

</div>

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.