Data Prepper is utilizing only 2 cpus even though number of workers are 8

Data Prepper
1

I am ingesting data at a very high rate in Data Prepper but Data Prepper is utilizing only 2 CPUs even though the number of workers is set to 8. After some time I get a buffer full exception. Prometheus metrics are showing 2 CPU utilization only.

Machine Config
CPU: 8
RAM: 16GB

@graytaylor0

2

@asif @dlv - could you give this a look? Thank you

3

@harshit_patel

Thank you for reaching out. The Data Prepper configuration for number of workers will create Java threads for each worker to process data. The operating system decides how it allocates threads, thus I’m unsure why it is not allocating to all available CPUs. Perhaps some more information on your setup would provide clues to the problem.

Regarding the buffer full exception, we have normally found the sink to be the bottleneck. What is your sink in this scenario?

David

4

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.