I am ingesting data at a very high rate in Data Prepper but Data Prepper is utilizing only 2 CPUs even though the number of workers is set to 8. After some time I get a buffer full exception. Prometheus metrics are showing 2 CPU utilization only.
Thank you for reaching out. The Data Prepper configuration for number of workers will create Java threads for each worker to process data. The operating system decides how it allocates threads, thus I’m unsure why it is not allocating to all available CPUs. Perhaps some more information on your setup would provide clues to the problem.
Regarding the buffer full exception, we have normally found the sink to be the bottleneck. What is your sink in this scenario?