Versions
Fluent-Bit 3.1.3
Data Prepper 2.8.0
OpenSearch 2.15.0
OpenSearch Dashboards 2.15.0
RHEL 7.9
Describe the issue:
I am new to OpenSearch and trying to setup a prototype of a basic SIEM solution.
I am able to configure a basic flow that pulls firewall log files through Tail plugin via Fluent-Bit, sends it through Data Prepper into OpenSearch and visualizes on Dashboards.
Data Prepper has a log-pipeline configured with http source, Grok processor to parse firewall logs, and OpenSearch sink. I could see records flowing into Dashboards UI when I manually append events into the sample source log file to test.
However, when live logs are read which have several events flowing in per second, Data Prepper throws below error continuously and I could not see any documents indexed into OpenSearch.
I tried increasing buffer sizes as seen below but to vain. I’ve been struggling to figure the missing bit since a couple of days now. Any help is much appreciated. Thanks.
Configuration:
Data Prepper log pipeline config:
cat pipelines/log_pipeline.yaml
log-pipeline:
workers: 16
source:
http:
authentication:
http_basic:
username:
password:
buffer:
bounded_blocking:
buffer_size: 100000
batch_size: 2000
processor:
- grok:
match:
log: [ “%{SYSLOGTIMESTAMP:timestamp} %{IPORHOST:dvchost} %{DATA},%{NUMBER:fw_sn},%{DATA:eventtype},%{DATA:subtype},%{NUMBER},%{DATA},%{IPV4:srcaddr},%{IPV4:dstaddr},%{IPV4},%{IPV4},%{DATA:rule},%{DATA},%{DATA},%{DATA:app},%{DATA:vsys},%{DATA},%{DATA},%{DATA},%{DATA},%{DATA},%{DATA},%{NUMBER:sid},%{DATA},%{NUMBER:srcport},%{NUMBER:dstport},%{DATA},%{DATA},%{DATA},%{WORD:proto},%{WORD:act},%{NUMBER:bytes},%{NUMBER:bytes_sent},%{NUMBER:bytes_rcvd},%{DATA},%{DATA},%{DATA},%{DATA:url_cat},%{GREEDYDATA}” ]
sink:
- opensearch:
hosts: [ “https://localhost:9200” ]
# Change to your credentials
username: “”
password: “”
# Add a certificate file if you are accessing an OpenSearch cluster with a self-signed certificate
#cert: /path/to/cert
index: paloalto_logs
Relevant Logs or Screenshots:
Data Prepper stdout logs:
2024-07-25T00:27:41,288 [log-pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.plugins.source.loghttp.HTTPSource - Started http source on port 2021…
2024-07-25T00:27:41,289 [log-pipeline-sink-worker-2-thread-1] INFO org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [log-pipeline] - Submitting request to initiate the pipeline processing
2024-07-25T00:28:32,562 [log-pipeline-processor-worker-1-thread-13] ERROR org.opensearch.dataprepper.plugins.processor.grok.GrokProcessor - Matching on record [org.opensearch.dataprepper.model.log.JacksonLog@1efabb69] took longer than [30000] and timed out
2024-07-25T00:28:35,475 [log-pipeline-processor-worker-1-thread-16] ERROR org.opensearch.dataprepper.plugins.processor.grok.GrokProcessor - Matching on record [org.opensearch.dataprepper.model.log.JacksonLog@a4d9cc6] took longer than [30000] and timed out
2024-07-25T00:28:36,244 [log-pipeline-processor-worker-1-thread-8] ERROR org.opensearch.dataprepper.plugins.processor.grok.GrokProcessor - Matching on record [org.opensearch.dataprepper.model.log.JacksonLog@5bbf245a] took longer than [30000] and timed out
org.opensearch.dataprepper.plugins.source.loghttp.LogHTTPService - Failed to write the request of size 528259 due to: Pipeline [log-pipeline] - Buffer does not have enough capacity left for the number of records: 665, timed out waiting for slots.
2024-07-25T00:29:42,182 [pool-23-thread-55] ERROR org.opensearch.dataprepper.plugins.source.loghttp.LogHTTPService - Failed to write the request of size 2063084 due to: Pipeline [log-pipeline] - Buffer does not have enough capacity left for the number of records: 2591, timed out waiting for slots.
2024-07-25T00:29:42,249 [log-pipeline-processor-worker-1-thread-5] ERROR org.opensearch.dataprepper.plugins.processor.grok.GrokProcessor - Matching on record [org.opensearch.dataprepper.model.log.JacksonLog@63eb12f] took longer than [30000] and timed out
2024-07-25T00:29:43,076 [pool-23-thread-56] ERROR org.opensearch.dataprepper.plugins.source.loghttp.LogHTTPService - Failed to write the request of size 45446 due to: Pipeline [log-pipeline] - Buffer does not have enough capacity left for the number of records: 59, timed out waiting for slots.
2024-07-25T00:29:44,097 [pool-23-thread-28] ERROR org.opensearch.dataprepper.plugins.source.loghttp.LogHTTPService - Failed to write the request of size 369872 due to: Pipeline [log-pipeline] - Buffer does not have enough capacity left for the number of records: 462, timed out waiting for slots.
2024-07-25T00:29:44,149 [pool-23-thread-1] ERROR org.opensearch.dataprepper.plugins.source.loghttp.LogHTTPService - Failed to write the request of size 2079994 due to: Pipeline [log-pipeline] - Buffer does not have enough capacity left for the number of records: 2635, timed out waiting for slots.
2024-07-25T00:29:44,212 [log-pipeline-processor-worker-1-thread-4] ERROR org.opensearch.dataprepper.plugins.processor.grok.GrokProcessor - Matching on record [org.opensearch.dataprepper.model.log.JacksonLog@3ec8fcd] took longer than [30000] and timed out
2024-07-25T00:29:44,256 [log-pipeline-processor-worker-1-thread-6] ERROR org.opensearch.dataprepper.plugins.processor.grok.GrokProcessor - Matching on record [org.opensearch.dataprepper.model.log.JacksonLog@36bbf1c8] took longer than [30000] and timed out
2024-07-25T00:29:45,081 [pool-23-thread-57] ERROR org.opensearch.dataprepper.plugins.source.loghttp.LogHTTPService - Failed to write the request of size 195468 due to: Pipeline [log-pipeline] - Buffer does not have enough capacity left for the number of records: 240, timed out waiting for slots.
2024-07-25T00:29:46,135 [pool-23-thread-29] ERROR org.opensearch.dataprepper.plugins.source.loghttp.LogHTTPService - Failed to write the request of size 792580 due to: Pipeline [log-pipeline] - Buffer does not have enough capacity left for the number of records: 1003, timed out waiting for slots.
2024-07-25T00:29:46,151 [log-pipeline-processor-worker-1-thread-1] ERROR org.opensearch.dataprepper.plugins.processor.grok.GrokProcessor - Matching on record [org.opensearch.dataprepper.model.log.JacksonLog@5e0f200e] took longer than [30000] and timed out
2024-07-25T00:29:46,194 [pool-23-thread-58] ERROR org.opensearch.dataprepper.plugins.source.loghttp.LogHTTPService - Failed to write the request of size 2080053 due to: Pipeline [log-pipeline] - Buffer does not have enough capacity left for the number of records: 2590, timed out waiting for slots.