Beats Client Error Messages sending data to AWS OpenSearch service

General Feedback
,
1

Hello,

I am new to OpenSearch and have minimal experience configuring ELK stack. Please let me know if there is a more appropriate category for my question.

Our enterprise has deployed AWS OpenSearch 2.5.0 as an endpoint for auditbeat, filebeat, journalbeat, metricbeat, and packetbeat on our RHEL7/RHEL8 EC2 instances.

auditbeat-8.7.1-1.x86_64
filebeat-8.7.1-1.x86_64
journalbeat-7.15.2-1.x86_64
metricbeat-8.7.1-1.x86_64
packetbeat-8.7.1-1.x86_64

/var/log/secure and /var/log/message are filling with tens of thousands of messages from the Beats daily. (see below for sample messages)

Questions:

  • Is this configuration of Beats and OpenSearch supported?
  • Would configuration errors on the Beats side, or OpenSearch side cause these errors?
  • What questions should I ask the Enterprise Team that manages the OpenSearch about their configuration to troubleshoot?
  • Where should I start reading Documentation (OpenSearch or Beats) to troubleshoot?

Thank you,
David

Sample messages from /var/log/messages:
May 23 18:04:32 vac10appcpe800 auditbeat[1834]: {“log.level”:“error”,“@timestamp”:“2023-05-23T18:04:32.623Z”,“log.logger”:“logstash”,“log.origin”:{“file.name”:“logstash/async.go”,“file.line”:280},“message”:“Failed to publish events caused by: EOF”,“service.name”:“auditbeat”,“ecs.version”:“1.6.0”}
May 23 18:04:32 vac10appcpe800 auditbeat[1834]: {“log.level”:“error”,“@timestamp”:“2023-05-23T18:04:32.624Z”,“log.logger”:“logstash”,“log.origin”:{“file.name”:“logstash/async.go”,“file.line”:280},“message”:“Failed to publish events caused by: EOF”,“service.name”:“auditbeat”,“ecs.version”:“1.6.0”}
May 23 18:04:32 vac10appcpe800 auditbeat[1834]: {“log.level”:“info”,“@timestamp”:“2023-05-23T18:04:32.624Z”,“log.logger”:“publisher_pipeline_output”,“log.origin”:{“file.name”:“pipeline/client_worker.go”,“file.line”:139},“message”:“Connecting to backoff(async(tcp://aws-logstash-west.ecs.vaec.va.gov:6046))”,“service.name”:“auditbeat”,“ecs.version”:“1.6.0”}
May 23 18:04:32 vac10appcpe800 auditbeat[1834]: {“log.level”:“info”,“@timestamp”:“2023-05-23T18:04:32.624Z”,“log.logger”:“publisher_pipeline_output”,“log.origin”:{“file.name”:“pipeline/client_worker.go”,“file.line”:139},“message”:“Connecting to backoff(async(tcp://aws-logstash-west.ecs.vaec.va.gov:6045))”,“service.name”:“auditbeat”,“ecs.version”:“1.6.0”}
May 23 18:04:32 vac10appcpe800 auditbeat[1834]: {“log.level”:“info”,“@timestamp”:“2023-05-23T18:04:32.624Z”,“log.logger”:“publisher_pipeline_output”,“log.origin”:{“file.name”:“pipeline/client_worker.go”,“file.line”:139},“message”:“Connecting to backoff(async(tcp://aws-logstash-west.ecs.vaec.va.gov:6044))”,“service.name”:“auditbeat”,“ecs.version”:“1.6.0”}
May 23 18:04:32 vac10appcpe800 auditbeat[1834]: {“log.level”:“error”,“@timestamp”:“2023-05-23T18:04:32.625Z”,“log.logger”:“logstash”,“log.origin”:{“file.name”:“logstash/async.go”,“file.line”:280},“message”:“Failed to publish events caused by: client is not connected”,“service.name”:“auditbeat”,“ecs.version”:“1.6.0”}
May 23 18:04:32 vac10appcpe800 auditbeat[1834]: {“log.level”:“error”,“@timestamp”:“2023-05-23T18:04:32.625Z”,“log.logger”:“logstash”,“log.origin”:{“file.name”:“logstash/async.go”,“file.line”:280},“message”:“Failed to publish events caused by: client is not connected”,“service.name”:“auditbeat”,“ecs.version”:“1.6.0”}
May 23 18:04:32 vac10appcpe800 auditbeat[1834]: {“log.level”:“info”,“@timestamp”:“2023-05-23T18:04:32.625Z”,“log.logger”:“publisher”,“log.origin”:{“file.name”:“pipeline/consumer.go”,“file.line”:181},“message”:“Drop batch”,“service.name”:“auditbeat”,“ecs.version”:“1.6.0”}
May 23 18:04:32 vac10appcpe800 auditbeat[1834]: {“log.level”:“info”,“@timestamp”:“2023-05-23T18:04:32.638Z”,“log.logger”:“publisher_pipeline_output”,“log.origin”:{“file.name”:“pipeline/client_worker.go”,“file.line”:147},“message”:“Connection to backoff(async(tcp://aws-logstash-west.ecs.vaec.va.gov:6046)) established”,“service.name”:“auditbeat”,“ecs.version”:“1.6.0”}
May 23 18:04:32 vac10appcpe800 auditbeat[1834]: {“log.level”:“info”,“@timestamp”:“2023-05-23T18:04:32.638Z”,“log.logger”:“publisher_pipeline_output”,“log.origin”:{“file.name”:“pipeline/client_worker.go”,“file.line”:147},“message”:“Connection to backoff(async(tcp://aws-logstash-west.ecs.vaec.va.gov:6045)) established”,“service.name”:“auditbeat”,“ecs.version”:“1.6.0”}
May 23 18:04:32 vac10appcpe800 auditbeat[1834]: {“log.level”:“info”,“@timestamp”:“2023-05-23T18:04:32.640Z”,“log.logger”:“publisher_pipeline_output”,“log.origin”:{“file.name”:“pipeline/client_worker.go”,“file.line”:147},“message”:“Connection to backoff(async(tcp://aws-logstash-west.ecs.vaec.va.gov:6044)) established”,“service.name”:“auditbeat”,“ecs.version”:“1.6.0”}

May 23 18:36:13 vac10appcpe800 filebeat[1836]: {“log.level”:“error”,“@timestamp”:“2023-05-23T18:36:13.515Z”,“log.logger”:“logstash”,“log.origin”:{“file.name”:“logstash/async.go”,“file.line”:280},“message”:“Failed to publish events caused by: EOF”,“service.name”:“filebeat”,“ecs.version”:“1.6.0”}
May 23 18:36:13 vac10appcpe800 filebeat[1836]: {“log.level”:“error”,“@timestamp”:“2023-05-23T18:36:13.516Z”,“log.logger”:“logstash”,“log.origin”:{“file.name”:“logstash/async.go”,“file.line”:280},“message”:“Failed to publish events caused by: EOF”,“service.name”:“filebeat”,“ecs.version”:“1.6.0”}
May 23 18:36:13 vac10appcpe800 filebeat[1836]: {“log.level”:“error”,“@timestamp”:“2023-05-23T18:36:13.518Z”,“log.logger”:“logstash”,“log.origin”:{“file.name”:“logstash/async.go”,“file.line”:280},“message”:“Failed to publish events caused by: EOF”,“service.name”:“filebeat”,“ecs.version”:“1.6.0”}
May 23 18:36:16 vac10appcpe800 filebeat[1836]: {“log.level”:“error”,“@timestamp”:“2023-05-23T18:36:16.751Z”,“log.logger”:“logstash”,“log.origin”:{“file.name”:“logstash/async.go”,“file.line”:280},“message”:“Failed to publish events caused by: client is not connected”,“service.name”:“filebeat”,“ecs.version”:“1.6.0”}
May 23 18:36:16 vac10appcpe800 filebeat[1836]: {“log.level”:“error”,“@timestamp”:“2023-05-23T18:36:16.754Z”,“log.logger”:“logstash”,“log.origin”:{“file.name”:“logstash/async.go”,“file.line”:280},“message”:“Failed to publish events caused by: client is not connected”,“service.name”:“filebeat”,“ecs.version”:“1.6.0”}
May 23 18:36:16 vac10appcpe800 filebeat[1836]: {“log.level”:“error”,“@timestamp”:“2023-05-23T18:36:16.757Z”,“log.logger”:“logstash”,“log.origin”:{“file.name”:“logstash/async.go”,“file.line”:280},“message”:“Failed to publish events caused by: client is not connected”,“service.name”:“filebeat”,“ecs.version”:“1.6.0”}
May 23 18:36:17 vac10appcpe800 filebeat[1836]: {“log.level”:“error”,“@timestamp”:“2023-05-23T18:36:17.752Z”,“log.logger”:“publisher_pipeline_output”,“log.origin”:{“file.name”:“pipeline/client_worker.go”,“file.line”:176},“message”:“failed to publish events: client is not connected”,“service.name”:“filebeat”,“ecs.version”:“1.6.0”}
May 23 18:36:18 vac10appcpe800 filebeat[1836]: {“log.level”:“error”,“@timestamp”:“2023-05-23T18:36:18.047Z”,“log.logger”:“publisher_pipeline_output”,“log.origin”:{“file.name”:“pipeline/client_worker.go”,“file.line”:176},“message”:“failed to publish events: client is not connected”,“service.name”:“filebeat”,“ecs.version”:“1.6.0”}
May 23 18:36:18 vac10appcpe800 filebeat[1836]: {“log.level”:“error”,“@timestamp”:“2023-05-23T18:36:18.098Z”,“log.logger”:“publisher_pipeline_output”,“log.origin”:{“file.name”:“pipeline/client_worker.go”,“file.line”:176},“message”:“failed to publish events: client is not connected”,“service.name”:“filebeat”,“ecs.version”:“1.6.0”}
May 23 18:36:33 vac10appcpe800 filebeat[1836]: {“log.level”:“info”,“@timestamp”:“2023-05-23T18:36:33.520Z”,“log.logger”:“publisher_pipeline_output”,“log.origin”:{“file.name”:“pipeline/client_worker.go”,“file.line”:139},“message”:“Connecting to backoff(async(tcp://aws-logstash-west.ecs.vaec.va.gov:6045))”,“service.name”:“filebeat”,“ecs.version”:“1.6.0”}
May 23 18:36:33 vac10appcpe800 filebeat[1836]: {“log.level”:“info”,“@timestamp”:“2023-05-23T18:36:33.520Z”,“log.logger”:“publisher_pipeline_output”,“log.origin”:{“file.name”:“pipeline/client_worker.go”,“file.line”:139},“message”:“Connecting to backoff(async(tcp://aws-logstash-west.ecs.vaec.va.gov:6048))”,“service.name”:“filebeat”,“ecs.version”:“1.6.0”}
May 23 18:36:33 vac10appcpe800 filebeat[1836]: {“log.level”:“info”,“@timestamp”:“2023-05-23T18:36:33.520Z”,“log.logger”:“publisher_pipeline_output”,“log.origin”:{“file.name”:“pipeline/client_worker.go”,“file.line”:139},“message”:“Connecting to backoff(async(tcp://aws-logstash-west.ecs.vaec.va.gov:6044))”,“service.name”:“filebeat”,“ecs.version”:“1.6.0”}
May 23 18:36:33 vac10appcpe800 filebeat[1836]: {“log.level”:“info”,“@timestamp”:“2023-05-23T18:36:33.530Z”,“log.logger”:“publisher_pipeline_output”,“log.origin”:{“file.name”:“pipeline/client_worker.go”,“file.line”:147},“message”:“Connection to backoff(async(tcp://aws-logstash-west.ecs.vaec.va.gov:6048)) established”,“service.name”:“filebeat”,“ecs.version”:“1.6.0”}
May 23 18:36:33 vac10appcpe800 filebeat[1836]: {“log.level”:“info”,“@timestamp”:“2023-05-23T18:36:33.533Z”,“log.logger”:“publisher_pipeline_output”,“log.origin”:{“file.name”:“pipeline/client_worker.go”,“file.line”:147},“message”:“Connection to backoff(async(tcp://aws-logstash-west.ecs.vaec.va.gov:6045)) established”,“service.name”:“filebeat”,“ecs.version”:“1.6.0”}
May 23 18:36:33 vac10appcpe800 filebeat[1836]: {“log.level”:“info”,“@timestamp”:“2023-05-23T18:36:33.535Z”,“log.logger”:“publisher_pipeline_output”,“log.origin”:{“file.name”:“pipeline/client_worker.go”,“file.line”:147},“message”:“Connection to backoff(async(tcp://aws-logstash-west.ecs.vaec.va.gov:6044)) established”,“service.name”:“filebeat”,“ecs.version”:“1.6.0”}

May 23 18:40:01 vac10appcpe800 filebeat[1836]: {“log.level”:“info”,“@timestamp”:“2023-05-23T18:40:01.667Z”,“log.logger”:“input.filestream”,“log.origin”:{“file.name”:“filestream/input.go”,“file.line”:321},“message”:“Reader was closed. Closing.”,“service.name”:“filebeat”,“id”:“6A4EBEDFAE2D028A”,“source_file”:“filestream::.global::native::18875971-64768”,“path”:“/var/McAfee/agent/logs/macompatsvc_vac10appcpe800.log”,“state-id”:“native::18875971-64768”,“ecs.version”:“1.6.0”}
May 23 18:40:04 vac10appcpe800 filebeat[1836]: {“log.level”:“info”,“@timestamp”:“2023-05-23T18:40:04.379Z”,“log.logger”:“publisher_pipeline_output”,“log.origin”:{“file.name”:“pipeline/client_worker.go”,“file.line”:139},“message”:“Connecting to backoff(async(tcp://aws-logstash-west.ecs.vaec.va.gov:6045))”,“service.name”:“filebeat”,“ecs.version”:“1.6.0”}
May 23 18:40:04 vac10appcpe800 filebeat[1836]: {“log.level”:“info”,“@timestamp”:“2023-05-23T18:40:04.392Z”,“log.logger”:“publisher_pipeline_output”,“log.origin”:{“file.name”:“pipeline/client_worker.go”,“file.line”:147},“message”:“Connection to backoff(async(tcp://aws-logstash-west.ecs.vaec.va.gov:6045)) established”,“service.name”:“filebeat”,“ecs.version”:“1.6.0”}
May 23 18:40:07 vac10appcpe800 filebeat[1836]: {“log.level”:“info”,“@timestamp”:“2023-05-23T18:40:07.679Z”,“log.logger”:“input.filestream”,“log.origin”:{“file.name”:“filestream/input.go”,“file.line”:321},“message”:“Reader was closed. Closing.”,“service.name”:“filebeat”,“id”:“6A4EBEDFAE2D028A”,“source_file”:“filestream::.global::native::18875986-64768”,“path”:“/var/McAfee/agent/logs/McScript.log”,“state-id”:“native::18875986-64768”,“ecs.version”:“1.6.0”}
[root@vac10appcpe800 etc]# clear; grep journalbeat /var/log/secure | tail -60
May 23 18:41:03 vac10appcpe800 journalbeat[1863]: 2023-05-23T18:41:03.843Z#011ERROR#011[logstash]#011logstash/async.go:280#011Failed to publish events caused by: EOF
May 23 18:41:03 vac10appcpe800 journalbeat[1863]: 2023-05-23T18:41:03.843Z#011INFO#011[publisher]#011pipeline/retry.go:219#011retryer: send unwait signal to consumer
May 23 18:41:03 vac10appcpe800 journalbeat[1863]: 2023-05-23T18:41:03.843Z#011INFO#011[publisher]#011pipeline/retry.go:223#011 done
May 23 18:41:03 vac10appcpe800 journalbeat[1863]: 2023-05-23T18:41:03.844Z#011ERROR#011[logstash]#011logstash/async.go:280#011Failed to publish events caused by: client is not connected
May 23 18:41:03 vac10appcpe800 journalbeat[1863]: 2023-05-23T18:41:03.844Z#011INFO#011[publisher]#011pipeline/retry.go:219#011retryer: send unwait signal to consumer
May 23 18:41:03 vac10appcpe800 journalbeat[1863]: 2023-05-23T18:41:03.844Z#011INFO#011[publisher]#011pipeline/retry.go:223#011 done
May 23 18:41:03 vac10appcpe800 journalbeat[1863]: 2023-05-23T18:41:03.849Z#011ERROR#011[logstash]#011logstash/async.go:280#011Failed to publish events caused by: client is not connected
May 23 18:41:03 vac10appcpe800 journalbeat[1863]: 2023-05-23T18:41:03.849Z#011INFO#011[publisher]#011pipeline/retry.go:219#011retryer: send unwait signal to consumer
May 23 18:41:03 vac10appcpe800 journalbeat[1863]: 2023-05-23T18:41:03.849Z#011INFO#011[publisher]#011pipeline/retry.go:223#011 done
May 23 18:41:04 vac10appcpe800 journalbeat[1863]: 2023-05-23T18:41:04.925Z#011ERROR#011[publisher_pipeline_output]#011pipeline/output.go:180#011failed to publish events: client is not connected
May 23 18:41:05 vac10appcpe800 journalbeat[1863]: 2023-05-23T18:41:05.069Z#011ERROR#011[publisher_pipeline_output]#011pipeline/output.go:180#011failed to publish events: client is not connected
May 23 18:41:06 vac10appcpe800 journalbeat[1863]: 2023-05-23T18:41:06.037Z#011ERROR#011[logstash]#011logstash/async.go:280#011Failed to publish events caused by: client is not connected
May 23 18:41:06 vac10appcpe800 journalbeat[1863]: 2023-05-23T18:41:06.037Z#011INFO#011[publisher]#011pipeline/retry.go:219#011retryer: send unwait signal to consumer
May 23 18:41:06 vac10appcpe800 journalbeat[1863]: 2023-05-23T18:41:06.037Z#011INFO#011[publisher]#011pipeline/retry.go:223#011 done
May 23 18:41:07 vac10appcpe800 journalbeat[1863]: 2023-05-23T18:41:07.137Z#011INFO#011[publisher_pipeline_output]#011pipeline/output.go:143#011Connecting to backoff(async(tcp://aws-logstash-west.ecs.vaec.va.gov:6048))
May 23 18:41:07 vac10appcpe800 journalbeat[1863]: 2023-05-23T18:41:07.138Z#011INFO#011[publisher]#011pipeline/retry.go:219#011retryer: send unwait signal to consumer
May 23 18:41:07 vac10appcpe800 journalbeat[1863]: 2023-05-23T18:41:07.138Z#011INFO#011[publisher]#011pipeline/retry.go:223#011 done
May 23 18:41:07 vac10appcpe800 journalbeat[1863]: 2023-05-23T18:41:07.153Z#011INFO#011[publisher_pipeline_output]#011pipeline/output.go:151#011Connection to backoff(async(tcp://aws-logstash-west.ecs.vaec.va.gov:6048)) established
May 23 18:41:07 vac10appcpe800 journalbeat[1863]: 2023-05-23T18:41:07.455Z#011ERROR#011[publisher_pipeline_output]#011pipeline/output.go:180#011failed to publish events: client is not connected
May 23 18:41:08 vac10appcpe800 journalbeat[1863]: 2023-05-23T18:41:08.257Z#011INFO#011[publisher_pipeline_output]#011pipeline/output.go:143#011Connecting to backoff(async(tcp://aws-logstash-west.ecs.vaec.va.gov:6046))
May 23 18:41:08 vac10appcpe800 journalbeat[1863]: 2023-05-23T18:41:08.257Z#011INFO#011[publisher]#011pipeline/retry.go:219#011retryer: send unwait signal to consumer
May 23 18:41:08 vac10appcpe800 journalbeat[1863]: 2023-05-23T18:41:08.257Z#011INFO#011[publisher]#011pipeline/retry.go:223#011 done
May 23 18:41:08 vac10appcpe800 journalbeat[1863]: 2023-05-23T18:41:08.266Z#011INFO#011[publisher_pipeline_output]#011pipeline/output.go:151#011Connection to backoff(async(tcp://aws-logstash-west.ecs.vaec.va.gov:6046)) established
May 23 18:42:08 vac10appcpe800 journalbeat[1863]: 2023-05-23T18:42:08.248Z#011INFO#011[publisher_pipeline_output]#011pipeline/output.go:143#011Connecting to backoff(async(tcp://aws-logstash-west.ecs.vaec.va.gov:6047))
May 23 18:42:08 vac10appcpe800 journalbeat[1863]: 2023-05-23T18:42:08.249Z#011INFO#011[publisher]#011pipeline/retry.go:219#011retryer: send unwait signal to consumer
May 23 18:42:08 vac10appcpe800 journalbeat[1863]: 2023-05-23T18:42:08.249Z#011INFO#011[publisher]#011pipeline/retry.go:223#011 done
May 23 18:42:08 vac10appcpe800 journalbeat[1863]: 2023-05-23T18:42:08.270Z#011INFO#011[publisher_pipeline_output]#011pipeline/output.go:151#011Connection to backoff(async(tcp://aws-logstash-west.ecs.vaec.va.gov:6047)) established
May 23 18:42:09 vac10appcpe800 journalbeat[1863]: 2023-05-23T18:42:09.352Z#011ERROR#011[logstash]#011logstash/async.go:280#011Failed to publish events caused by: EOF
May 23 18:42:09 vac10appcpe800 journalbeat[1863]: 2023-05-23T18:42:09.352Z#011INFO#011[publisher]#011pipeline/retry.go:219#011retryer: send unwait signal to consumer
May 23 18:42:09 vac10appcpe800 journalbeat[1863]: 2023-05-23T18:42:09.352Z#011INFO#011[publisher]#011pipeline/retry.go:223#011 done

May 23 16:03:34 vac10appcpe800 metricbeat[1867]: {“log.level”:“error”,“@timestamp”:“2023-05-23T16:03:34.245Z”,“log.logger”:“logstash”,“log.origin”:{“file.name”:“logstash/async.go”,“file.line”:280},“message”:“Failed to publish events caused by: write tcp 10.247.148.20:47778->10.247.2.139:6045: write: broken pipe”,“service.name”:“metricbeat”,“ecs.version”:“1.6.0”}
May 23 16:03:34 vac10appcpe800 metricbeat[1867]: {“log.level”:“error”,“@timestamp”:“2023-05-23T16:03:34.262Z”,“log.logger”:“logstash”,“log.origin”:{“file.name”:“logstash/async.go”,“file.line”:280},“message”:“Failed to publish events caused by: write tcp 10.247.148.20:53236->10.247.2.241:6047: write: broken pipe”,“service.name”:“metricbeat”,“ecs.version”:“1.6.0”}
May 23 16:03:34 vac10appcpe800 metricbeat[1867]: {“log.level”:“error”,“@timestamp”:“2023-05-23T16:03:34.293Z”,“log.logger”:“logstash”,“log.origin”:{“file.name”:“logstash/async.go”,“file.line”:280},“message”:“Failed to publish events caused by: write tcp 10.247.148.20:42682->10.247.2.139:6044: write: broken pipe”,“service.name”:“metricbeat”,“ecs.version”:“1.6.0”}
May 23 16:03:34 vac10appcpe800 metricbeat[1867]: {“log.level”:“error”,“@timestamp”:“2023-05-23T16:03:34.309Z”,“log.logger”:“logstash”,“log.origin”:{“file.name”:“logstash/async.go”,“file.line”:280},“message”:“Failed to publish events caused by: write tcp 10.247.148.20:58234->10.247.2.241:6046: write: broken pipe”,“service.name”:“metricbeat”,“ecs.version”:“1.6.0”}
May 23 16:03:34 vac10appcpe800 metricbeat[1867]: {“log.level”:“info”,“@timestamp”:“2023-05-23T16:03:34.309Z”,“log.logger”:“publisher”,“log.origin”:{“file.name”:“pipeline/consumer.go”,“file.line”:181},“message”:“Drop batch”,“service.name”:“metricbeat”,“ecs.version”:“1.6.0”}
May 23 16:03:35 vac10appcpe800 metricbeat[1867]: {“log.level”:“error”,“@timestamp”:“2023-05-23T16:03:35.507Z”,“log.logger”:“publisher_pipeline_output”,“log.origin”:{“file.name”:“pipeline/client_worker.go”,“file.line”:176},“message”:“failed to publish events: write tcp 10.247.148.20:58234->10.247.2.241:6046: write: broken pipe”,“service.name”:“metricbeat”,“ecs.version”:“1.6.0”}
May 23 16:03:35 vac10appcpe800 metricbeat[1867]: {“log.level”:“error”,“@timestamp”:“2023-05-23T16:03:35.553Z”,“log.logger”:“publisher_pipeline_output”,“log.origin”:{“file.name”:“pipeline/client_worker.go”,“file.line”:176},“message”:“failed to publish events: write tcp 10.247.148.20:47778->10.247.2.139:6045: write: broken pipe”,“service.name”:“metricbeat”,“ecs.version”:“1.6.0”}
May 23 16:03:35 vac10appcpe800 metricbeat[1867]: {“log.level”:“error”,“@timestamp”:“2023-05-23T16:03:35.659Z”,“log.logger”:“publisher_pipeline_output”,“log.origin”:{“file.name”:“pipeline/client_worker.go”,“file.line”:176},“message”:“failed to publish events: write tcp 10.247.148.20:42682->10.247.2.139:6044: write: broken pipe”,“service.name”:“metricbeat”,“ecs.version”:“1.6.0”}
May 23 16:03:35 vac10appcpe800 metricbeat[1867]: {“log.level”:“error”,“@timestamp”:“2023-05-23T16:03:35.722Z”,“log.logger”:“publisher_pipeline_output”,“log.origin”:{“file.name”:“pipeline/client_worker.go”,“file.line”:176},“message”:“failed to publish events: write tcp 10.247.148.20:53236->10.247.2.241:6047: write: broken pipe”,“service.name”:“metricbeat”,“ecs.version”:“1.6.0”}
May 23 16:03:36 vac10appcpe800 metricbeat[1867]: {“log.level”:“info”,“@timestamp”:“2023-05-23T16:03:36.785Z”,“log.logger”:“publisher_pipeline_output”,“log.origin”:{“file.name”:“pipeline/client_worker.go”,“file.line”:139},“message”:“Connecting to backoff(async(tcp://aws-logstash-west.ecs.vaec.va.gov:6046))”,“service.name”:“metricbeat”,“ecs.version”:“1.6.0”}
May 23 16:03:36 vac10appcpe800 metricbeat[1867]: {“log.level”:“info”,“@timestamp”:“2023-05-23T16:03:36.785Z”,“log.logger”:“publisher_pipeline_output”,“log.origin”:{“file.name”:“pipeline/client_worker.go”,“file.line”:139},“message”:“Connecting to backoff(async(tcp://aws-logstash-west.ecs.vaec.va.gov:6045))”,“service.name”:“metricbeat”,“ecs.version”:“1.6.0”}
May 23 16:03:36 vac10appcpe800 metricbeat[1867]: {“log.level”:“info”,“@timestamp”:“2023-05-23T16:03:36.786Z”,“log.logger”:“publisher_pipeline_output”,“log.origin”:{“file.name”:“pipeline/client_worker.go”,“file.line”:139},“message”:“Connecting to backoff(async(tcp://aws-logstash-west.ecs.vaec.va.gov:6044))”,“service.name”:“metricbeat”,“ecs.version”:“1.6.0”}
May 23 16:03:36 vac10appcpe800 metricbeat[1867]: {“log.level”:“info”,“@timestamp”:“2023-05-23T16:03:36.786Z”,“log.logger”:“publisher_pipeline_output”,“log.origin”:{“file.name”:“pipeline/client_worker.go”,“file.line”:139},“message”:“Connecting to backoff(async(tcp://aws-logstash-west.ecs.vaec.va.gov:6047))”,“service.name”:“metricbeat”,“ecs.version”:“1.6.0”}
May 23 16:03:36 vac10appcpe800 metricbeat[1867]: {“log.level”:“info”,“@timestamp”:“2023-05-23T16:03:36.796Z”,“log.logger”:“publisher_pipeline_output”,“log.origin”:{“file.name”:“pipeline/client_worker.go”,“file.line”:147},“message”:“Connection to backoff(async(tcp://aws-logstash-west.ecs.vaec.va.gov:6044)) established”,“service.name”:“metricbeat”,“ecs.version”:“1.6.0”}
May 23 16:03:36 vac10appcpe800 metricbeat[1867]: {“log.level”:“info”,“@timestamp”:“2023-05-23T16:03:36.799Z”,“log.logger”:“publisher_pipeline_output”,“log.origin”:{“file.name”:“pipeline/client_worker.go”,“file.line”:147},“message”:“Connection to backoff(async(tcp://aws-logstash-west.ecs.vaec.va.gov:6047)) established”,“service.name”:“metricbeat”,“ecs.version”:“1.6.0”}
May 23 16:03:36 vac10appcpe800 metricbeat[1867]: {“log.level”:“info”,“@timestamp”:“2023-05-23T16:03:36.799Z”,“log.logger”:“publisher_pipeline_output”,“log.origin”:{“file.name”:“pipeline/client_worker.go”,“file.line”:147},“message”:“Connection to backoff(async(tcp://aws-logstash-west.ecs.vaec.va.gov:6046)) established”,“service.name”:“metricbeat”,“ecs.version”:“1.6.0”}
May 23 16:03:36 vac10appcpe800 metricbeat[1867]: {“log.level”:“info”,“@timestamp”:“2023-05-23T16:03:36.799Z”,“log.logger”:“publisher_pipeline_output”,“log.origin”:{“file.name”:“pipeline/client_worker.go”,“file.line”:147},“message”:“Connection to backoff(async(tcp://aws-logstash-west.ecs.vaec.va.gov:6045)) established”,“service.name”:“metricbeat”,“ecs.version”:“1.6.0”}

May 23 18:38:12 vac10appcpe800 packetbeat[1858]: {“log.level”:“error”,“@timestamp”:“2023-05-23T18:38:12.127Z”,“log.logger”:“logstash”,“log.origin”:{“file.name”:“logstash/async.go”,“file.line”:280},“message”:“Failed to publish events caused by: EOF”,“service.name”:“packetbeat”,“ecs.version”:“1.6.0”}
May 23 18:38:52 vac10appcpe800 packetbeat[1858]: {“log.level”:“error”,“@timestamp”:“2023-05-23T18:38:52.951Z”,“log.logger”:“logstash”,“log.origin”:{“file.name”:“logstash/async.go”,“file.line”:280},“message”:“Failed to publish events caused by: client is not connected”,“service.name”:“packetbeat”,“ecs.version”:“1.6.0”}
May 23 18:38:53 vac10appcpe800 packetbeat[1858]: {“log.level”:“error”,“@timestamp”:“2023-05-23T18:38:53.353Z”,“log.logger”:“logstash”,“log.origin”:{“file.name”:“logstash/async.go”,“file.line”:280},“message”:“Failed to publish events caused by: read tcp 10.247.148.20:48658->10.247.2.139:6048: i/o timeout”,“service.name”:“packetbeat”,“ecs.version”:“1.6.0”}
May 23 18:38:54 vac10appcpe800 packetbeat[1858]: {“log.level”:“error”,“@timestamp”:“2023-05-23T18:38:54.292Z”,“log.logger”:“publisher_pipeline_output”,“log.origin”:{“file.name”:“pipeline/client_worker.go”,“file.line”:176},“message”:“failed to publish events: client is not connected”,“service.name”:“packetbeat”,“ecs.version”:“1.6.0”}
May 23 18:39:13 vac10appcpe800 packetbeat[1858]: {“log.level”:“error”,“@timestamp”:“2023-05-23T18:39:13.110Z”,“log.logger”:“logstash”,“log.origin”:{“file.name”:“logstash/async.go”,“file.line”:280},“message”:“Failed to publish events caused by: client is not connected”,“service.name”:“packetbeat”,“ecs.version”:“1.6.0”}
May 23 18:39:14 vac10appcpe800 packetbeat[1858]: {“log.level”:“error”,“@timestamp”:“2023-05-23T18:39:14.877Z”,“log.logger”:“publisher_pipeline_output”,“log.origin”:{“file.name”:“pipeline/client_worker.go”,“file.line”:176},“message”:“failed to publish events: client is not connected”,“service.name”:“packetbeat”,“ecs.version”:“1.6.0”}
May 23 18:39:53 vac10appcpe800 packetbeat[1858]: {“log.level”:“info”,“@timestamp”:“2023-05-23T18:39:53.429Z”,“log.logger”:“publisher_pipeline_output”,“log.origin”:{“file.name”:“pipeline/client_worker.go”,“file.line”:139},“message”:“Connecting to backoff(async(tcp://aws-logstash-west.ecs.vaec.va.gov:6044))”,“service.name”:“packetbeat”,“ecs.version”:“1.6.0”}
May 23 18:39:53 vac10appcpe800 packetbeat[1858]: {“log.level”:“info”,“@timestamp”:“2023-05-23T18:39:53.437Z”,“log.logger”:“publisher_pipeline_output”,“log.origin”:{“file.name”:“pipeline/client_worker.go”,“file.line”:147},“message”:“Connection to backoff(async(tcp://aws-logstash-west.ecs.vaec.va.gov:6044)) established”,“service.name”:“packetbeat”,“ecs.version”:“1.6.0”}
May 23 18:39:54 vac10appcpe800 packetbeat[1858]: {“log.level”:“info”,“@timestamp”:“2023-05-23T18:39:54.437Z”,“log.logger”:“publisher_pipeline_output”,“log.origin”:{“file.name”:“pipeline/client_worker.go”,“file.line”:139},“message”:“Connecting to backoff(async(tcp://aws-logstash-west.ecs.vaec.va.gov:6048))”,“service.name”:“packetbeat”,“ecs.version”:“1.6.0”}
May 23 18:39:54 vac10appcpe800 packetbeat[1858]: {“log.level”:“info”,“@timestamp”:“2023-05-23T18:39:54.451Z”,“log.logger”:“publisher_pipeline_output”,“log.origin”:{“file.name”:“pipeline/client_worker.go”,“file.line”:147},“message”:“Connection to backoff(async(tcp://aws-logstash-west.ecs.vaec.va.gov:6048)) established”,“service.name”:“packetbeat”,“ecs.version”:“1.6.0”}

2

So hard to say for sure as I’ve never worked with the service but this is where I would start. Seems there is some sort of connection or authentication issue between your beats and logstash deployment. It might be worth doing some testing to make sure these services can connect with one another. Do you know that these services are authed so they can connect? Are they in different VPC’s and are there rules to allow the traffic?

That’s a starting point at least :sweat_smile:

1 Like
3

Hey @ddeaderick

In all the log/s files shown, it does state what line that might be causing the issue.

“logstash/async.go”,“file.line”:280} message”:“Failed to publish events caused by

Same with the other beats. I personal look in each corresponding file that the log states and check it out. Just an idea.

if need be, you can always test your logstash file.

 sudo bin/logstash --config.test_and_exit -f config/logstash.conf

Not sure how you configured you logstash file but might try something like this to see if that helps.

ecs_compatibility => disabled