for sending cw logs into opensearch aws assumes its managed service either need arn or same account
I have self hosted opensearch on ec2 running docker containers as usecase is minimal data
if functionbeat works ? read from cw logs and send to opensearch
kinesis + logstash is the only option ?
I currently use the logstash-kinesis-input (requires some form of credentials via IAM for both kinesis and a dynamodb for tracking purposes) for each of my log groups in cloudwatch that is created either automatically or manually I have a lambda function that adds a subscription filter to those log groups to forward to kinesis where logstash then pulls from.
A few things I discovered a few years back when trying to ingest direct from cloudwatch was that I would hit API rate limiting for the cloudwatch API when trying to ingest too much at once so this kenesis solution was necessary.
Hope that helps.
thx jason for reply
how does logstash pull ,thought we have to send to its input on its port like filebeat or do you mean http endpoint and logstash api which kinesis dumps into logstash input ?
theres few kinesis firehose vs data stream so just trying to understand your setup
I actually need simple cloudwatch into logstash but the plugin example doesnt tell how it sends to logstash
theres very little docs on logstash and cloudwatch integration when i was looking around for example working
thank you again
Logstash uses the AWS api to pull the data from kinesis. So as long as the cloud watch subscription filters are on your log group to forward to mimesis, and logstash can successfully connect to kinesis then it should work.
AWS cloud watch → subscription filter → kinesis → logstash pulling from kinesis → opensearch
Think of the log stash configuration as:
Input { kinesis stream} → filter { whatever you need} → output{ open search }