Hi Team,
I have been tetsing logstash s3snssqs plugin to fetch logs from sqs s3.
Earlier the stack was s3 ← sqs ← filebeat → logstash → opensearch
Now we are trying to eliminate filebeat and want to grab logs through the input plugin
It worked fine in dev env ( test ) and now I am trying to implement the same in qa env but while configuring the s3snssqs plugin as input we are receiving a log count is 100’s ( filebeat service was stopped at this point ) where as with filebeat config we are receiving as expected logs ( which is in min 1k logs per minute )
Anyone faced this type of issue with this plugin ( s3snssqs ) or is it something happening in the backend between sqs and the plugin trying to fetch the logs.
Here’s the sample snippet of the logstash config input section
s3snssqs {
region => "us-west-1"
from_sns => false
consumer_threads => 2
s3_default_options => { "endpoint_discovery" => true }
queue => "n11-us-va-03-alb-logs-message-queue"
access_key_id => "#########"
secret_access_key => "#############"
type => "sqs-logs"
tags => ["alb-logs"]
sqs_skip_delete => false
codec => json
s3_options_by_bucket => [
{ bucket_name => "n11-us-va-03-alb-logs-message-queue"
folders => [
{ key => ".*/alb.*"
codec => "json_stream"
type => "ALB"}
]
}
]
}
Can anyone please help me with this please, Thank you