Logstash input plugin s3snssqs issue

Hi Team,

I have been tetsing logstash s3snssqs plugin to fetch logs from sqs s3.

Earlier the stack was s3 ← sqs ← filebeat → logstash → opensearch

Now we are trying to eliminate filebeat and want to grab logs through the input plugin
It worked fine in dev env ( test ) and now I am trying to implement the same in qa env but while configuring the s3snssqs plugin as input we are receiving a log count is 100’s ( filebeat service was stopped at this point ) where as with filebeat config we are receiving as expected logs ( which is in min 1k logs per minute )

Anyone faced this type of issue with this plugin ( s3snssqs ) or is it something happening in the backend between sqs and the plugin trying to fetch the logs.

Here’s the sample snippet of the logstash config input section

  s3snssqs {
    region                     => "us-west-1"
    from_sns                   => false
    consumer_threads           => 2
    s3_default_options         => { "endpoint_discovery" => true }
    queue                      => "n11-us-va-03-alb-logs-message-queue"
    access_key_id              => "#########"
    secret_access_key          => "#############"
    type                       => "sqs-logs"
    tags                       => ["alb-logs"]
    sqs_skip_delete            => false
    codec                      => json
    s3_options_by_bucket       => [
      { bucket_name => "n11-us-va-03-alb-logs-message-queue"
        folders => [
          { key => ".*/alb.*"
            codec => "json_stream"
            type => "ALB"}

Can anyone please help me with this please, Thank you

There was a issue with Logstash configuration which was reading only first line of the json file in s3
Thanks to this below link. this resolved my issue.

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.