# Logstash input plugin s3snssqs issue

**URL:** <https://forum.opensearch.org/t/logstash-input-plugin-s3snssqs-issue/16154>\
**Category:** OpenSearch\
**Tags:** troubleshoot\
**Created:** [October 5, 2023, 6:58am UTC](https://forum.opensearch.org/t/logstash-input-plugin-s3snssqs-issue/16154 "2023-10-05T06:58:15Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![raj1209](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/raj1209/32/6115_2.png) [@raj1209](https://forum.opensearch.org/u/raj1209)\
**Post date:** [October 5, 2023, 6:58am UTC](https://forum.opensearch.org/t/logstash-input-plugin-s3snssqs-issue/16154/1 "2023-10-05T06:58:15Z")

</div>

Hi Team,

I have been tetsing logstash **s3snssqs** plugin to fetch logs from sqs s3.

Earlier the stack was s3 ← sqs ← filebeat → logstash → opensearch

Now we are trying to eliminate filebeat and want to grab logs through the input plugin  
It worked fine in dev env ( test ) and now I am trying to implement the same in qa env but while configuring the s3snssqs plugin as input we are receiving a log count is 100’s ( filebeat service was stopped at this point ) where as with filebeat config we are receiving as expected logs ( which is in min 1k logs per minute )

Anyone faced this type of issue with this plugin ( s3snssqs ) or is it something happening in the backend between sqs and the plugin trying to fetch the logs.

Here’s the sample snippet of the logstash config input section

```auto
  s3snssqs {
    region => "us-west-1"
    from_sns => false
    consumer_threads => 2
    s3_default_options => { "endpoint_discovery" => true }
    queue => "n11-us-va-03-alb-logs-message-queue"
    access_key_id => "#########"
    secret_access_key => "#############"
    type => "sqs-logs"
    tags => ["alb-logs"]
    sqs_skip_delete => false
    codec => json
    s3_options_by_bucket => [
      { bucket_name => "n11-us-va-03-alb-logs-message-queue"
        folders => [
          { key => ".*/alb.*"
            codec => "json_stream"
            type => "ALB"}
        ]
      }
    ]
  }

```

Can anyone please help me with this please, Thank you

---

<div class="post-metadata">

**Author:** ![raj1209](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/raj1209/32/6115_2.png) [@raj1209](https://forum.opensearch.org/u/raj1209)\
**Post date:** [October 20, 2023, 7:34am UTC](https://forum.opensearch.org/t/logstash-input-plugin-s3snssqs-issue/16154/2 "2023-10-20T07:34:32Z")

</div>

There was a issue with Logstash configuration which was reading only first line of the json file in s3  
Thanks to this below link. this resolved my issue.

> **[Parse single line multi object json with logstash](https://discuss.elastic.co/t/parse-single-line-multi-object-json-with-logstash/207608/4)**
>
> Move my suggestion to before the json filter: filter { mutate { gsub =\> \["message", "}{", "} {" \] } split { field =\> "message" } json { source =\> "message" } split { field...

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/3/33547ea01a5b12dcca2958411d3edd97ae2ea8c1.png) [@system](https://forum.opensearch.org/u/system)\
**Post date:** [December 19, 2023, 7:35am UTC](https://forum.opensearch.org/t/logstash-input-plugin-s3snssqs-issue/16154/3 "2023-12-19T07:35:01Z")

</div>

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.