Ability to change payload size send from logstash to opensearch in AWS

grh · September 22, 2021, 4:17pm

Hi All,
is there any option to limit size of bulk request from logstash to OpenSearch Service in AWS?
On our dev envs. we have quite small instances, and we get errors because of too big payload reaching OpenSearch (AWS limits - Amazon OpenSearch Service quotas - Amazon OpenSearch Service )
Maybe some split filter or something similar? - for now I couldn’t find any working solution.
Thanks for hints in advance.

searchymcsearchface · September 23, 2021, 8:55pm

@grh I assume you’re meaning limiting on the Logstash side?

grh · September 24, 2021, 7:07am

yes, Logstash side - as we don’t have access to modify OpenSearch Service in AWS, for now we’re using
“pipeline.batch.size: XX” but this is only workaround and not good for overall performance.
It would be perfect to have ability to set max payload size to i.e. 9MB, and even smallest OSS instances in AWS will cope with it.

searchymcsearchface · September 24, 2021, 2:09pm

Seems fairly reasonable to me. I’m not an expert in the codebase, but I think that this is the line that determines the payload size:

github.com

opensearch-project/logstash-output-opensearch/blob/f1e7e41faa1ee4540592bd3d45530b6f9827d1a5/lib/logstash/outputs/opensearch/http_client.rb#L31


      
            # The criteria used are:
            # 1. We need a number that's less than 100MiB because OpenSearch
            #    won't accept bulks larger than that.
            # 2. It must be large enough to amortize the connection constant
            #    across multiple requests.
            # 3. It must be small enough that even if multiple threads hit this size
            #    we won't use a lot of heap.
            #
            # We wound up agreeing that a number greater than 10 MiB and less than 100MiB
            # made sense. We picked one on the lowish side to not use too much heap.
            TARGET_BULK_BYTES = 20 * 1024 * 1024 # 20MiB
          
          
  class HttpClient
              attr_reader :client, :options, :logger, :pool, :action_count, :recv_count
              # This is here in case we use DEFAULT_OPTIONS in the future
              # DEFAULT_OPTIONS = {
              #   :setting => value
              # }
          
          
#
              # The `options` is a hash where the following symbol keys have meaning:

The comments are enlightening. I’ve found that a lot of the ‘limits’ in the forked code are pretty much… wild estimations. Which kinda drives me nuts, personally. You can experiment if you want to modify the output plugin - it’s not a terribly complex codebase.

That being said, maybe you want to put a feature request in for this:

searchymcsearchface · September 24, 2021, 2:24pm

Actually looks like there is a open PR:

https://github.com/opensearch-project/logstash-output-opensearch/pull/71

grh · September 24, 2021, 2:30pm

yepp, thanks for this, hopefully they will approve and merge this PR

Topic		Replies	Views
How to limit Logstash message size? OpenSearch	2	566	October 26, 2023
Is it possible to get http.max_content_length in OpenSearch cluster hosted on AWS? Open Source Elasticsearch and Kibana	1	2208	August 29, 2023
Logstash bulk 401 error OpenSearch	7	700	August 11, 2023
Need to send data from .log file to AWS opensearch OpenSearch	1	188	December 29, 2023
High heap size on data nodes OpenSearch troubleshoot	5	5144	June 7, 2023

Ability to change payload size send from logstash to opensearch in AWS

Related topics