Snapshot performance

OpenSearch
,
1

Versions (OpenSearch/Dashboard/Server OS/Browser):

  • OpenSearch v3.3.0
  • RedHat/Docker

Describe the issue:

Each node at most sends 40mb/s to S3 snapshot repository. Is there a way to increase or configure this? Snapshots are taking far longer than I believe they should.

Configuration:

Most settings are defaults.

indices.recovery.max_bytes_per_sec: 4gb

Relevant Logs or Screenshots:

JVM metrics show heap at ~50%

CPU is at ~20%

The Thread pool for snapshots seems to be maxed at 5. Some nodes have 100+ queued jobs.

2

@doug_f According to OpenSearch documentation, 40 MB/s is the default snapshot speed.

max_snapshot_bytes_per_sec