Snapshot performance

[doug_f]

Versions (OpenSearch/Dashboard/Server OS/Browser):

• OpenSearch v3.3.0
• RedHat/Docker

Describe the issue:

Each node at most sends 40mb/s to S3 snapshot repository. Is there a way to increase or configure this? Snapshots are taking far longer than I believe they should.

Configuration:

Most settings are defaults.

indices.recovery.max_bytes_per_sec: 4gb

Relevant Logs or Screenshots:

JVM metrics show heap at ~50%

CPU is at ~20%

The Thread pool for snapshots seems to be maxed at 5. Some nodes have 100+ queued jobs.

[pablo]

@doug_f According to OpenSearch documentation, 40 MB/s is the default snapshot speed.

max_snapshot_bytes_per_sec

s3 repository - Register snapshot repository

[matijasever55]

@doug_f

Describe the issue:

Each node at most sends 40mb/s to S3 snapshot repository. Is there a way to increase or configure this? Snapshots are taking far longer than I believe they should.

Configuration:

Most settings are defaults.

indices.recovery.max_bytes_per_sec: 4gb

Relevant Logs or Screenshots:

JVM metrics show heap at ~50%

CPU is at ~20%

The Thread pool for snapshots seems to be maxed at 5. Some nodes have https://www.plastikatrcek.si/3d-filamenti/ 100+ queued jobs. https://www.shopsy.si/

Yes, the default is very conservative (40mb). You should be able to bump this up significantly (e.g., to 100mb or 200mb) by updating the repository settings. Just keep an eye on your search latency when the snapshots are running at higher speeds.