How to move old indices to Microsoft Sharepoint?

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser): OpenSearch (Wazuh)

Describe the issue: Hello! In summary, I’m using Wazuh that’s is based on OpenSearch and I would like some help about retention methods for the indices in the Indexer. I was studying a lot about ISM and Snapshots to help me build a real retention scenario for my company, but unfortunally I didn’t find any forum or docs to help me.

My goal is to have indices less than 60 or 90 days in hot and after 60-90 days move this old indices to a cold storage, this cold storage will be Microsoft Sharepoint. I know that Snapshots have the repositories that have integration to NFS or AWS S3, Azure and other plugins, but I didn’t find anything that could help me to integrate with Sharepoint. I know that Sharepoint have an API, but what should be the workflow to move these snapshots to there?

I couldn’t find anything in Wazuh docs and Opensearch as well. I look for someone that could guide me to have this retention workflow working with Sharepoint to store old data and teach me the best method to do this.

2

@Joao I don’t believe there is currently a connector that would work with SharePoint. It would be a good idea to raise a feature request for this here

In the mean time have to explored the option of saving snapshots using available methods like repository-azure plugin and using something like rclone tool to copy them to sharepoint?

3

@Anthony Hello! My company has more than 100TB of storage available in Sharepoint and was requested to me to make the integration with it. We don’t have azure services for now… But, I thought in a possible solution and I would like your opnion if it would work.

In summary, I know that the snapshots creates the following files and folder:

  • File “index-*”: have the history of snapshot and indices that are attached to it
  • Files “meta-*”: Are snapshots metadata
  • Files “snap-*”: Maps which binary files belong to this snapshot
  • Folder “índices”: Are all indices created with the snapshots

I think that they are the main files for a snapshot to work. So, I thought in creating 2 repositories in the cluster (or in a NFS server).

  1. Main Repository (called “current-snapshots-repo”): This repository would manage snapshots that are being created during the current month. It would be in the path like /tmp/current_snapshots_repo.

  2. Restore Repository (called “restore-snapshots-repo”): This repository would work only to restore snapshots from Sharepoint (via a python script). The goal of create this other repository is to do not corrupt the Main Repository if I want to restore some old indices to the Opensearch server. It would be in the path like /tmp/restore_snapshots_repo.

Workflow would be:

  1. Take snapshot everyday at 1:00 AM (after the creation of a new indice) and save it in current-snapshots-repo
    1. Condition: Always take snapshot for all the indices related to the current month. Example.:
      1. Snapshot (snapshot-2025-08-10) would contain indices from wazuh-alerts-2025-08-01 to wazuh-alerts-2025-08-09
  2. A script would detect modification in file index-* file
  3. A script compresses everything inside the current-snapshots-repo into a .tar.gz file.
  4. A script uploads the compressed file to Sharepoint via Graph API.
  5. ISM Policy is defined to always delete indices older than 90 days in the indexer.

In summay, everyday would take a snapshot and they would been uploaded everyday as well to Sharepoint (after the completion). The script would delete the snapshot of the previous day (via Opensearch API) and let available only the last snapshot made, this is to save storage of the machine.

I tested in my lab environment and it works. Like, I could have snapshots being created in the current-snapshots-repo and also I could restore the old snapshots to the restore-snapshots-repo.

4

If I want to restore old snapshots from Sharepoint, I could simply create a python script that would download to the restore-snapshots-repo and then uncompress it, Opensearch would detected it automatically and I could restore the indices to my indexer.

5

@Joao yes, this approach should work, I would also recommend to mark “restore-snapshot-repo”, see example below:

PUT _snapshot/my_repo 
{ 
   "type": "fs", 
   "settings": { 
            "location": "my_repo_location", 
            "readonly": true 
     }
}

Also, you will need to include in the script checking that the snapshot is finished (“SUCCESS”) before it is copied.

1 Like