CronJob in k8s unable to take snapshot backup of OpenSearch 1.3.3

DevOps
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

OpenSearch: 1.3.13

Describe the issue:

CronJob in k8s is unable to take snapshot backup.

If I do the same by running a pod using same image with same steps in docker, snapshot is getting created after executing the shell script.

The job runs successfully with no errors. But the snapshot is not created when I check by using http://localhost:9200/_snapshot/opensearch-dev/_all/ [port forwarding to access the opensearch pod]

Configuration:

  1. Dockerfile to create a simple image with curl
FROM alpine
RUN apk update
RUN apk add curl
RUN mkdir -p /script
COPY snapshot_bkp.sh /script/snapshot_bkp.sh
RUN chmod +x /script/snapshot_bkp.sh
WORKDIR /script 
CMD ["./snapshot_bkp.sh"]
  1. Configmap
#!/bin/sh
set -e
HOST="http://gmsp-es-master:9200"
REPOSITORY="opensearch-dev"
SNAPSHOT="snapshot_$(date +%Y%m%d%H%M)"
INDICES="report1,report2,report3"  # comma-separated list of indices to backup

# Create snapshot with specific indices
curl -X PUT "$HOST/_snapshot/$REPOSITORY/$SNAPSHOT" -H "Accept: application/json" -H "Content-Type: application/json" -d '{
"indices": "'$INDICES'",
"ignore_unavailable": true,
"include_global_state": false
}'
  1. cronJob file
apiVersion: batch/v1
kind: CronJob
metadata:
  name: opensearch-backup
  namespace: test-app-dev
spec:
  # Schedule to run at 2:30 AM every day
  schedule: "00 4 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: opensearch-backup
            image: internal.docker/curl:0.0.1
          volumes:
          - name: backup-script
            configMap:
                name: opensearch-backup-script
          restartPolicy: Never
  1. Configmap
apiVersion: v1
kind: ConfigMap
metadata:
  name: opensearch-backup-script
  namespace: test-app-dev
data:
  backup.sh: |
    #!/bin/sh
    set -e
    HOST="http://gmsp-es-master:9200"
    REPOSITORY="opensearch-dev"
    SNAPSHOT="snapshot_$(date +%Y%m%d%H%M)"
    INDICES="report1,report2,report3"  # comma-separated list of indices to backup

    # Create snapshot with specific indices
    curl -X PUT "$HOST/_snapshot/$REPOSITORY/$SNAPSHOT" -H "Accept: application/json" -H "Content-Type: application/json" -d '{
      "indices": "'$INDICES'",
      "ignore_unavailable": true,
      "include_global_state": false
    }'

Relevant Logs or Screenshots:

N/A

2

I was able to capture the error =

$ k logs opensearch-backup-manual-kyl6h-mnjk4
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   462  100   357  100   105  72872  21432 --:--:-- --:--:-- --:--:--  112k
{"error":{"root_cause":[{"type":"invalid_snapshot_name_exception","reason":"[opensearch-dev:snapshot_20240705Jul07] Invalid snapshot name [snapshot_20240705Jul07], must be lowercase"}],"type":"invalid_snapshot_name_exception","reason":"[opensearch-dev:snapshot_20240705Jul07] Invalid snapshot name [snapshot_20240705Jul07], must be lowercase"},"status":400}

My indices names are in lowercase.
Above output is when I trigger the CronJob manually.

But this step works when I execute from inside the k8s pod manually.
CronJob in k8s causing same problem I guess.

3

@abhyankar The error refers to the snapshot name. Based on your configmap, you’re using date format which generates month names with a capital letter.

That’s why

4

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.