Snapshot issues

Open Source Elasticsearch and Kibana
1

Hi,
Im having some issues while trying to take a snapshot of my ES cluster in a docker enviroment.

When I issue this command

PUT _snapshot/my_fs_backup
{
    "type": "fs",
    "settings": {
        "location": "snap"
    }
}

This is the error

{
  "error": {
    "root_cause": [
      {
        "type": "repository_verification_exception",
        "reason": "[my_fs_backup] [[myq68PTIR3GrN5ImB1Qk-w, 'RemoteTransportException[[myq68PT][172.27.0.3:9300][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[my_fs_backup] a file written by master to the store [/usr/share/elasticsearch/repo/snap] cannot be accessed on the node [{myq68PT}{myq68PTIR3GrN5ImB1Qk-w}{0Z6ZYoymT8WVgYd0-9Wtlw}{172.27.0.3}{172.27.0.3:9300}]. This might indicate that the store [/usr/share/elasticsearch/repo/snap] is not shared between this node and the master node or that permissions on the store don't allow reading files written by the master node];']]"
      }
    ],
    "type": "repository_verification_exception",
    "reason": "[my_fs_backup] [[myq68PTIR3GrN5ImB1Qk-w, 'RemoteTransportException[[myq68PT][172.27.0.3:9300][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[my_fs_backup] a file written by master to the store [/usr/share/elasticsearch/repo/snap] cannot be accessed on the node [{myq68PT}{myq68PTIR3GrN5ImB1Qk-w}{0Z6ZYoymT8WVgYd0-9Wtlw}{172.27.0.3}{172.27.0.3:9300}]. This might indicate that the store [/usr/share/elasticsearch/repo/snap] is not shared between this node and the master node or that permissions on the store don't allow reading files written by the master node];']]"
  },
  "status": 500
}

The cluster is sharing a NFS volume via docker, the changes and file created can be seen in both nodes of the cluster. I dont known what else to check.

This is the docker file that I’m using

version: '3'
services:
  odfe-node1:
    image: amazon/opendistro-for-elasticsearch:0.8.0
    container_name: odfe-node1
    environment:
      - cluster.name=odfe-cluster1
      - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
      - reindex.remote.whitelist=10.200.0.220:9201
      - "ES_JAVA_OPTS=-Xms2048m -Xmx2048m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
      - path.repo=["/usr/share/elasticsearch/repo"]
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - odfe-data1:/usr/share/elasticsearch/data
#      - odfe-repo:/usr/share/elasticsearch/repo
      - docker_odfe-nfs-bkp:/usr/share/elasticsearch/repo
    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer
    networks:
      - odfe-net

  odfe-node2:
    image: amazon/opendistro-for-elasticsearch:0.8.0
    container_name: odfe-node2
    environment:
      - cluster.name=odfe-cluster1
      - bootstrap.memory_lock=true
      - reindex.remote.whitelist=10.200.0.220:9201
      - "ES_JAVA_OPTS=-Xms2048m -Xmx2048m"
      - path.repo=/usr/share/elasticsearch/repo
      - discovery.zen.ping.unicast.hosts=odfe-node1
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - odfe-data2:/usr/share/elasticsearch/data
#      - odfe-repo:/usr/share/elasticsearch/repo
      - docker_odfe-nfs-bkp:/usr/share/elasticsearch/repo
    networks:
      - odfe-net

  kibana:
    image: amazon/opendistro-for-elasticsearch-kibana:0.8.0
    container_name: odfe-kibana
    ports:
      - 5601:5601
    expose:
      - "5601"
    environment:
      ELASTICSEARCH_URL: https://odfe-node1:9200
      kibana.defaultAppId: dashboard/4bcad200-5626-11e9-9fc4-e7c2d711c54f
    networks:
      - odfe-net

volumes:
  odfe-data1:
  odfe-data2:
  docker_odfe-nfs-bkp:
    external: true


networks:
  odfe-net: