Changing settings via GUI/DevTools always gives RequestTimeout

OpenSearch Dashboards
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Opensearch Dashboards V2.12.0 (Docker)

Describe the issue:
I have an Opensearch dashboard running in Docker. This has been running fine for many months.
I changed my indexes to data streams a while ago (months) since it’s mostly logging from network devices.
Recently I noticed disk usage going up, and found that it was because my ISM policy was not matching the backing indexes.
So I tried changing my existing ISM policy to also match “.ds-*”, but always got RequestTimeout after 30 seconds.

The same happened when trying to make a new ISM policy via GUI and also via DevTools.
In the logs I do not see any error except for the timeout.
In the dev tools I can execute other commands without any problems btw. (e.g.: GET _cat/allocation?v)

For troubleshooting purposes I tried to create a rollup job and a transform job, but both gave the same result. So it seems to not be specific to the State management policies.

Can anyone maybe point me in the right direction?

Thanks in advance.

Configuration:
opensearch_dashboards.yml

server.name: os_dashboards
server.host: "172.19.0.2"

opensearch.username: ***
opensearch.password: ***
opensearch.requestHeadersWhitelist: [authorization, securitytenant]

opensearch.hosts: ["https://os02:9200","https://os03:9200","https://os04:9200","https://os05:9200"]

opensearch_security.multitenancy.enabled: true
opensearch_security.multitenancy.tenants.preferred: [Private, Global]
opensearch_security.readonly_mode.roles: [kibana_read_only]
# Use this setting if you are running opensearch-dashboards without https
opensearch_security.cookie.secure: false

# Encrypt traffic between the browser and OpenSearch-Dashboards
server.ssl.enabled: true
server.ssl.certificate: "/usr/share/opensearch-dashboards/config/certificates/os-dashboards/os-dashboards.pem"
server.ssl.key: "/usr/share/opensearch-dashboards/config/certificates/os-dashboards/os-dashboards.key"

# Encrypt traffic between OpenSearch-Dashboards and Opensearch
opensearch.ssl.certificateAuthorities: ["/usr/share/opensearch-dashboards/config/certificates/ca/ca.pem"]

Relevant Logs or Screenshots:

Index Management - PolicyService - putPolicy: StatusCodeError: Request Timeout after 30000ms
    at /usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:397:9
    at Timeout.<anonymous> (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:429:7)
    at listOnTimeout (node:internal/timers:569:17)
    at processTimers (node:internal/timers:512:7) {
  status: undefined,
  displayName: 'RequestTimeout',
  body: undefined
}
PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z tags=[] PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z tags=[] pid=1 PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z tags=[] PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z tags=[] pid=1 method=put PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z tags=[] PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z tags=[] pid=1 PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z tags=[] PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z tags=[] pid=1 method=put statusCode=200 PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z tags=[] PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z tags=[] pid=1 PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z tags=[] PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z tags=[] pid=1 method=put PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z tags=[] PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z tags=[] pid=1 PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z tags=[] PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z tags=[] pid=1 method=put statusCode=200 req={"url":"/api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1","method":"put","headers":{"host":"10.194.23.140:5601","connection":"keep-alive","content-length":"872","sec-ch-ua":"\"Google Chrome\";v=\"123\", \"Not:A-Brand\";v=\"8\", \"Chromium\";v=\"123\"","content-type":"application/json","osd-xsrf":"osd-fetch","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36","osd-version":"2.12.0","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","origin":"https://10.194.23.140:5601","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://10.194.23.140:5601/app/opensearch_index_management_dashboards","accept-encoding":"gzip, deflate, br, zstd","accept-language":"en-US,en;q=0.9","securitytenant":"admin_tenant"},"remoteAddress":"10.194.12.148","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36","referer":"https://10.194.23.140:5601/app/opensearch_index_management_dashboards"} PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z tags=[] PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z tags=[] pid=1 PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z tags=[] PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z tags=[] pid=1 method=put PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z tags=[] PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z tags=[] pid=1 PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z tags=[] PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z tags=[] pid=1 method=put statusCode=200 PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z tags=[] PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z tags=[] pid=1 PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z tags=[] PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z tags=[] pid=1 method=put PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z tags=[] PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z tags=[] pid=1 PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z tags=[] PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | PUT /api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1 200 30037ms - 9.0B | type=response @timestamp=2024-04-12T08:40:37Z tags=[] pid=1 method=put statusCode=200 req={"url":"/api/ism/policies/hot_warm_delete?seqNo=0&primaryTerm=1","method":"put","headers":{"host":"10.194.23.140:5601","connection":"keep-alive","content-length":"872","sec-ch-ua":"\"Google Chrome\";v=\"123\", \"Not:A-Brand\";v=\"8\", \"Chromium\";v=\"123\"","content-type":"application/json","osd-xsrf":"osd-fetch","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36","osd-version":"2.12.0","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","origin":"https://10.194.23.140:5601","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://10.194.23.140:5601/app/opensearch_index_management_dashboards","accept-encoding":"gzip, deflate, br, zstd","accept-language":"en-US,en;q=0.9","securitytenant":"admin_tenant"},"remoteAddress":"10.194.12.148","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36","referer":"https://10.194.23.140:5601/app/opensearch_index_management_dashboards"} res={"statusCode":200,"responseTime":30037,"contentLength":9}
2

UPDATE:

I was further investigating and when doing a curl command to directly do the API call on one of the opensearch nodes, I get an error which tells me more.

{"error":{"root_cause":[{"type":"shard_not_in_primary_mode_exception","reason":"CurrentState[STARTED] shard is not in primary mode","index":".opendistro-ism-config","shard":"0","index_uuid":"0pM2WNJyRs2VDcAspYbgAg"}],"type":"retry_on_primary_exception","reason":"shard is not in primary mode","index":".opendistro-ism-config","shard":"0","index_uuid":"0pM2WNJyRs2VDcAspYbgAg","caused_by":{"type":"shard_not_in_primary_mode_exception","reason":"CurrentState[STARTED] shard is not in primary mode","index":".opendistro-ism-config","shard":"0","index_uuid":"0pM2WNJyRs2VDcAspYbgAg"}},"status":500}

When doing more research this might have something to do with the fact that I removed a node from the cluster a while ago.
To me it seems that dashboards is still trying to edit the ISM policy on the node I removed.

Anyone can tell me how I could work around this?

edit: I also get this when trying to make a new ism policy with the api call for some reason

My API call:

curl -XPUT --insecure -u 'admin:***' 'https://x.x.x.x:9200/_plugins/_ism/policies/delete_after_30days' -H 'Content-Type: application/json' -d '
{
  "policy": {
    "description": "Example policy.",
    "default_state": "init",
    "states": [
      {
        "name": "init",
        "transitions": [{
          "state_name": "delete",
          "conditions": {
            "min_index_age": "30d"
          }
          }]
      },
      {
        "name": "delete",
        "actions": [
          {
            "delete": {}
          }
          ]
      }
      
      
      ],
    "ism_template": {
      "index_patterns": [".ds-*"],
      "priority": 100
    }
  }
}'
3

It states that .opendistro-ism-config index has no primary shard. That’s why it can’t read or write the ISM policies.

4

Hi Pablo,

Do you know how I can make a primary shard for .opendistro-ism-config?

I just tested with “GET _plugins/_ism/policies/hot_warm_delete”

and it is able to read it. So it seems only PUT is not working:

{
  "_id": "hot_warm_delete",
  "_version": 1,
  "_seq_no": 0,
  "_primary_term": 1,
  "policy": {
    "policy_id": "hot_warm_delete",
    "description": "hot warm delete workflow",
    "last_updated_time": 1705327779544,
    "schema_version": 18,
    "error_notification": null,
    "default_state": "hot",
    "states": [
      {
        "name": "hot",
        "actions": [
          {
            "retry": {
              "count": 3,
              "backoff": "exponential",
              "delay": "1m"
            },
            "rollover": {
              "min_index_age": "1d",
              "min_primary_shard_size": "30gb",
              "copy_alias": false
            }
          }
        ],
        "transitions": [
          {
            "state_name": "warm"
          }
        ]
      },
      {
        "name": "warm",
        "actions": [
          {
            "retry": {
              "count": 3,
              "backoff": "exponential",
              "delay": "1m"
            },
            "replica_count": {
              "number_of_replicas": 1
            }
          }
        ],
        "transitions": [
          {
            "state_name": "delete",
            "conditions": {
              "min_index_age": "30d"
            }
          }
        ]
      },
      {
        "name": "delete",
        "actions": [
          {
            "retry": {
              "count": 3,
              "backoff": "exponential",
              "delay": "1m"
            },
            "delete": {}
          }
        ],
        "transitions": []
      }
    ],
    "ism_template": [
      {
        "index_patterns": [
          "fortigate-*"
        ],
        "priority": 100,
        "last_updated_time": 1705327779543
      }
    ]
  }
}
5

update:

When I do this:

curl -XGET --insecure -u 'admin:***' 'https://x.x.x.x:9200/.opendistro-ism-config'

I get output:

{".opendistro-ism-config":{"aliases":{},"mappings":{"dynamic":"strict","_meta":{"schema_version":21},"properties":{"managed_index":{"properties":{"change_policy":{"properties":{"is_safe":{"type":"boolean"},"policy_id":{"type":"keyword"},"state":{"type":"keyword"},"user":{"properties":{"backend_roles":{"type":"text","fields":{"keyword":{"type":"keyword"}}},"custom_attribute_names":{"type":"text","fields":{"keyword":{"type":"keyword"}}},"name":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"roles":{"type":"text","fields":{"keyword":{"type":"keyword"}}}}}}},"enabled":{"type":"boolean"},"enabled_time":{"type":"date","format":"strict_date_time||epoch_millis"},"index":{"type":"keyword"},"index_uuid":{"type":"keyword"},"jitter":{"type":"double"},"last_updated_time":{"type":"date","format":"strict_date_time||epoch_millis"},"name":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"policy":{"type":"object","enabled":false},"policy_id":{"type":"keyword"},"policy_primary_term":{"type":"long"},"policy_seq_no":{"type":"long"},"schedule":{"properties":{"cron":{"properties":{"expression":{"type":"keyword"},"schedule_delay":{"type":"long"},"timezone":{"type":"keyword"}}},"interval":{"properties":{"period":{"type":"integer"},"schedule_delay":{"type":"long"},"start_time":{"type":"date","format":"strict_date_time||epoch_millis"},"unit":{"type":"keyword"}}}}}}},"managed_index_metadata":{"properties":{"action":{"properties":{"action_properties":{"properties":{"has_rollup_failed":{"type":"boolean"},"max_num_segments":{"type":"integer"},"rollup_id":{"type":"keyword"},"shrink_action_properties":{"type":"object","enabled":false},"snapshot_name":{"type":"keyword"},"transform_action_properties":{"type":"object","enabled":false}}},"consumed_retries":{"type":"integer"},"failed":{"type":"boolean"},"index":{"type":"integer"},"last_retry_time":{"type":"date","format":"strict_date_time||epoch_millis"},"name":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"start_time":{"type":"date","format":"strict_date_time||epoch_millis"}}},"index":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"index_creation_date":{"type":"date","format":"strict_date_time||epoch_millis"},"index_uuid":{"type":"keyword"},"info":{"type":"object","enabled":false},"policy_completed":{"type":"boolean"},"policy_id":{"type":"keyword"},"policy_primary_term":{"type":"long"},"policy_seq_no":{"type":"long"},"retry_info":{"properties":{"consumed_retries":{"type":"integer"},"failed":{"type":"boolean"}}},"rolled_over":{"type":"boolean"},"rolled_over_index_name":{"type":"keyword"},"state":{"properties":{"name":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"start_time":{"type":"date","format":"strict_date_time||epoch_millis"}}},"step":{"properties":{"name":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"start_time":{"type":"date","format":"strict_date_time||epoch_millis"},"step_status":{"type":"keyword"}}},"transition_to":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"policy":{"properties":{"default_state":{"type":"keyword"},"description":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"error_notification":{"properties":{"channel":{"properties":{"id":{"type":"keyword"}}},"destination":{"properties":{"chime":{"properties":{"url":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"custom_webhook":{"properties":{"header_params":{"type":"object","enabled":false},"host":{"type":"text"},"password":{"type":"text"},"path":{"type":"keyword"},"port":{"type":"integer"},"query_params":{"type":"object","enabled":false},"scheme":{"type":"keyword"},"url":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"username":{"type":"text"}}},"last_update_time":{"type":"date","format":"strict_date_time||epoch_millis"},"name":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"slack":{"properties":{"url":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"type":{"type":"keyword"}}},"message_template":{"type":"object","enabled":false}}},"ism_template":{"properties":{"index_patterns":{"type":"keyword"},"last_updated_time":{"type":"date","format":"strict_date_time||epoch_millis"},"priority":{"type":"long"}}},"last_updated_time":{"type":"date","format":"strict_date_time||epoch_millis"},"policy_id":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"schema_version":{"type":"long"},"states":{"type":"nested","properties":{"actions":{"type":"nested","properties":{"alias":{"properties":{"actions":{"type":"object","enabled":false}}},"allocation":{"properties":{"exclude":{"type":"object","enabled":false},"include":{"type":"object","enabled":false},"require":{"type":"object","enabled":false},"wait_for":{"type":"boolean"}}},"close":{"type":"object"},"custom":{"type":"object","enabled":false},"delete":{"type":"object"},"force_merge":{"properties":{"max_num_segments":{"type":"integer"}}},"index_priority":{"properties":{"priority":{"type":"integer"}}},"notification":{"properties":{"channel":{"properties":{"id":{"type":"keyword"}}},"destination":{"properties":{"chime":{"properties":{"url":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"custom_webhook":{"properties":{"header_params":{"type":"object","enabled":false},"host":{"type":"text"},"password":{"type":"text"},"path":{"type":"keyword"},"port":{"type":"integer"},"query_params":{"type":"object","enabled":false},"scheme":{"type":"keyword"},"url":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"username":{"type":"text"}}},"last_update_time":{"type":"date","format":"strict_date_time||epoch_millis"},"name":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"slack":{"properties":{"url":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"type":{"type":"keyword"}}},"message_template":{"type":"object","enabled":false}}},"open":{"type":"object"},"read_only":{"type":"object"},"read_write":{"type":"object"},"replica_count":{"properties":{"number_of_replicas":{"type":"integer"}}},"retry":{"properties":{"backoff":{"type":"keyword"},"count":{"type":"integer"},"delay":{"type":"keyword"}}},"rollover":{"properties":{"copy_alias":{"type":"boolean"},"min_doc_count":{"type":"keyword"},"min_index_age":{"type":"keyword"},"min_primary_shard_size":{"type":"keyword"},"min_size":{"type":"keyword"}}},"rollup":{"properties":{"ism_rollup":{"properties":{"description":{"type":"text"},"dimensions":{"type":"nested","properties":{"date_histogram":{"properties":{"calendar_interval":{"type":"keyword"},"fixed_interval":{"type":"keyword"},"source_field":{"type":"keyword"},"target_field":{"type":"keyword"},"timezone":{"type":"keyword"}}},"histogram":{"properties":{"interval":{"type":"double"},"source_field":{"type":"keyword"},"target_field":{"type":"keyword"}}},"terms":{"properties":{"source_field":{"type":"keyword"},"target_field":{"type":"keyword"}}}}},"metrics":{"type":"nested","properties":{"metrics":{"type":"nested","properties":{"avg":{"type":"object"},"max":{"type":"object"},"min":{"type":"object"},"sum":{"type":"object"},"value_count":{"type":"object"}}},"source_field":{"type":"keyword"},"target_field":{"type":"keyword"}}},"page_size":{"type":"long"},"target_index":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}}}},"shrink":{"properties":{"aliases":{"type":"object","enabled":false},"force_unsafe":{"type":"boolean"},"max_shard_size":{"type":"keyword"},"num_new_shards":{"type":"integer"},"percentage_of_source_shards":{"type":"double"},"switch_aliases":{"type":"boolean"},"target_index_name_template":{"type":"object","enabled":false}}},"snapshot":{"properties":{"include_global_state":{"type":"boolean"},"repository":{"type":"keyword"},"snapshot":{"type":"keyword"}}},"timeout":{"type":"keyword"},"transform":{"properties":{"ism_transform":{"properties":{"aggregations":{"type":"object","enabled":false},"data_selection_query":{"type":"object","enabled":false},"description":{"type":"text"},"groups":{"properties":{"date_histogram":{"properties":{"calendar_interval":{"type":"keyword"},"fixed_interval":{"type":"keyword"},"source_field":{"type":"keyword"},"target_field":{"type":"keyword"},"timezone":{"type":"keyword"}}},"histogram":{"properties":{"interval":{"type":"double"},"source_field":{"type":"keyword"},"target_field":{"type":"keyword"}}},"terms":{"properties":{"source_field":{"type":"keyword"},"target_field":{"type":"keyword"}}}}},"page_size":{"type":"long"},"target_index":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}}}}}},"name":{"type":"keyword"},"transitions":{"type":"nested","properties":{"conditions":{"type":"object","enabled":false},"state_name":{"type":"keyword"}}}}},"user":{"properties":{"backend_roles":{"type":"text","fields":{"keyword":{"type":"keyword"}}},"custom_attribute_names":{"type":"text","fields":{"keyword":{"type":"keyword"}}},"name":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"roles":{"type":"text","fields":{"keyword":{"type":"keyword"}}}}}}},"rollup":{"properties":{"continuous":{"type":"boolean"},"delay":{"type":"long"},"description":{"type":"text"},"dimensions":{"properties":{"date_histogram":{"properties":{"calendar_interval":{"type":"keyword"},"fixed_interval":{"type":"keyword"},"source_field":{"type":"keyword"},"target_field":{"type":"keyword"},"timezone":{"type":"keyword"}}},"histogram":{"properties":{"interval":{"type":"double"},"source_field":{"type":"keyword"},"target_field":{"type":"keyword"}}},"terms":{"properties":{"source_field":{"type":"keyword"},"target_field":{"type":"keyword"}}}}},"enabled":{"type":"boolean"},"enabled_time":{"type":"date","format":"strict_date_time||epoch_millis"},"last_updated_time":{"type":"date","format":"strict_date_time||epoch_millis"},"metadata_id":{"type":"keyword"},"metrics":{"type":"nested","properties":{"metrics":{"type":"nested","properties":{"avg":{"type":"object"},"max":{"type":"object"},"min":{"type":"object"},"sum":{"type":"object"},"value_count":{"type":"object"}}},"source_field":{"type":"keyword"},"target_field":{"type":"keyword"}}},"page_size":{"type":"long"},"roles":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"rollup_id":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"schedule":{"properties":{"cron":{"properties":{"expression":{"type":"keyword"},"schedule_delay":{"type":"long"},"timezone":{"type":"keyword"}}},"interval":{"properties":{"period":{"type":"integer"},"schedule_delay":{"type":"long"},"start_time":{"type":"date","format":"strict_date_time||epoch_millis"},"unit":{"type":"keyword"}}}}},"schema_version":{"type":"long"},"source_index":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"target_index":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"user":{"properties":{"backend_roles":{"type":"text","fields":{"keyword":{"type":"keyword"}}},"custom_attribute_names":{"type":"text","fields":{"keyword":{"type":"keyword"}}},"name":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"roles":{"type":"text","fields":{"keyword":{"type":"keyword"}}}}}}},"rollup_metadata":{"properties":{"after_key":{"type":"object","enabled":false},"continuous":{"properties":{"next_window_end_time":{"type":"date","format":"strict_date_time||epoch_millis"},"next_window_start_time":{"type":"date","format":"strict_date_time||epoch_millis"}}},"failure_reason":{"type":"text"},"last_updated_time":{"type":"date","format":"strict_date_time||epoch_millis"},"rollup_id":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"stats":{"type":"object","enabled":false},"status":{"type":"keyword"}}},"sm_metadata":{"properties":{"creation":{"properties":{"current_state":{"type":"keyword"},"latest_execution":{"properties":{"end_time":{"type":"date","format":"strict_date_time||epoch_millis"},"info":{"properties":{"cause":{"type":"text"},"message":{"type":"text"}}},"start_time":{"type":"date","format":"strict_date_time||epoch_millis"},"status":{"type":"keyword"}}},"retry":{"properties":{"count":{"type":"integer"}}},"started":{"type":"keyword"},"trigger":{"properties":{"time":{"type":"date","format":"strict_date_time||epoch_millis"}}}}},"current_state":{"type":"keyword"},"deletion":{"properties":{"current_state":{"type":"keyword"},"latest_execution":{"properties":{"end_time":{"type":"date","format":"strict_date_time||epoch_millis"},"info":{"properties":{"cause":{"type":"text"},"message":{"type":"text"}}},"start_time":{"type":"date","format":"strict_date_time||epoch_millis"},"status":{"type":"keyword"}}},"retry":{"properties":{"count":{"type":"integer"}}},"started":{"type":"keyword"},"trigger":{"properties":{"time":{"type":"date","format":"strict_date_time||epoch_millis"}}}}},"name":{"type":"keyword"},"policy_primary_term":{"type":"long"},"policy_seq_no":{"type":"long"}}},"sm_policy":{"properties":{"creation":{"properties":{"schedule":{"properties":{"cron":{"properties":{"expression":{"type":"keyword"},"schedule_delay":{"type":"long"},"timezone":{"type":"keyword"}}}}},"time_limit":{"type":"keyword"}}},"deletion":{"properties":{"condition":{"properties":{"max_age":{"type":"keyword"},"max_count":{"type":"integer"},"min_count":{"type":"integer"}}},"schedule":{"properties":{"cron":{"properties":{"expression":{"type":"keyword"},"schedule_delay":{"type":"long"},"timezone":{"type":"keyword"}}}}},"time_limit":{"type":"keyword"}}},"description":{"type":"text"},"enabled":{"type":"boolean"},"enabled_time":{"type":"date","format":"strict_date_time||epoch_millis"},"last_updated_time":{"type":"date","format":"strict_date_time||epoch_millis"},"name":{"type":"keyword"},"notification":{"properties":{"channel":{"properties":{"id":{"type":"keyword"}}},"conditions":{"properties":{"creation":{"type":"boolean"},"deletion":{"type":"boolean"},"failure":{"type":"boolean"},"time_limit_exceeded":{"type":"boolean"}}}}},"schedule":{"properties":{"interval":{"properties":{"period":{"type":"integer"},"schedule_delay":{"type":"long"},"start_time":{"type":"date","format":"strict_date_time||epoch_millis"},"unit":{"type":"keyword"}}}}},"schema_version":{"type":"long"},"snapshot_config":{"type":"object","enabled":false},"user":{"properties":{"backend_roles":{"type":"text","fields":{"keyword":{"type":"keyword"}}},"custom_attribute_names":{"type":"text","fields":{"keyword":{"type":"keyword"}}},"name":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"roles":{"type":"text","fields":{"keyword":{"type":"keyword"}}}}}}},"transform":{"properties":{"aggregations":{"type":"object","enabled":false},"continuous":{"type":"boolean"},"data_selection_query":{"type":"object","enabled":false},"description":{"type":"text"},"enabled":{"type":"boolean"},"enabled_at":{"type":"date","format":"strict_date_time||epoch_millis"},"groups":{"properties":{"date_histogram":{"properties":{"calendar_interval":{"type":"keyword"},"fixed_interval":{"type":"keyword"},"format":{"type":"keyword"},"source_field":{"type":"keyword"},"target_field":{"type":"keyword"},"timezone":{"type":"keyword"}}},"histogram":{"properties":{"interval":{"type":"double"},"source_field":{"type":"keyword"},"target_field":{"type":"keyword"}}},"terms":{"properties":{"source_field":{"type":"keyword"},"target_field":{"type":"keyword"}}}}},"metadata_id":{"type":"keyword"},"page_size":{"type":"long"},"roles":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"schedule":{"properties":{"cron":{"properties":{"expression":{"type":"keyword"},"schedule_delay":{"type":"long"},"timezone":{"type":"keyword"}}},"interval":{"properties":{"period":{"type":"integer"},"schedule_delay":{"type":"long"},"start_time":{"type":"date","format":"strict_date_time||epoch_millis"},"unit":{"type":"keyword"}}}}},"schema_version":{"type":"long"},"source_index":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"target_index":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"transform_id":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"updated_at":{"type":"date","format":"strict_date_time||epoch_millis"},"user":{"properties":{"backend_roles":{"type":"text","fields":{"keyword":{"type":"keyword"}}},"custom_attribute_names":{"type":"text","fields":{"keyword":{"type":"keyword"}}},"name":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"roles":{"type":"text","fields":{"keyword":{"type":"keyword"}}}}}}},"transform_metadata":{"properties":{"after_key":{"type":"object","enabled":false},"continuous_stats":{"type":"object","enabled":false},"failure_reason":{"type":"text"},"last_updated_at":{"type":"date","format":"strict_date_time||epoch_millis"},"shard_id_to_global_checkpoint":{"type":"object","enabled":false},"stats":{"type":"object","enabled":false},"status":{"type":"keyword"},"transform_id":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}}}},"settings":{"index":{"replication":{"type":"DOCUMENT"},"hidden":"true","number_of_shards":"1","provided_name":".opendistro-ism-config","creation_date":"1705327779571","number_of_replicas":"1","uuid":"0pM2WNJyRs2VDcAspYbgAg","version":{"created":"136307827","upgraded":"136337827"}}}}}

So it is able to find the shard at least.

EDIT:

when I do “GET _cat/shards/.opendistro-ism-config?v”

The output says:

index                  shard prirep state      docs store ip            node
.opendistro-ism-config 0     p      STARTED    1345 2.9mb 10.194.15.141 os04
.opendistro-ism-config 0     r      UNASSIGNED

The node on which I did the direct API call is os04

6

@Sosha The last curl command will be successful as that is index metadata that you can read even when the index data are not accessible.
It looks like your .opendistro-ism-config has only a replica index. You can confirm it by running the below command.

GET _cat/shards/.opendistro-ism-config

The replica shard is synced with the primary replica and used for GET API calls.
I’m not aware of the process of promoting replica shard to primary shard.
I think the only way is to save all the policies, delete the index and recreate all the policies.

7

@pablo you’re my hero! After deleting the index and remaking the policies it works again!
Thank you!

1 Like