Security_Exception on .opendistro-alerting-config index


Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
I’m using a 2.11.1 version of Opensearch (but this issue happened in earlier versions I think)

Describe the issue:
Actually, I’ve got a single-node cluster in a yellow state, due to .opendistro-alerting-* indexes with 1 replicas.

If I try to lower the number of replicas using the following API call :

PUT /.opendistro-alerting-config/_settings
 "index" : {
   "number_of_replicas" : 0

I’ve got the following error message :

  "error": {
    "root_cause": [
        "type": "security_exception",
        "reason": "no permissions for [] and User [name=admin, backend_roles=[admin], requestedTenant=__user__]"
    "type": "security_exception",
    "reason": "no permissions for [] and User [name=admin, backend_roles=[admin], requestedTenant=__user__]"
  "status": 403

I’m using the “admin” account

How to resolve this?


@vincent2mots Try using admin certificate, defined in admin_dn in opensearch.yml, and curl command to modify this index.

curl --insecure --cert <admin_cert.pem> --key <admin_cert.key> 'Content-Type: application/json' PUT https://<OpenSearch_node>:9200/.opendistro-alerting-config/_settings -d `{"index":{"number_of_replicas": 0}}`

What’s the point of changing this? This not the only index that has 1 primary and 1 replica shards out-of-the-box.

health status index                            uuid                   pri rep docs.count docs.deleted store.size
green  open   .opensearch-observability        Dzia5C1oQb-wJ18xrD9ITA   1   1          0            0       416b           208b
green  open   .plugins-ml-config               WY1rqTVkRLyp0mqta6cMVw   1   1          1            0      7.8kb          3.9kb
green  open   security-auditlog-2024.01.29     WVN-gdosTj2k5BUoo3HSKQ   1   1         25            0    276.6kb        140.2kb
green  open   security-auditlog-2024.02.07     eAxgRT6VTKyLATRUGdNZAQ   1   1         18            0    152.4kb         90.9kb
green  open   .opendistro-reports-definitions  EX2UZIOmRUWmBC7vhCVq_A   1   1          1            0      9.9kb          4.9kb
green  open   .opendistro_security             MrpvpRgfQpiK4w5fL4mqNw   1   1         10            0    154.9kb         77.4kb
green  open   .kibana_1                        iPqNFwD_Q-Wv7ChGC_GjbA   1   1          9            0       84kb           42kb
green  open   .opendistro-reports-instances    iphC-Q9RQtmZBUdCWxfPUg   1   1          1            0     12.1kb            6kb
green  open   security-auditlog-2024.02.01     O6k3C10ATiK9DlBIazMsXA   1   1        100            0      349kb        181.9kb
green  open   .opensearch-notifications-config 6XCCZwGzRdixOx643VUepg   1   1          1            0     40.3kb         20.1kb

The yellow state doesn’t affect functionality of your single-node cluster. You just don’t have a backup (replica) of your primary shard.

Thanks @pablo for your answer, I’ll give it a try!

Why do I want to make a change? Because, at some point, I’m having a lot of .opendistro-alterting-history indexes which consume 2 shards each (1 active and 1 unassigned). This way, the limit of 1000 shards can be reached very quickly

You can tune the alerting settings so that less alerting history indices are created. For example:

plugins.alerting.alert_history_max_age: 30d
plugins.alerting.alert_history_max_docs: 2500000
plugins.alerting.alert_history_retention_period: 60d
plugins.alerting.alert_history_rollover_period: 1d

@vincent2mots Agree with @spapadop
Also, the unsigned doesn’t equal active shard. The shard would be created if eligible data node would be added to the cluster. As long as you have single-node deployment, replicas will remain unassigned.
Your indices could have more than 1 replicas configured and they would be in unassigned state too as they couldn’t be placed on any additional node.


Thanks for your answers!

I’m gonna check the cluster settings as suggested to maintain the number of indexes.

However, when I try to reduce the number of replicas for the index .opendistro-alerting-alerts using the curl method, it doesn’t work either.

Do you know why?

I tried this :

curl -v --insecure --cert config/esnode.pem --key config/esnode-key.pem -X PUT -H 'Content-Type: application/json' https://localhost:9200/.opendistro-alerting-alerts/_settings -d '{"index":{"number_of_replicas": 0}}'

Best regards,

@vincent2mots You need to use admin cert instead of the node one.
The admin certificate is defined in the in opensearch.yml.
If you’re using demo configuration and certificates then you should have this in your opensearch.yml
  - CN=kirk,OU=client,O=client,L=test, C=de

The kirk cert is located in the config folder with esnode cert.