Entirely lost a cluster because I made a typo in securityconfig

Security
,
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
v2.18 - using operator

Describe the issue:
I entirely lost a cluster after I made a modification in the securityconfig file:

        _meta:
          type: "rolesmapping"
          config_version: 2
        all_access:
          reserved: false
          backend_roles:
          - "admin"
          description: "Maps admin to all_access"
      roles.yml: |-
        _meta:
          type: "roles"
          config_version: 2
        cloudfire-admin:
          reserved: false
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - "indices:*"
          tenant_permissions: []
          description: "Admin role with full access to cluster"
        dashboard_read_only:
          reserved: true
        security_rest_api_access:
          reserved: true
        # Allows users to view monitors, destinations and alerts
        alerting_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/alerting/alerts/get'
            - 'cluster:admin/opendistro/alerting/destination/get'
            - 'cluster:admin/opendistro/alerting/monitor/get'
            - 'cluster:admin/opendistro/alerting/monitor/search'
        # Allows users to view and acknowledge alerts
        alerting_ack_alerts:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/alerting/alerts/*'
        # Allows users to use all alerting functionality
        alerting_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster_monitor'
            - 'cluster:admin/opendistro/alerting/*'
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - 'indices_monitor'
                - 'indices:admin/aliases/get'
                - 'indices:admin/mappings/get'
        # Allow users to read Anomaly Detection detectors and results
        anomaly_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/ad/detector/info'
            - 'cluster:admin/opendistro/ad/detector/search'
            - 'cluster:admin/opendistro/ad/detectors/get'
            - 'cluster:admin/opendistro/ad/result/search'
            - 'cluster:admin/opendistro/ad/tasks/search'
            - 'cluster:admin/opendistro/ad/detector/validate'
            - 'cluster:admin/opendistro/ad/result/topAnomalies'
        # Allows users to use all Anomaly Detection functionality
        anomaly_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster_monitor'
            - 'cluster:admin/opendistro/ad/*'
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - 'indices_monitor'
                - 'indices:admin/aliases/get'
                - 'indices:admin/mappings/get'
        # Allows users to read Notebooks
        notebooks_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/notebooks/list'
            - 'cluster:admin/opendistro/notebooks/get'
        # Allows users to all Notebooks functionality
        notebooks_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/notebooks/create'
            - 'cluster:admin/opendistro/notebooks/update'
            - 'cluster:admin/opendistro/notebooks/delete'
            - 'cluster:admin/opendistro/notebooks/get'
            - 'cluster:admin/opendistro/notebooks/list'
        # Allows users to read observability objects
        observability_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opensearch/observability/get'
        # Allows users to all Observability functionality
        observability_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opensearch/observability/create'
            - 'cluster:admin/opensearch/observability/update'
            - 'cluster:admin/opensearch/observability/delete'
            - 'cluster:admin/opensearch/observability/get'
        # Allows users to read and download Reports
        reports_instances_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/reports/instance/list'
            - 'cluster:admin/opendistro/reports/instance/get'
            - 'cluster:admin/opendistro/reports/menu/download'
        # Allows users to read and download Reports and Report-definitions
        reports_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/reports/definition/get'
            - 'cluster:admin/opendistro/reports/definition/list'
            - 'cluster:admin/opendistro/reports/instance/list'
            - 'cluster:admin/opendistro/reports/instance/get'
            - 'cluster:admin/opendistro/reports/menu/download'
        # Allows users to all Reports functionality
        reports_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/reports/definition/create'
            - 'cluster:admin/opendistro/reports/definition/update'
            - 'cluster:admin/opendistro/reports/definition/on_demand'
            - 'cluster:admin/opendistro/reports/definition/delete'
            - 'cluster:admin/opendistro/reports/definition/get'
            - 'cluster:admin/opendistro/reports/definition/list'
            - 'cluster:admin/opendistro/reports/instance/list'
            - 'cluster:admin/opendistro/reports/instance/get'
            - 'cluster:admin/opendistro/reports/menu/download'
        # Allows users to use all asynchronous-search functionality
        asynchronous_search_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/asynchronous_search/*'
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - 'indices:data/read/search*'
        # Allows users to read stored asynchronous-search results
        asynchronous_search_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/asynchronous_search/get'
        # Allows user to use all index_management actions - ism policies, rollups, transforms
        index_management_full_access:
          reserved: true
          cluster_permissions:
            - "cluster:admin/opendistro/ism/*"
            - "cluster:admin/opendistro/rollup/*"
            - "cluster:admin/opendistro/transform/*"
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - 'indices:admin/opensearch/ism/*'
        # Allows users to use all cross cluster replication functionality at leader cluster
        cross_cluster_replication_leader_full_access:
          reserved: true
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - "indices:admin/plugins/replication/index/setup/validate"
                - "indices:data/read/plugins/replication/changes"
                - "indices:data/read/plugins/replication/file_chunk"
        # Allows users to use all cross cluster replication functionality at follower cluster
        cross_cluster_replication_follower_full_access:
          reserved: true
          cluster_permissions:
            - "cluster:admin/plugins/replication/autofollow/update"
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - "indices:admin/plugins/replication/index/setup/validate"
                - "indices:data/write/plugins/replication/changes"
                - "indices:admin/plugins/replication/index/start"
                - "indices:admin/plugins/replication/index/pause"
                - "indices:admin/plugins/replication/index/resume"
                - "indices:admin/plugins/replication/index/stop"
                - "indices:admin/plugins/replication/index/update"
                - "indices:admin/plugins/replication/index/status_check"

I changed admin user to cloudfire-admin:

[2024-12-21T18:43:29,204][INFO ][o.o.s.p.PrivilegesEvaluator] [opensearch-cluster-masters-0] No cluster-level perm match for User [name=cloudfire-admin, backend_roles=[admin], requestedTenant=null] Resolved [aliases=[*], allIndices=[*], types=[*], originalRequested=[*], remoteIndices=[]] [Action [cluster:monitor/main]] [RolesChecked []]. No permissions for [cluster:monitor/main]

Now i can’t revert to what i just did to return to the ‘admin’ state

Could you please help out to fix this?

Configuration:
see above

Relevant Logs or Screenshots:
see above

2

Hi @simone.benati,

You can change it back using internal_users.yml and update the configuration with securityadmin.sh

see more here:

just a tip, back up your config file before applying changes:

./securityadmin.sh -backup

best,
mj

3

Hi @Mantas , believe me I tried multiple times, following the guide, but inside the container I cannot find the correct certs.

It always returns either that I don’t have the CN=admin because the CN is opensearch-cluster.
I literally don’t have certs named kirk.*

How can I proceed?

Thank you

4

Could you share the output below and the content of your opensearch.yml?

run this in the container of any node:

ls -l /usr/share/opensearch/config

Best,
mj

5

hey there, here’s the output:

ls -l /usr/share/opensearch/config
total 44
-rw-rw---- 1 opensearch opensearch 3041 Dec 15 19:14 jvm.options
drwxr-x--- 2 opensearch opensearch 4096 Oct 28 19:44 jvm.options.d
-rw-rw---- 1 opensearch opensearch  514 Dec 15 19:13 log4j2.properties
drwxr-x--- 2 opensearch opensearch 4096 Oct 31 19:56 opensearch-notifications
drwxr-x--- 2 opensearch opensearch 4096 Oct 31 19:56 opensearch-notifications-core
drwxr-x--- 2 opensearch opensearch 4096 Oct 31 19:56 opensearch-observability
drwxr-x--- 2 opensearch opensearch 4096 Oct 31 19:56 opensearch-performance-analyzer
drwxr-x--- 2 opensearch opensearch 4096 Oct 31 19:56 opensearch-reports-scheduler
drwxr-x--- 2 opensearch opensearch 4096 Oct 31 19:56 opensearch-security
-rw-rw---- 1 opensearch opensearch  196 Dec 22 18:28 opensearch.keystore
-rw-r--r-- 1 root       root       1372 Dec 22 18:28 opensearch.yml
drwxrwxrwt 3 root       root        140 Dec 22 18:28 tls-http
drwxrwxrwt 3 root       root        140 Dec 22 18:28 tls-transport

@Mantas thank you!

6

You should be able to call the Security APIs with the cloudfire-admin user since its mapped to the all_access role. The all_access role can call the Security REST APIs.