Securityadmin detecting securityindex as legacy index incorrectly

shivani · November 10, 2022, 3:49pm

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Opensearch: 2.2.1

Describe the issue:

Installed Opensearch with security plugin enabled (in kubernetes env as helm chart)
Executed securityadmin.sh and the opensearch_security index got initialised successfully.
Re-ran securityadmin.sh with exact same configurations again - Now it detects the security index to be of legacy (6) format.
The configurations are of 2.x Opensearch format only - then how is the plugin detecting it as legacy format on upgrade?

Configuration:

      internal_users.yml:
        ---
        _meta:
          type: "internalusers"
          config_version: 2
        admin:
          reserved: false
          hidden: false
          hash: "$2a$12$VcCDgh2NDk07JGN0rjGbM.Ad41qVR/YFJcgHp0UGns5JDymv..TOG"
          backend_roles:
          - "admin"
        kibanaserver:
          reserved: false
          hash: "$2a$12$4AcgAt3xwOWadA5s5blL6ev39OXDNhmOesEoo33eZtrq2N0YrU3H."

      action_groups.yml:
        ---
        _meta:
          type: "actiongroups"
          config_version: 2

      config_yml:
        ---
        _meta:
          type: "config"
          config_version: 2
        config:
          dynamic:
            kibana:
               multitenancy_enabled: false
               server_username: kibanaserver
            http:
              anonymous_auth_enabled: false
              xff:
                enabled: false
                internalProxies: ".+"
            authc:
              basic_internal_auth_domain:
                http_enabled: true
                transport_enabled: true
                order: 0
                http_authenticator:
                  type: "basic"
                  challenge: true 
                  config: {}
                authentication_backend:
                  type: "intern"
                  config: {}


      roles.yml:
        ---
        _meta:
          type: "roles"
          config_version: 2

        kibana_read_only:
          reserved: false

        security_rest_api_access:
          reserved: false

        kibana_multitenancy_user:
          reserved: false
          hidden: false
          index_permissions:
          - index_patterns:
            - ".kibana_*"
            allowed_actions:
            - manage
            - read
            - delete
            - index
          tenant_permissions:
          - tenant_patterns:
            - global_tenant
            allowed_actions:
            - kibana_all_write
			
      roles_mapping_yml:
        ---
        _meta:
          type: "rolesmapping"
          config_version: 2
        all_access:
          reserved: false
          hidden: false
          backend_roles:
          - "admin"
          description: "Migrated from v6"

        own_index:
          reserved: false
          hidden: false
          users:
          - "*"
        kibana_user:
          reserved: false
          backend_roles:
          - "kibanauser"
          description: "Maps kibanauser to kibana_user role"
        readall:
          reserved: false
          backend_roles:
          - "readall"
        kibana_server:
          reserved: false
          users:
          - "kibanaserver"
        kibana_multitenancy_user:
          reserved: "false"
          hidden: "false"
          users:
          - "*"
      tenants.yml:
        ---
        _meta:
          type: "tenants"
          config_version: 2

      allowlist.yml:
      ---
      _meta:
        type: "allowlist"
        config_version: 2
     
     audit.yml:
      ---
      _meta:
        type: "audit"
        config_version: 2
      config:
        # enable/disable audit logging
        enabled: false

      nodes_dn.yml
      ---
      _meta:
        type: "nodesdn"
        config_version: 2

Relevant Logs or Screenshots:
Logs of 1st run of securityadmin.sh:

Security Admin v7
Will connect to shiv-opensearch-client-d9c8b6545-lf9q5:9200 ... done
Connected as "CN=admin,C=AU"
OpenSearch Version: 2.2.1
Contacting opensearch cluster 'shiv-shiv-es' and wait for YELLOW clusterstate ...
Clustername: shiv-shiv-es
Clusterstate: GREEN
Number of nodes: 3
Number of data nodes: 1
.opendistro_security index does not exists, attempt to create it ... done (0-all replicas)
Populate config from /etc/opensearch/opensearch-security/
Will update '/config' with /etc/opensearch/opensearch-security/config.yml
   SUCC: Configuration for 'config' created or updated
Will update '/roles' with /etc/opensearch/opensearch-security/roles.yml
   SUCC: Configuration for 'roles' created or updated
Will update '/rolesmapping' with /etc/opensearch/opensearch-security/roles_mapping.yml
   SUCC: Configuration for 'rolesmapping' created or updated
Will update '/internalusers' with /etc/opensearch/opensearch-security/internal_users.yml
   SUCC: Configuration for 'internalusers' created or updated
Will update '/actiongroups' with /etc/opensearch/opensearch-security/action_groups.yml
   SUCC: Configuration for 'actiongroups' created or updated
Will update '/tenants' with /etc/opensearch/opensearch-security/tenants.yml
   SUCC: Configuration for 'tenants' created or updated
Will update '/nodesdn' with /etc/opensearch/opensearch-security/nodes_dn.yml
   SUCC: Configuration for 'nodesdn' created or updated
Will update '/whitelist' with /etc/opensearch/opensearch-security/whitelist.yml
   SUCC: Configuration for 'whitelist' created or updated
Will update '/audit' with /etc/opensearch/opensearch-security/audit.yml
   SUCC: Configuration for 'audit' created or updated
SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"]) due to: null
SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"]) due to: null
SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"]) due to: null
Done with success
securityadmin ended

Logs of 2nd run of securityadmin with same configs →

Security Admin v7
Will connect to shiv-opensearch-client-d9c8b6545-lf9q5:9200 ... done
Connected as "CN=admin,C=AU"
OpenSearch Version: 2.2.1
Contacting opensearch cluster 'shiv-shiv-es' and wait for YELLOW clusterstate ...
Clustername: shiv-shiv-es
Clusterstate: GREEN
Number of nodes: 3
Number of data nodes: 1
.opendistro_security index already exists, so we do not need to create one.
Legacy index '.opendistro_security' (ES 6) detected (or forced). You should migrate the configuration!
Populate config from /etc/opensearch/opensearch-security/
Will update '/config' with /etc/opensearch/opensearch-security/config.yml (legacy mode)
   SUCC: Configuration for 'config' created or updated
Will update '/roles' with /etc/opensearch/opensearch-security/roles.yml (legacy mode)
   SUCC: Configuration for 'roles' created or updated
Will update '/rolesmapping' with /etc/opensearch/opensearch-security/roles_mapping.yml (legacy mode)
   SUCC: Configuration for 'rolesmapping' created or updated
Will update '/internalusers' with /etc/opensearch/opensearch-security/internal_users.yml (legacy mode)
   SUCC: Configuration for 'internalusers' created or updated
Will update '/actiongroups' with /etc/opensearch/opensearch-security/action_groups.yml (legacy mode)
   SUCC: Configuration for 'actiongroups' created or updated
Will update '/nodesdn' with /etc/opensearch/opensearch-security/nodes_dn.yml (legacy mode)
   SUCC: Configuration for 'nodesdn' created or updated
Will update '/whitelist' with /etc/opensearch/opensearch-security/whitelist.yml (legacy mode)
   SUCC: Configuration for 'whitelist' created or updated
Will update '/audit' with /etc/opensearch/opensearch-security/audit.yml (legacy mode)
   SUCC: Configuration for 'audit' created or updated
SUCC: Expected 7 config types for node {"updated_config_types":["config","roles","rolesmapping","internalusers","actiongroups","nodesdn","audit"],"updated_config_size":7,"message":null} is 7 (["config","roles","rolesmapping","internalusers","actiongroups","nodesdn","audit"]) due to: null
SUCC: Expected 7 config types for node {"updated_config_types":["config","roles","rolesmapping","internalusers","actiongroups","nodesdn","audit"],"updated_config_size":7,"message":null} is 7 (["config","roles","rolesmapping","internalusers","actiongroups","nodesdn","audit"]) due to: null
SUCC: Expected 7 config types for node {"updated_config_types":["config","roles","rolesmapping","internalusers","actiongroups","nodesdn","audit"],"updated_config_size":7,"message":null} is 7 (["config","roles","rolesmapping","internalusers","actiongroups","nodesdn","audit"]) due to: null
Done with success

Also observed: In the second run, it does not update have a successful updation msg for tenants.yml file. Why is this inconsistency seen with this file?

How do we resolve this from being incorrectly detected as legacy index?

pablo · November 10, 2022, 4:09pm

@shivani securityadmin.sh had some issues up to version 2.2.1. This has been fixed in version 2.3.0.

Please review release notes.

github.com

opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-2.3.0.md

# OpenSearch and Dashboards 2.3.0 Release Notes

## Release Highlights
OpenSearch and OpenSearch Dashboards 2.3.0 unlocks new approaches to data replication, storage, and visualization with three experimental features. These features are disabled by default and can be enabled per the [release documentation](https://opensearch.org/docs/latest).
* Segment replication offers users a new data replication strategy. With segment replication, OpenSearch copies Lucene file segments from the primary shard to its replicas, offering performance improvements for high-ingestion workloads.
* Remote-backed storage lets users deploy cloud storage for increased data durability. Users can back up and restore data from their clusters on a per-index basis using cloud-based storage solutions.
* A new drag-and-drop visualization tool lets users generate different types of visualizations more quickly and intuitively. Users can drag and drop data fields to generate line, bar, area, and metric charts.

## Release Details

OpenSearch and OpenSearch Dashboards 2.3.0 includes the following features, enhancements, bug fixes, infrastructure, documentation, maintenance, and refactoring updates.

OpenSearch [Release Notes](https://github.com/opensearch-project/OpenSearch/blob/main/release-notes/opensearch.release-notes-2.3.0.md).

OpenSearch Dashboards [Release Notes](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/main/release-notes/opensearch-dashboards.release-notes-2.3.0.md).

## FEATURES

### OpenSearch SQL
* Add maketime and makedate datetime functions ([#755](https://github.com/opensearch-project/sql/pull/755))

This file has been truncated. show original

github.com/opensearch-project/security

Fix legacy check in SecurityAdmin

opensearch-project:main ← cwperks:fix-legacy-check-in-security-admin

opened 02:09PM - 25 Aug 22 UTC

cwperks

+123 -1

Signed-off-by: Craig Perkins <cwperx@amazon.com> ### Description Security …Admin tool contains a check to see if the security index is in the legacy v6 format. It works by checking to see if the security index has a property called `security`. If the `security` property exists it is in legacy format, otherwise its in the v7 format. Since OS 2.0.0 the check for the `security` property was removed which made the security admin tool always think the index was in the legacy format. The regression was introduced in this PR: https://github.com/opensearch-project/security/commit/4581a3d76ae3287c0821cd9beb3734b9fdcc57e8#diff-84a766aa2dc8279bc00a6447b9de6d8918a41f6522b150071f3f7d969b8d83c0L719 * Category (Enhancement, New feature, Bug fix, Test fix, Refactoring, Maintenance, Documentation) Bug Fix * Why these changes are required? Fix for a regression in the `securityadmin.sh`. ### Issues Resolved Issue where this was reported: https://github.com/opensearch-project/security/issues/1876 ### Testing Tested by running `securityadmin.sh` before and after the change with 2.2.0 cluster to verify that it is not identified as being in the legacy format. Before the change you will see output from the `securityadmin.sh` tool with the following line `Legacy index '.opendistro_security' (ES 6) detected (or forced). You should migrate the configuration!`, after the change this line does not appear. Also tested and verified by running old ODFE-0.10.0 cluster (backed by ES6) to see the legacy security index format. Here's a sample of the index with ODFE-0.10.0 ``` { ".opendistro_security" : { "aliases" : { }, "mappings" : { "security" : { "properties" : { "actiongroups" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "config" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "internalusers" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "roles" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "rolesmapping" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } } } } }, "settings" : { "index" : { "number_of_shards" : "1", "auto_expand_replicas" : "0-all", "provided_name" : ".opendistro_security", "creation_date" : "1661434679747", "number_of_replicas" : "0", "uuid" : "mBp1PkBuRIC7dE2q93CA2Q", "version" : { "created" : "6080199" } } } } } ``` Here's a sample of the v7 format of the security index: ``` { ".opendistro_security" : { "aliases" : { }, "mappings" : { "properties" : { "actiongroups" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "audit" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "config" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "internalusers" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "nodesdn" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "roles" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "rolesmapping" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "tenants" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "whitelist" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } } } }, "settings" : { "index" : { "number_of_shards" : "1", "auto_expand_replicas" : "0-all", "provided_name" : ".opendistro_security", "creation_date" : "1661434851048", "number_of_replicas" : "1", "uuid" : "URNM7zbnTTaPO0yL1eCVSw", "version" : { "created" : "7100299" } } } } } ``` ### Check List - [X] New functionality includes testing - [ ] New functionality has been documented - [X] Commits are signed per the DCO using --signoff By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check [here](https://github.com/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#developer-certificate-of-origin).

Topic		Replies	Views
OpenSearch Dashboards \| unable to login \| index_not_found_exception no such index Security	4	2645	January 4, 2023
Security plugin multinenancy with multiple kibana/opensearch-dashboard instances Security	7	522	August 27, 2021
Security audit logs timestamp format Security	14	493	October 28, 2022
Tenant Indices Don't Work As Expected - Prevents Reporting Security troubleshoot , configure	3	550	April 4, 2024
Non admin users are unable to access indexed data from Kibana Discovery Security	1	437	May 24, 2021

Securityadmin detecting securityindex as legacy index incorrectly

Related Topics