It seems that opensearch-security data is not being saved to disk

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

Describe the issue:
Hello,

I have installed an OpenSearch cluster using Helm. To ensure data persistence, I set the persistence to true for the master, data, and dashboards.

When I perform a helm uninstall, the Persistent Volumes (PVs) remain intact. After reinstalling and checking the configuration, I can see that the index and policy data are preserved. However, the settings for “actiongroup, tenant, internaluser, role, and role mapping” disappear.

I would appreciate your assistance with this issue.

Thank you!

Configuration:

helm chart.yaml
annotations:
  category: Analytics
  images: |
    - name: opensearch
      image: docker.io/bitnami/opensearch:2.16.0-debian-12-r0
    - name: opensearch-dashboards
      image: docker.io/bitnami/opensearch-dashboards:2.15.0-debian-12-r3
    - name: os-shell
      image: docker.io/bitnami/os-shell:12-debian-12-r27
  licenses: Apache-2.0
apiVersion: v2
appVersion: 2.16.0
dependencies:
- name: common
  repository: oci://registry-1.docker.io/bitnamicharts
  tags:
  - bitnami-common
  version: 2.x.x
description: OpenSearch is a scalable open-source solution for search, analytics,
  and observability. Features full-text queries, natural language processing, custom
  dictionaries, amongst others.
home:
https://bitnami.com
icon:
https://bitnami.com/assets/stacks/opensearch/img/opensearch-stack-220x234.png
keywords:
- opensearch
maintainers:
- name: Broadcom, Inc. All Rights Reserved.
  url:
https://github.com/bitnami/charts
name: opensearch
sources:
-
https://github.com/bitnami/charts/tree/main/bitnami/opensearch
version: 1.2.10

Relevant Logs or Screenshots:
Bitnami: Packaged Applications for Any Platform - Cloud, Container, Virtual Machine

2

I have additional information. When restarting master-0, the tenant information disappears. Upon checking the logs, it seems to be updated with the default security YAML settings. When restarting master-2, it appears to skip the YAML settings. What is the reason for updating the security settings to the initial configuration on master-0

│ │ opensearch ************************************************************************** │ │ opensearch Security Admin v7 │ │ opensearch Will connect to common-opensearch-master-0.common-opensearch-master-hl.xdr-common.svc.cluster.local:9200 ... done │ │ opensearch Connected as "CN=admin" │ │ opensearch OpenSearch Version: 2.16.0 │ │ opensearch Contacting opensearch cluster 'open' and wait for YELLOW clusterstate ... │ │ opensearch Clustername: open │ │ opensearch Clusterstate: GREEN │ │ opensearch Number of nodes: 9 │ │ opensearch Number of data nodes: 6 │ │ opensearch .opendistro_security index already exists, so we do not need to create one. │ │ opensearch Populate config from /opt/bitnami/opensearch/config/opensearch-security/ │ │ opensearch Will update '/config' with /opt/bitnami/opensearch/config/opensearch-security/config.yml │ │ opensearch SUCC: Configuration for 'config' created or updated │ │ opensearch Will update '/roles' with /opt/bitnami/opensearch/config/opensearch-security/roles.yml │ │ opensearch SUCC: Configuration for 'roles' created or updated │ │ opensearch Will update '/rolesmapping' with /opt/bitnami/opensearch/config/opensearch-security/roles_mapping.yml │ │ opensearch SUCC: Configuration for 'rolesmapping' created or updated │ │ opensearch Will update '/internalusers' with /opt/bitnami/opensearch/config/opensearch-security/internal_users.yml │ │ opensearch SUCC: Configuration for 'internalusers' created or updated │ │ opensearch Will update '/actiongroups' with /opt/bitnami/opensearch/config/opensearch-security/action_groups.yml │ │ opensearch SUCC: Configuration for 'actiongroups' created or updated │ │ opensearch Will update '/tenants' with /opt/bitnami/opensearch/config/opensearch-security/tenants.yml │ │ opensearch SUCC: Configuration for 'tenants' created or updated │ │ opensearch Will update '/nodesdn' with /opt/bitnami/opensearch/config/opensearch-security/nodes_dn.yml │ │ opensearch SUCC: Configuration for 'nodesdn' created or updated │ │ opensearch Will update '/whitelist' with /opt/bitnami/opensearch/config/opensearch-security/whitelist.yml │ │ opensearch SUCC: Configuration for 'whitelist' created or updated │ │ opensearch Will update '/audit' with /opt/bitnami/opensearch/config/opensearch-security/audit.yml │ │ opensearch SUCC: Configuration for 'audit' created or updated │ │ opensearch Will update '/allowlist' with /opt/bitnami/opensearch/config/opensearch-security/allowlist.yml │ │ opensearch SUCC: Configuration for 'allowlist' created or updated │ │ opensearch SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","actiongroups","config","internalusers"],"updated_co │ │ opensearch SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","actiongroups","config","internalusers"],"updated_co │ │ opensearch SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","actiongroups","config","internalusers"],"updated_co │ │ opensearch SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","actiongroups","config","internalusers"],"updated_co │ │ opensearch SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","actiongroups","config","internalusers"],"updated_co │ │ opensearch SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","actiongroups","config","internalusers"],"updated_co │ │ opensearch SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","actiongroups","config","internalusers"],"updated_co │ │ opensearch SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","actiongroups","config","internalusers"],"updated
3

When I ran securityadmin.sh on master-0, I noticed that the security data modified by the user was lost.
However, shouldn’t the existing values be updated even when applying the default YAML again?
I don’t understand why the user settings are being erased.

4

I resolved the issue.
In the case of master-0,container set OPENSEARCH_SECURITY_BOOTSTRAP to true and then runsecurityadmin.sh.
When I set OPENSEARCH_SECURITY_BOOTSTRAP to false and restart master-0, the security data (tenants) is retained.

However, I am wondering if it is intentional for securityadmin.sh to always be executed whenever master-0 is restarted.

5

@kkoki When you run securityadmin.sh script to upload the configuration it will always overwrite the existing data in the ‘.opendistro_security’. It will overwrite any changes made through UI.
To avoid that first backup the existing configuration, make changes and then upload back.

6

The issue was that the securityadmin.sh script was set to run when master-0 was active in Bitnami Helm.
I resolved it by overriding the master.command in the Helm chart.
Thank you!