Custom Tenants Cannot Save Data but the Global Tenant Can

Any help on this is greatly appreciated!!!

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpenSearch 2.19.0
Dashboard 2.19.0
Ubuntu 22.04 Server
Google Chrome 133

Describe the issue:
Saving any index patterns/dashboards/alerts/reports/ect. under any custom tenant does not work (however saving this data works fine under the global tenant).

Configuration:

dashboards user is currently mapped to the internal role of all_access to help troubleshoot issue. However this does not resolve the saved data not being saved.

Dashboards/index patterns/ect. I have tried to create with both the admin user as well as users mapped to just that tenant, but to no avail… It doesn’t appear to be a permissions issue?

opensearch_dashboards.yml

opensearch.hosts:
  - https://1:9200
  - https://2:9200
  - https://3:9200
  - https://4:9200
  - https://5:9200
  - https://6:9200
  - https://7:9200
  - https://8:9200
opensearch.ssl.verificationMode: full
opensearch.ssl.certificateAuthorities:
  - /u/opensearch-dashboards/config/ca-certs.pem
opensearch.username: dashboards
opensearch.password: XXXX
opensearch.requestHeadersAllowlist: [ 'authorization', 'securitytenant' ]
opensearchDashboards.index: .opensearch_dashboards

# Server
server.host: XXX.XXX.XXX.XXX
server.port: 5601
server.ssl.enabled: true
server.ssl.certificate: /u/opensearch-dashboards/config/node.pem
server.ssl.key: /u/opensearch-dashboards/config/node_private.pem
server.ssl.keyPassphrase: XXXX

# Multitenancy
opensearch_security.multitenancy.enabled: true
opensearch_security.multitenancy.tenants.enable_global: true
opensearch_security.multitenancy.tenants.enable_private: true
#opensearch_security.multitenancy.tenants.preferred: ["Custom Tenant Name"]
opensearch_security.multitenancy.enable_filter: false
opensearch_security.cookie.secure: true

# Session
opensearch_security.cookie.ttl: 86400000  # 24 hours
opensearch_security.session.ttl: 86400000 # 24 hours
opensearch_security.session.keepalive: true

# Logging
logging.dest: "/u/log/dashboards/dash.log"
logging.rotate.enabled: true
logging.rotate.everyBytes: 1073741824
logging.rotate.keepFiles: 10

# Custom
home.disableWelcomeScreen: true

Relevant Logs or Screenshots:
The dash.log and default logging from OpenSearch do not have any WARN or ERROR logs. Dash log only has 200 responses.

Hi @AdamJames,

Would you mind sharing the output of the below:

curl --insecure -u <username>:<password> -XGET https://localhost:9200/_plugins/_security/authinfo?pretty

Any error in the logs when saving fails?

Best,
mj