New 2.4.0 installation fails whereas 1.3.6 works

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Opensearch 2.4.0
OpenSearch Dashboards 2.4.0

OpenSearch Helm Chart 2.8.0/2.4.0
OpenSearch Dashboards Helm Chart 2.6.0/2.4.0

Describe the issue:

I’ve installed Opensearch via the official helm charts. Running 1.3.6 results in everything working just fine, meaning, dashboards can communicate with the master cluster

If I do an upgrade to 2.x.x everything also works fine! Dashboard gets updated and manages to communicate with the cluster. However, if I delete my PVs (meaning my storage) and thus do a fresh install, this does not seem to work and it gives the following very descriptive error message:

{"type":"log","@timestamp":"2022-11-28T15:53:41Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]:
 Response Error"}

…this gets repeated forever.

These are the logs right before it fails:

{"type":"log","@timestamp":"2022-11-28T15:28:15Z","tags":["info","plugins-service"],"pid":1,"message":"Plugin \"dataSourceManagement\" has been disabled since the following direct or transitive dependencies are missing or disabled: [dataSource]"}
{"type":"log","@timestamp":"2022-11-28T15:28:16Z","tags":["info","plugins-service"],"pid":1,"message":"Plugin \"dataSource\" is disabled."}
{"type":"log","@timestamp":"2022-11-28T15:28:16Z","tags":["info","plugins-service"],"pid":1,"message":"Plugin \"visTypeXy\" is disabled."}
{"type":"log","@timestamp":"2022-11-28T15:28:16Z","tags":["warning","config","deprecation"],"pid":1,"message":"\"cpu.cgroup.path.override\" is deprecated and has been replaced by \"ops.cGroupOverrides.cpuPath\""}
{"type":"log","@timestamp":"2022-11-28T15:28:16Z","tags":["warning","config","deprecation"],"pid":1,"message":"\"cpuacct.cgroup.path.override\" is deprecated and has been replaced by \"ops.cGroupOverrides.cpuAcctPath\""}
{"type":"log","@timestamp":"2022-11-28T15:28:17Z","tags":["warning","config","deprecation"],"pid":1,"message":"\"opensearch.requestHeadersWhitelist\" is deprecated and has been replaced by \"opensearch.requestHeadersAllowlist\""}
{"type":"log","@timestamp":"2022-11-28T15:28:19Z","tags":["info","plugins-system"],"pid":1,"message":"Setting up [49] plugins: [securityAnalyticsDashboards,alertingDashboards,usageCollection,opensearchDashboardsUsageCollection,opensearchDashboardsLegacy,mapsLegacy,share,opensearchUiShared,legacyExport,embeddable,expressions,data,home,console,apmOss,management,indexPatternManagement,advancedSettings,savedObjects,reportsDashboards,indexManagementDashboards,anomalyDetectionDashboards,dashboard,visualizations,visTypeVega,visTypeTimeline,timeline,visTypeTable,visTypeMarkdown,visBuilder,tileMap,regionMap,customImportMapDashboards,inputControlVis,ganttChartDashboards,visualize,searchRelevanceDashboards,queryWorkbenchDashboards,notificationsDashboards,charts,visTypeVislib,visTypeTimeseries,visTypeTagcloud,visTypeMetric,observabilityDashboards,discover,savedObjectsManagement,securityDashboards,bfetch]"}
{"type":"log","@timestamp":"2022-11-28T15:28:28Z","tags":["info","savedobjects-service"],"pid":1,"message":"Waiting until all OpenSearch nodes are compatible with OpenSearch Dashboards before starting saved objects migrations..."}
{"type":"log","@timestamp":"2022-11-28T15:28:28Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}
{"type":"log","@timestamp":"2022-11-28T15:28:29Z","tags":["error","savedobjects-service"],"pid":1,"message":"Unable to retrieve version information from OpenSearch nodes."}

On the master/data nodes, no error message gets send, so I’m guessing it doesn’t even try to connect. Is there a way to log which service it tries to connect to or at least get something more…verbose?
Configuration:

opensearch_dashboards.yml: |
  server:
    name: opensearch-dashboards
    host: "0"

  opensearch:
    ssl:
      verificationMode: certificate
      certificate: /usr/share/opensearch/config/pki/dashboard.pem
      key: /usr/share/opensearch/config/pki/dashboard.key
      certificateAuthorities: [/usr/share/opensearch/config/pki/ca.pem]
    requestHeadersWhitelist: 
    - "securitytenant"
    - "Authorization"
    - "x-forwarded-for"
    - "x-proxy-user"
    - "x-proxy-roles"

  opensearch_security:
    multitenancy:
      enabled: true
      tenants:
        preferred:
        - "Private"
        - "Global"
    readonly_mode:
      roles: ["kibana_read_only"]
    cookie:
      secure: false
    auth:
      type: "proxy"
    proxycache:
      user_header: "x-proxy-user"
      roles_header: "x-proxy-roles"

After a bit more investigative work, it seems I cannot get client auth to work in 2.x.x

When running the following command, I get “Unauthorized” in 2.x.x but in 1.3.6 everything seems to work just fine…

url --cert /usr/share/opensearch/config/pki/dashboard.pem --key /usr/share/opense`arch/config/pki/dashboard.key --cacert /usr/share/opensearch/config/pki/ca.pem https://cluster-master:9200/ -i

This is the config I use in cluster:

config.yml: |-
          _meta:
            type: "config"
            config_version: 2
          config:
            dynamic:
              http:
                anonymous_auth_enabled: false
                xff:
                  enabled: true
                  internalProxies: '.*' # regex pattern
                  #internalProxies: '.*' # trust all internal proxies, regex pattern
                  remoteIpHeader:  'x-forwarded-for'
              authc:
                proxy_auth_domain:
                  description: "Authenticate via proxy"
                  http_enabled: true
                  transport_enabled: false
                  order: 2
                  http_authenticator:
                    type: proxy
                    challenge: false
                    config:
                      user_header: "x-proxy-user"
                      roles_header: "x-proxy-roles"
                  authentication_backend:
                    type: noop
                clientcert_auth_domain:
                  description: "Authenticate via SSL client certificates"
                  http_enabled: true
                  transport_enabled: true
                  order: 1
                  http_authenticator:
                    type: clientcert
                    config:
                      username_attribute: cn #optional, if omitted DN becomes username
                    challenge: false
                  authentication_backend:
                    type: noop

And this is my role other roles related config files:

roles_mapping.yml

          _meta:
            type: "rolesmapping"
            config_version: 2

          dashboard_role:
            reserved: true
            users:
            - "kibanaserver"
            - "dashboard"

roles.yml

          _meta:
            type: "roles"
            config_version: 2

          dashboard_role:
            reserved: true
            cluster_permissions:
              - 'cluster_monitor'
              - 'cluster:monitor/nodes/info'
              - 'cluster:monitor/state'
            tenant_permissions:
              - tenant_patterns:
                - "global_tenant"
                allowed_actions:
                - "kibana_all_read"
            index_permissions:
              - index_patterns:
                  - '*'
                allowed_actions:
                  - 'indices:admin/get'
                  - 'indices:admin/create'
                  - 'indices:admin/refresh'
                  - 'indices:admin/refresh[s]'
                  - 'indices:admin/aliases'
                  - 'indices:admin/aliases/get'
                  - 'indices:data/read/search'
                  - 'indices:admin/template/put'

Since an upgrade works but a fresh install doesn’t I’m kinda guessing that it might have to do with synchronizing the initial security config…

Good news everyone… the problem is known! And I didn’t know what to look for until I looked if my files where actually synced correctly… surprise, they weren’t! :stuck_out_tongue:

Anyway, for people like me who thaught they were going insane, this is the PR to resolve the issue:

1 Like

Oh yeah, in case somebody wonders, this is the work-around to get it to work:

securityConfig:
    enabled: true
    path: "/usr/share/opensearch/config/opensearch-security"
    config:
      data: