Unable to login through Dashboards, API works

OpenSearch Dashboards
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpenSearch 2.14.0
OpenSearch Dashboards 2.14.0
Server OS: Debian Bookworm, hardened.
Browser: Vivaldi 6.7.3329.35 / Firefox 115.11.0esr

Describe the issue:
I’ve got OpenSearch installed and working, with admin setup in internal_users.yml.

Accessing the API on port 9200 works as expected, but I can’t login to the dashboards…

If I do a curl https://192.168.122.121:9200 -u admin:password -k it works fine.
If I try to login as admin in the dashboard at https://192.168.122.121:5601, it loads the Dashboards login page where I enter the credentials and click “Login”.
Then a new popup appears, asking for username and password(looks like standard http auth). When I then enter the credentials again, I just get a page that says:{“statusCode”:401,“error”:“Unauthorized”,“message”:“Authentication Exception”}

At the same time I get a load of 401’s in the dashboards log file(I have set up debug logging):

{"type":"response","@timestamp":"2024-06-06T08:19:25Z","tags":[],"pid":18393,"method":"get","statusCode":401,"req":{"url":"/api/v1/configuration/account","method":"get","headers":{"host":"oser01.domain:5601","user-agent":"Mozilla/5.0 (Windows NT 10.0; rv:109.0) Gecko/20100101 Firefox/115.0","accept":"*/*","accept-language":"en-GB,en;q=0.5","accept-encoding":"gzip, deflate, br","referer":"https://oser01.domain:5601/app/login?","content-type":"application/json","osd-version":"2.14.0","osd-xsrf":"osd-fetch","dnt":"1","connection":"keep-alive","sec-fetch-dest":"empty","sec-fetch-mode":"cors","sec-fetch-site":"same-origin","sec-gpc":"1"},"remoteAddress":"192.168.122.1","userAgent":"Mozilla/5.0 (Windows NT 10.0; rv:109.0) Gecko/20100101 Firefox/115.0","referer":"https://oser01.domain:5601/app/login?"},"res":{"statusCode":401,"responseTime":37,"contentLength":9},"message":"GET /api/v1/configuration/account 401 37ms - 9.0B"}

Configuration:
opensearch.yml:

cluster.name: Oser
node.name: oser01
path.data: /var/lib/opensearch
path.logs: /var/log/opensearch
network.host: 192.168.122.121
http.port: 9200
discovery.type: single-node


plugins.security.ssl.transport.pemcert_filepath: oser01-peer.pem
plugins.security.ssl.transport.pemkey_filepath: oser01-peer-key.pkcs8
plugins.security.ssl.transport.pemtrustedcas_filepath: domain-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: oser01-peer.pem
plugins.security.ssl.http.pemkey_filepath: oser01-peer-key.pkcs8
plugins.security.ssl.http.pemtrustedcas_filepath: domain-ca.pem
plugins.security.allow_unsafe_democertificates: true
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn: ['CN=oser01.domain,L=Allingaabro,C=DK']
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: [all_access, security_rest_api_access]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [.plugins-ml-agent, .plugins-ml-config, .plugins-ml-connector,
  .plugins-ml-controller, .plugins-ml-model-group, .plugins-ml-model, .plugins-ml-task,
  .plugins-ml-conversation-meta, .plugins-ml-conversation-interactions, .plugins-ml-memory-meta,
  .plugins-ml-memory-message, .plugins-ml-stop-words, .opendistro-alerting-config,
  .opendistro-alerting-alert*, .opendistro-anomaly-results*, .opendistro-anomaly-detector*,
  .opendistro-anomaly-checkpoints, .opendistro-anomaly-detection-state, .opendistro-reports-*,
  .opensearch-notifications-*, .opensearch-notebooks, .opensearch-observability, .ql-datasources,
  .opendistro-asynchronous-search-response*, .replication-metadata-store, .opensearch-knn-models,
  .geospatial-ip2geo-data*, .plugins-flow-framework-config, .plugins-flow-framework-templates,
  .plugins-flow-framework-state]
node.max_local_storage_nodes: 3

opensearch-security/config.yml:

_meta:                                                                                                            
  type: "config"                                         
  config_version: 2                                                                                               
                                                                                                                  
config:                                                                                                           
  dynamic:                                               
    http:                                                
      anonymous_auth_enabled: false                                                                               
      xff:                                               
        enabled: false                                   
    authc:                                                                                                        
      kerberos_auth_domain:                                                                                       
        http_enabled: false                                                                                       
        transport_enabled: false                                                                                  
        order: 6                                                                                                  
        http_authenticator:                                                                                       
          type: kerberos                                                                                          
          challenge: true                                                                                         
          config:                                                                                                 
            krb_debug: false                                                                                      
            strip_realm_from_principal: true                                                                      
        authentication_backend:                                                                                   
          type: noop                                     
      basic_internal_auth_domain:
        description: "Authenticate via HTTP Basic against internal users database"                                
        http_enabled: true                                                                                        
        transport_enabled: true                                                                                   
        order: 4                                                                                                  
        http_authenticator:                                                                                       
          type: basic                                                                                             
          challenge: true                                                                                         
        authentication_backend:                          
          type: intern
      proxy_auth_domain:                                                                                          
        description: "Authenticate via proxy"                                                                     
        http_enabled: false                                                                                       
        transport_enabled: false                         
        order: 3                                         
        http_authenticator:                              
          type: proxy                                    
          challenge: false                               
          config:
            user_header: "x-proxy-user"
            roles_header: "x-proxy-roles"
        authentication_backend:
          type: noop
      jwt_auth_domain:
        description: "Authenticate via Json Web Token"
        http_enabled: false
        transport_enabled: false
        order: 0
        http_authenticator:
          type: jwt
          challenge: false
          config:
            signing_key: "base64 encoded HMAC key or public RSA/ECDSA pem key"
            jwt_header: "Authorization"
            jwt_url_parameter: null
            jwt_clock_skew_tolerance_seconds: 30
            roles_key: null
            subject_key: null
        authentication_backend:
          type: noop
      clientcert_auth_domain:
        description: "Authenticate via SSL client certificates"
        http_enabled: false
        transport_enabled: false
        order: 2
        http_authenticator:
          type: clientcert
          config:
          challenge: false
        authentication_backend:
          type: noop
      ldap:
        description: "Authenticate via LDAP or Active Directory"
        http_enabled: false
        transport_enabled: false
        order: 5
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          type: ldap
          config:
            enable_ssl: false
            enable_start_tls: false
            enable_ssl_client_auth: false
            verify_hostnames: true
            hosts:
            - localhost:8389 
            bind_dn: null
            password: null
            userbase: 'ou=people,dc=example,dc=com'
            usersearch: '(sAMAccountName={0})'
            username_attribute: null
    authz:
      roles_from_myldap:
        description: "Authorize via LDAP or Active Directory"
        http_enabled: false
        transport_enabled: false
        authorization_backend:
          type: ldap
          config:
            enable_ssl: false
            enable_start_tls: false
            enable_ssl_client_auth: false
            verify_hostnames: true
            hosts:
            - localhost:8389 
            bind_dn: null
            password: null
            rolebase: 'ou=groups,dc=example,dc=com'
            rolesearch: '(member={0})'
            userroleattribute: null
            userrolename: disabled
            rolename: cn
            resolve_nested_roles: true
            userbase: 'ou=people,dc=example,dc=com'
            usersearch: '(uid={0})'
      roles_from_another_ldap:
        description: "Authorize via another Active Directory"
        http_enabled: false
        transport_enabled: false
        authorization_backend:
          type: ldap

opensearch-security/internal_users.yml:

---
_meta:
  type: "internalusers"
  config_version: 2
admin:
  hash: "some-hash"
  reserved: true
  backend_roles:
  - "admin"
  description: "Demo admin user"
anomalyadmin:
  hash: "some-other-hash"
  reserved: false
  opendistro_security_roles:
  - "anomaly_full_access"
  description: "Demo anomaly admin user, using internal role"
kibanaserver:
  hash: "yet-another-hash"
  reserved: true
  description: "Demo OpenSearch Dashboards user"
...

/etc/opensearch-dashboards/opensearch_dashboards.yml:

---
server.port: 5601
server.host: "oser01.domain"
server.name: "oser01.domain"
opensearch.hosts: ["https://oser01.domain:9200"]
opensearch.username: "kibanaserver"                                                                               
opensearch.password: "kibapass"
server.ssl.enabled: true
server.ssl.certificate: /etc/opensearch-dashboards/oser01-peer.pem
server.ssl.key: /etc/opensearch-dashboards/oser01-peer-key.pkcs8
opensearch.ssl.certificate: /etc/opensearch-dashboards/oser01-peer.pem
opensearch.ssl.key: /etc/opensearch-dashboards/oser01-peer-key.pkcs8
opensearch.ssl.certificateAuthorities: /etc/opensearch-dashboards/domain-ca.pem 
opensearch.ssl.verificationMode: full
logging.dest: "/var/log/opensearch-dashboards/dash.log"
logging.verbose: true
opensearch.requestHeadersWhitelist: ["authorization, securitytenant"]
opensearch_security.multitenancy.enabled: true
opensearch_security.multitenancy.tenants.preferred: ["Private, Global"]
opensearch_security.readonly_mode.roles: ["kibana_read_only"]
opensearch_security.cookie.secure: true

Relevant Logs or Screenshots:
Opensearch dashboards log:

{"type":"response","@timestamp":"2024-06-06T09:25:35Z","tags":[],"pid":18393,"method":"post","statusCode":200,"req":{"url":"/auth/login?dataSourceId=","method":"post","headers":{"host":"oser01.domain:5601","user-agent":"Mozilla/5.0 (Windows NT 10.0; rv:109.0) Gecko/20100101 Firefox/115.0","accept":"*/*","accept-language":"en-GB,en;q=0.5","accept-encoding":"gzip, deflate, br","referer":"https://oser01.domain:5601/app/login?","content-type":"application/json","osd-version":"2.14.0","osd-xsrf":"osd-fetch","content-length":"44","origin":"https://oser01.domain:5601","dnt":"1","connection":"keep-alive","sec-fetch-dest":"empty","sec-fetch-mode":"cors","sec-fetch-site":"same-origin","sec-gpc":"1"},"remoteAddress":"192.168.122.1","userAgent":"Mozilla/5.0 (Windows NT 10.0; rv:109.0) Gecko/20100101 Firefox/115.0","referer":"https://oser01.domain:5601/app/login?"},"res":{"statusCode":200,"responseTime":227,"contentLength":9},"message":"POST /auth/login?dataSourceId= 200 227ms - 9.0B"}
{"type":"response","@timestamp":"2024-06-06T09:25:35Z","tags":[],"pid":18393,"method":"get","statusCode":401,"req":{"url":"/","method":"get","headers":{"host":"oser01.domain:5601","user-agent":"Mozilla/5.0 (Windows NT 10.0; rv:109.0) Gecko/20100101 Firefox/115.0","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8","accept-language":"en-GB,en;q=0.5","accept-encoding":"gzip, deflate, br","referer":"https://oser01.domain:5601/app/login?","dnt":"1","connection":"keep-alive","upgrade-insecure-requests":"1","sec-fetch-dest":"document","sec-fetch-mode":"navigate","sec-fetch-site":"same-origin","sec-fetch-user":"?1","sec-gpc":"1"},"remoteAddress":"192.168.122.1","userAgent":"Mozilla/5.0 (Windows NT 10.0; rv:109.0) Gecko/20100101 Firefox/115.0","referer":"https://oser01.domain:5601/app/login?"},"res":{"statusCode":401,"responseTime":10,"contentLength":9},"message":"GET / 401 10ms - 9.0B"}
{"type":"ops","@timestamp":"2024-06-06T09:25:39Z","tags":[],"pid":18393,"os":{"load":[0,0,0],"mem":{"total":4105408512,"free":1532469248},"uptime":184637.07},"proc":{"uptime":4019.063184166,"mem":{"rss":222085120,"heapTotal":144384000,"heapUsed":128931912,"external":1869995,"arrayBuffers":486544},"delay":1.37779900431633},"load":{"requests":{"5601":{"total":2,"disconnects":0,"statusCodes":{"200":1,"401":1}}},"responseTimes":{"5601":{"avg":118.5,"max":227}},"sockets":{"http":{"total":0},"https":{"total":0}}},"message":"memory: 123.0MB uptime: 1:06:59 load: [0.00 0.00 0.00] delay: 1.378"}
{"type":"ops","@timestamp":"2024-06-06T09:25:44Z","tags":[],"pid":18393,"os":{"load":[0,0,0],"mem":{"total":4105408512,"free":1532227584},"uptime":184642.07},"proc":{"uptime":4024.063801649,"mem":{"rss":221347840,"heapTotal":144384000,"heapUsed":128259384,"external":1867015,"arrayBuffers":483564},"delay":0.15895602107048035},"load":{"requests":{"5601":{"total":0,"disconnects":0,"statusCodes":{}}},"responseTimes":{"5601":{"avg":null,"max":0}},"sockets":{"http":{"total":0},"https":{"total":0}}},"message":"memory: 122.3MB uptime: 1:07:04 load: [0.00 0.00 0.00] delay: 0.159"}

So, obviously I’ve gotten something wrong. But I can’t figure out what it is…

Any ideas are much appreciated, thanks.

/tony

2

Hi @tonyalbers,

Could you run the below and share the output:

curl --insecure -u admin:password -XGET https://oser01.domain:9200/_plugins/_security/api/rolesmapping?pretty

curl --insecure -u kibanaserver:kibapass -XGET https://oser01.domain:9200/_plugins/_security/api/rolesmapping?pretty

Thanks,
mj

3

Hi @Mantas,

Thanks, I’ll see what it says. Can’t until monday though, but I’ll get back to you.

/tony

4 
tba@taggart:~$ curl --insecure -u admin:password -XGET https://oser01.domain:9200/_plugins/_security/api/rolesmapping?pretty
{
  "manage_snapshots" : {
    "hosts" : [ ],
    "users" : [ ],
    "reserved" : false,
    "hidden" : false,
    "backend_roles" : [
      "snapshotrestore"
    ],
    "and_backend_roles" : [ ]
  },
  "logstash" : {
    "hosts" : [ ],
    "users" : [ ],
    "reserved" : false,
    "hidden" : false,
    "backend_roles" : [
      "logstash"
    ],
    "and_backend_roles" : [ ]
  },
  "own_index" : {
    "hosts" : [ ],
    "users" : [
      "*"
    ],
    "reserved" : false,
    "hidden" : false,
    "backend_roles" : [ ],
    "and_backend_roles" : [ ],
    "description" : "Allow full access to an index named like the username"
  },
  "kibana_user" : {
    "hosts" : [ ],
    "users" : [ ],
    "reserved" : false,
    "hidden" : false,
    "backend_roles" : [
      "kibanauser"
    ],
    "and_backend_roles" : [ ],
    "description" : "Maps kibanauser to kibana_user"
  },
  "all_access" : {
    "hosts" : [ ],
    "users" : [ ],
    "reserved" : false,
    "hidden" : false,
    "backend_roles" : [
      "admin"
    ],
    "and_backend_roles" : [ ],
    "description" : "Maps admin to all_access"
  },
  "readall" : {
    "hosts" : [ ],
    "users" : [ ],
    "reserved" : false,
    "hidden" : false,
    "backend_roles" : [
      "readall"
    ],
    "and_backend_roles" : [ ]
  },
  "kibana_server" : {
    "hosts" : [ ],
    "users" : [
      "kibanaserver"
    ],
    "reserved" : true,
    "hidden" : false,
    "backend_roles" : [ ],
    "and_backend_roles" : [ ]
  }
}
tba@taggart:~$

tba@taggart:~$ curl --insecure -u kibanaserver:kspassword -XGET https://oser01.domain:9200/_plugins/_security/api/rolesmapping?pretty
{
  "status" : "FORBIDDEN",
  "message" : "No permission to access REST API: User kibanaserver with Security roles [own_index, kibana_server] does not have any role privileged for admin access. No client TLS certificate found in request"
}
tba@taggart:~$

well… uhm… kibana_server is a static role, so this ought to work, right? Or is it mad because I use a peer certificate and not one that is client only ?

The role is there:

tba@taggart:~$ curl --insecure -u admin:password -XGET https://oser01.domain:9200/_plugins/_security/api/roles?pretty
...
  "kibana_server" : {
    "reserved" : true,
    "hidden" : false,
    "description" : "Provide the minimum permissions for the Kibana server",
    "cluster_permissions" : [
      "cluster_monitor",
      "cluster_composite_ops",
      "manage_point_in_time",
      "indices:admin/template*",
      "indices:admin/index_template*",
      "indices:data/read/scroll*"
    ],
    "index_permissions" : [
      {
        "index_patterns" : [
          ".kibana",
          ".opensearch_dashboards"
        ],
        "fls" : [ ],
        "masked_fields" : [ ],
        "allowed_actions" : [
          "indices_all"
        ]
      },
      {
        "index_patterns" : [
          ".kibana-6",
          ".opensearch_dashboards-6"
        ],
        "fls" : [ ],
        "masked_fields" : [ ],
        "allowed_actions" : [
          "indices_all"
        ]
etc. ...

Weird…

Update:

Ok, if I specify the peer cert in my curl, it works as the ‘kibanaserver’ user…

Now I just need to figure out how to get dashboards to use the peer cert…

5

would you mind running the below as well to see what your users have been assigned to?

curl --insecure -u admin:password -XGET https://oser01.domain:9200/_plugins/_security/authinfo?pretty

curl --insecure -u kibanaserver:kibapass -XGET https://oser01.domain:9200/_plugins/_security/authinfo?pretty

thanks,
mj

6 
tba@taggart:~$ curl --insecure -u admin:password -XGET https://oser01.domain:9200/_plugins/_security/authinfo?pretty
{
  "user" : "User [name=admin, backend_roles=[admin], requestedTenant=null]",
  "user_name" : "admin",
  "user_requested_tenant" : null,
  "remote_address" : "192.168.122.1:47950",
  "backend_roles" : [
    "admin"
  ],
  "custom_attribute_names" : [ ],
  "roles" : [
    "own_index",
    "all_access"
  ],
  "tenants" : {
    "global_tenant" : true,
    "admin_tenant" : true,
    "admin" : true
  },
  "principal" : null,
  "peer_certificates" : "0",
  "sso_logout_url" : null
}
tba@taggart:~$

and:

tba@taggart:~$ curl --insecure -u kibanaserver:kspassword -XGET https://oser01.domain:9200/_plugins/_security/authinfo?pretty
{
  "user" : "User [name=kibanaserver, backend_roles=[], requestedTenant=null]",
  "user_name" : "kibanaserver",
  "user_requested_tenant" : null,
  "remote_address" : "192.168.122.1:34172",
  "backend_roles" : [ ],
  "custom_attribute_names" : [ ],
  "roles" : [
    "own_index",
    "kibana_server"
  ],
  "tenants" : {
    "kibanaserver" : true
  },
  "principal" : null,
  "peer_certificates" : "0",
  "sso_logout_url" : null
}
tba@taggart:~$

So it looks like it maps to kibana_server role as it should…

/tony

7

Hi @tonyalbers, your configuration looks good! What I think is happening - you get 401 from the higher priority Authentication option before the basic_internal_auth_domain.

could you please swap the order for jwt_auth_domain: and basic_internal_auth_domain: and test it again?

like so:

opensearch-security/config.yml:

_meta:                                                                                                            
  type: "config"                                         
  config_version: 2                                                                                               
                                                                                                                  
config:                                                                                                           
  dynamic:                                               
    http:                                                
      anonymous_auth_enabled: false                                                                               
      xff:                                               
        enabled: false                                   
    authc:                                                                                                        
      kerberos_auth_domain:                                                                                       
        http_enabled: false                                                                                       
        transport_enabled: false                                                                                  
        order: 6                                                                                                  
        http_authenticator:                                                                                       
          type: kerberos                                                                                          
          challenge: true                                                                                         
          config:                                                                                                 
            krb_debug: false                                                                                      
            strip_realm_from_principal: true                                                                      
        authentication_backend:                                                                                   
          type: noop                                     
      basic_internal_auth_domain:
        description: "Authenticate via HTTP Basic against internal users database"                                
        http_enabled: true                                                                                        
        transport_enabled: true                                                                                   
        order: 0                                                                                             
        http_authenticator:                                                                                       
          type: basic                                                                                             
          challenge: true                                                                                         
        authentication_backend:                          
          type: intern
      proxy_auth_domain:                                                                                          
        description: "Authenticate via proxy"                                                                     
        http_enabled: false                                                                                       
        transport_enabled: false                         
        order: 3                                         
        http_authenticator:                              
          type: proxy                                    
          challenge: false                               
          config:
            user_header: "x-proxy-user"
            roles_header: "x-proxy-roles"
        authentication_backend:
          type: noop
      jwt_auth_domain:
        description: "Authenticate via Json Web Token"
        http_enabled: false
        transport_enabled: false
        order: 4
        http_authenticator:
          type: jwt
          challenge: false
          config:
            signing_key: "base64 encoded HMAC key or public RSA/ECDSA pem key"
            jwt_header: "Authorization"
            jwt_url_parameter: null
            jwt_clock_skew_tolerance_seconds: 30
            roles_key: null
            subject_key: null
        authentication_backend:
          type: noop
      clientcert_auth_domain:
        description: "Authenticate via SSL client certificates"
        http_enabled: false
        transport_enabled: false
        order: 2
        http_authenticator:
          type: clientcert
          config:
          challenge: false
        authentication_backend:
          type: noop
      ldap:
        description: "Authenticate via LDAP or Active Directory"
        http_enabled: false
        transport_enabled: false
        order: 5
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          type: ldap
          config:
            enable_ssl: false
            enable_start_tls: false
            enable_ssl_client_auth: false
            verify_hostnames: true
            hosts:
            - localhost:8389 
            bind_dn: null
            password: null
            userbase: 'ou=people,dc=example,dc=com'
            usersearch: '(sAMAccountName={0})'
            username_attribute: null
    authz:
      roles_from_myldap:
        description: "Authorize via LDAP or Active Directory"
        http_enabled: false
        transport_enabled: false
        authorization_backend:
          type: ldap
          config:
            enable_ssl: false
            enable_start_tls: false
            enable_ssl_client_auth: false
            verify_hostnames: true
            hosts:
            - localhost:8389 
            bind_dn: null
            password: null
            rolebase: 'ou=groups,dc=example,dc=com'
            rolesearch: '(member={0})'
            userroleattribute: null
            userrolename: disabled
            rolename: cn
            resolve_nested_roles: true
            userbase: 'ou=people,dc=example,dc=com'
            usersearch: '(uid={0})'
      roles_from_another_ldap:
        description: "Authorize via another Active Directory"
        http_enabled: false
        transport_enabled: false
        authorization_backend:
          type: ldap

best,
mj

8

Right, I swapped their orders, so that basic is 0 and jwt is 4. That didn’t change anything when using curl, but the web UI login no longer shows the second username/password prompt but instead goes directly to the 401 error page.

9

I’ve just noticed that the only enabled authentication method on your set-up is basic_internal_auth_domain. I would’ve expected changing the order wouldn’t affect much.

Would you mind running the below:

curl --insecure -u admin:password -XGET https://oser01.domain:9200/_plugins/_security/api/securityconfig?pretty

I’ll run some tests on my lab.

best,
mj

10

Yeah, that was what I thought -but then again I’m pretty new at this :slight_smile:

I’ve run the curl, here’s the resulting output:

tba@taggart:~$ curl --insecure -u admin:password -XGET https://oser01.taggart.domain:9200/_plugins/_security/api/securityconfig?pretty
{
  "config" : {
    "dynamic" : {
      "filtered_alias_mode" : "warn",
      "disable_rest_auth" : false,
      "disable_intertransport_auth" : false,
      "respect_request_indices_options" : false,
      "kibana" : {
        "multitenancy_enabled" : true,
        "private_tenant_enabled" : true,
        "default_tenant" : "",
        "server_username" : "kibanaserver",
        "index" : ".kibana",
        "sign_in_options" : [
          "BASIC"
        ]
      },
      "http" : {
        "anonymous_auth_enabled" : false,
        "xff" : {
          "enabled" : false,
          "internalProxies" : "192\\.168\\.0\\.10|192\\.168\\.0\\.11",
          "remoteIpHeader" : "X-Forwarded-For"
        }
      },
      "authc" : {
        "jwt_auth_domain" : {
          "http_enabled" : false,
          "order" : 4,
          "http_authenticator" : {
            "challenge" : false,
            "type" : "jwt",
            "config" : {
              "signing_key" : "base64 encoded HMAC key or public RSA/ECDSA pem key",
              "jwt_header" : "Authorization",
              "jwt_clock_skew_tolerance_seconds" : 30
            }
          },
          "authentication_backend" : {
            "type" : "noop",
            "config" : { }
          },
          "description" : "Authenticate via Json Web Token"
        },
        "ldap" : {
          "http_enabled" : false,
          "order" : 5,
          "http_authenticator" : {
            "challenge" : false,
            "type" : "basic",
            "config" : { }
          },
          "authentication_backend" : {
            "type" : "ldap",
            "config" : {
              "enable_ssl" : false,
              "enable_start_tls" : false,
              "enable_ssl_client_auth" : false,
              "verify_hostnames" : true,
              "hosts" : [
                "localhost:8389"
              ],
              "userbase" : "ou=people,dc=example,dc=com",
              "usersearch" : "(sAMAccountName={0})"
            }
          },
          "description" : "Authenticate via LDAP or Active Directory"
        },
        "basic_internal_auth_domain" : {
          "http_enabled" : true,
          "order" : 0,
          "http_authenticator" : {
            "challenge" : true,
            "type" : "basic",
            "config" : { }
          },
          "authentication_backend" : {
            "type" : "intern",
            "config" : { }
          },
          "description" : "Authenticate via HTTP Basic against internal users database"
        },
        "proxy_auth_domain" : {
          "http_enabled" : false,
          "order" : 3,
          "http_authenticator" : {
            "challenge" : false,
            "type" : "proxy",
            "config" : {
              "user_header" : "x-proxy-user",
              "roles_header" : "x-proxy-roles"
            }
          },
          "authentication_backend" : {
            "type" : "noop",
            "config" : { }
          },
          "description" : "Authenticate via proxy"
        },
        "clientcert_auth_domain" : {
          "http_enabled" : false,
          "order" : 2,
          "http_authenticator" : {
            "challenge" : false,
            "type" : "clientcert",
            "config" : {
              "username_attribute" : "cn"
            }
          },
          "authentication_backend" : {
            "type" : "noop",
            "config" : { }
          },
          "description" : "Authenticate via SSL client certificates"
        },
        "kerberos_auth_domain" : {
          "http_enabled" : false,
          "order" : 6,
          "http_authenticator" : {
            "challenge" : true,
            "type" : "kerberos",
            "config" : {
              "krb_debug" : false,
              "strip_realm_from_principal" : true
            }
          },
          "authentication_backend" : {
            "type" : "noop",
            "config" : { }
          }
        }
      },
      "authz" : {
        "roles_from_another_ldap" : {
          "http_enabled" : false,
          "authorization_backend" : {
            "type" : "ldap",
            "config" : { }
          },
          "description" : "Authorize via another Active Directory"
        },
        "roles_from_myldap" : {
          "http_enabled" : false,
          "authorization_backend" : {
            "type" : "ldap",
            "config" : {
              "enable_ssl" : false,
              "enable_start_tls" : false,
              "enable_ssl_client_auth" : false,
              "verify_hostnames" : true,
              "hosts" : [
                "localhost:8389"
              ],
              "rolebase" : "ou=groups,dc=example,dc=com",
              "rolesearch" : "(member={0})",
              "userrolename" : "disabled",
              "rolename" : "cn",
              "resolve_nested_roles" : true,
              "userbase" : "ou=people,dc=example,dc=com",
              "usersearch" : "(uid={0})"
            }
          },
          "description" : "Authorize via LDAP or Active Directory"
        }
      },
      "auth_failure_listeners" : { },
      "do_not_fail_on_forbidden" : false,
      "multi_rolespan_enabled" : true,
      "hosts_resolver_mode" : "ip-only",
      "do_not_fail_on_forbidden_empty" : false,
      "on_behalf_of" : {
        "enabled" : false
      }
    }
  }
}

Doesn’t say much…

11

Hi @tonyalbers,

Could you try commenting on the above from your opensearch_dashboards.yml: file and test it again (for testing reasons)?

As per below:

/etc/opensearch-dashboards/opensearch_dashboards.yml:

---
server.port: 5601
server.host: "oser01.domain"
server.name: "oser01.domain"
opensearch.hosts: ["https://oser01.domain:9200"]
opensearch.username: "kibanaserver"                                                                               
opensearch.password: "kibapass"
server.ssl.enabled: true
server.ssl.certificate: /etc/opensearch-dashboards/oser01-peer.pem
server.ssl.key: /etc/opensearch-dashboards/oser01-peer-key.pkcs8
#opensearch.ssl.certificate: /etc/opensearch-dashboards/oser01-peer.pem
#opensearch.ssl.key: /etc/opensearch-dashboards/oser01-peer-key.pkcs8
#opensearch.ssl.certificateAuthorities: /etc/opensearch-dashboards/domain-ca.pem 
#opensearch.ssl.verificationMode: full
logging.dest: "/var/log/opensearch-dashboards/dash.log"
logging.verbose: true
opensearch.requestHeadersWhitelist: ["authorization, securitytenant"]
opensearch_security.multitenancy.enabled: true
opensearch_security.multitenancy.tenants.preferred: ["Private, Global"]
opensearch_security.readonly_mode.roles: ["kibana_read_only"]
opensearch_security.cookie.secure: true

Thanks,
mj

12

Hi mj,
It wouldn’t start until I set:
opensearch.ssl.verificationMode: none

And then I’m back to the double username/password prompts followed by:
{“statusCode”:401,“error”:“Unauthorized”,“message”:“Authentication Exception”}

/tony

Oh yeah, the dashboard debug log says:
{“type”:“error”,“@timestamp”:“2024-06-12T14:58:22Z”,“tags”:[“connection”,“client”,“error”],“pid”:1509,“level”:“error”,“error”:{“message”:“40ECEA220F720000:error:0A000416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:…/deps/openssl/openssl/ssl/record/rec_layer_s3.c:1586:SSL alert number 46\n”,“name”:“Error”,“stack”:“Error: 40ECEA220F720000:error:0A000416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:…/deps/openssl/openssl/ssl/record/rec_layer_s3.c:1586:SSL alert number 46\n”,“code”:“ERR_SSL_SSLV3_ALERT_CERTIFICATE_UNKNOWN”},“message”:“40ECEA220F720000:error:0A000416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:…/deps/openssl/openssl/ssl/record/rec_layer_s3.c:1586:SSL alert number 46\n”}
…

13

Hi @tonyalbers,

Could you please disable the SSL on your OpenSearch dashboards and run the test again, to lock on the issue (TLS or Authentication).

Best,
mj

14

Hi mj,
Thanks for trying to help me out, it’s much appreciated.

I commented out the lines, but this results in the OS-dashboards login prompt just reloading when I enter username/password for admin, saying “Invalid username or password. Please try again.”
And this shows up in dashboard’s debug log:

{"type":"log","@timestamp":"2024-06-17T08:50:20Z","tags":["error","opensearch","data"],"pid":1852,"message":"[ConnectionError]: write EPROTO 40FCC619BE7D0000:error:0A00010B:SSL routines:ssl3_get_record:wrong version number:../deps/openssl/openssl/ssl/record/ssl3_record.c:355:\n"}

I think I’m going to try setting it all up from scratch on another vm, just to see if I end up in the same situation. I’m going to keep this one though, it would be nice to know what went wrong.

/tony

15

Hi @Mantas,
Ok, so I installed from scratch in a new vm and everything is working fine now. The only difference I made was to have Opensearch-dashboards communicate with Opensearch on 127.0.0.1 instead of the primary interface on the vm(192.168.122.121). I can’t say why this matters though. I had disabled the firewall and there were no complains from AppArmor
Well, anyhow, thanks a lot for your help -I now have a better understanding of Opensearch.

/tony

16

Hi @tonyalbers,

Thanks for sharing and glad to hear you made it work, it is indeed interesting. If I happen to figure out an answer ill let you know.

Best,
mj