Unable to get JWT to work

I am using the latest docker for opensearch.

I have made some changes to the docker_compose.yml to start up a single node cluster.
I also have supplied in the Volumes section of the docker_compose.yml

  • ./custom-opensearch.yml:/usr/share/opensearch/config/opensearch.yml where I tried to enable debugging
plugins.security.audit.type: debug

and a custom security config where I’m trying to enable JWT.


---

_meta:
  type: "config"
  config_version: 2

config:
  dynamic:

    http:
      anonymous_auth_enabled: false

    authc:

      basic_internal_auth_domain:
        description: "Authenticate via HTTP Basic against internal users database"
        http_enabled: true
        transport_enabled: true
        order: 4
        http_authenticator:
          type: basic
          challenge: true
        authentication_backend:
          type: intern

      jwt_auth_domain:
        description: "Authenticate via JWT"


        http_enabled: true
        transport_enabled: false
        order: 0
        http_authenticator:
          type: jwt
          challenge: false
          config:
            signing_key: "base64 encoded key"
            jwt_header: "Authorization"
            jwt_url_parameter: null
            subject_key: null
            roles_key: null

        authentication_backend:
          type: noop

    authz:

I tried to apply my changes using an interactive shell on the container:

16:13:16 es-master:~/opensearch$ sudo docker container exec -it opensearch-node1 /bin/sh
sh-4.2$ /usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh -nhnv -icl -cd /usr/share/opensearch/config/opensearch-security -cacert /usr/share/opensearch/config/root-ca.pem -cert /usr/share/opensearch/config/kirk.pem -key /usr/share/opensearch/config/kirk-key.pem
Security Admin v7
Will connect to localhost:9300 ... done
Connected as CN=kirk,OU=client,O=client,L=test,C=de
OpenSearch Version: 1.3.2
OpenSearch Security Version: 1.3.2.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Clustername: opensearch-cluster
Clusterstate: YELLOW
Number of nodes: 1
Number of data nodes: 1
.opendistro_security index already exists, so we do not need to create one.
Populate config from /usr/share/opensearch/config/opensearch-security/
Will update '_doc/config' with /usr/share/opensearch/config/opensearch-security/config.yml
   SUCC: Configuration for 'config' created or updated
Will update '_doc/roles' with /usr/share/opensearch/config/opensearch-security/roles.yml
   SUCC: Configuration for 'roles' created or updated
Will update '_doc/rolesmapping' with /usr/share/opensearch/config/opensearch-security/roles_mapping.yml
   SUCC: Configuration for 'rolesmapping' created or updated
Will update '_doc/internalusers' with /usr/share/opensearch/config/opensearch-security/internal_users.yml
   SUCC: Configuration for 'internalusers' created or updated
Will update '_doc/actiongroups' with /usr/share/opensearch/config/opensearch-security/action_groups.yml
   SUCC: Configuration for 'actiongroups' created or updated
Will update '_doc/tenants' with /usr/share/opensearch/config/opensearch-security/tenants.yml
   SUCC: Configuration for 'tenants' created or updated
Will update '_doc/nodesdn' with /usr/share/opensearch/config/opensearch-security/nodes_dn.yml
   SUCC: Configuration for 'nodesdn' created or updated
Will update '_doc/whitelist' with /usr/share/opensearch/config/opensearch-security/whitelist.yml
   SUCC: Configuration for 'whitelist' created or updated
Done with success
sh-4.2$

But the docker container log suggest that my attempt to install JWT failed.

 [2022-05-23T20:14:45,906][WARN ][o.o.s.h.HTTPBasicAuthenticator] [opensearch-node1] No 'Basic Authorization' header, send 401 and 'WWW-Authenticate Basic'
16:14:35 es-master:~/opensearch$ curl -v -k -H "Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJyb2xlcyI6ImFkbWluIiwiaXNzIjoiaHR0cHM6Ly9sb2NhbGhvc3QiLCJzdWIiOiJhZG1pbiIsImV4cCI6MTU1NDUyMjQzMH0.cFuVSeylBMHoudZjptUsuH529ipBjuxLLj6TSoghZY8"  https://localhost:9200/_cat/nodes
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 9200 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Unknown (8):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Client hello (1):
* TLSv1.3 (OUT), TLS Unknown, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS Unknown, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: DC=de; L=test; O=node; OU=node; CN=node-0.example.com
*  start date: Apr 22 03:43:47 2018 GMT
*  expire date: Apr 19 03:43:47 2028 GMT
*  issuer: DC=com; DC=example; O=Example Com Inc.; OU=Example Com Inc. Root CA; CN=Example Com Inc. Root CA
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
> GET /_cat/nodes HTTP/1.1
> Host: localhost:9200
> User-Agent: curl/7.58.0
> Accept: */*
> Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJyb2xlcyI6ImFkbWluIiwiaXNzIjoiaHR0cHM6Ly9sb2NhbGhvc3QiLCJzdWIiOiJhZG1pbiIsImV4cCI6MTU1NDUyMjQzMH0.cFuVSeylBMHoudZjptUsuH529ipBjuxLLj6TSoghZY8
>
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
< HTTP/1.1 401 Unauthorized
< WWW-Authenticate: Basic realm="OpenSearch Security"
< content-type: text/plain; charset=UTF-8
< content-length: 12
<
* Connection #0 to host localhost left intact

IIUC, JWT should accept the “Authorization: Bearer ”, but instead the server seems to expect only "Authorization: Basic .

Note, that http basic auth does work as expected for the out of the box creds:

16:22:47 es-master:~/opensearch$ curl -k -u admin:admin https://localhost:9200/
{
  "name" : "opensearch-node1",
  "cluster_name" : "opensearch-cluster",
  "cluster_uuid" : "jY8qm053R6-Gh95YZXK2Mg",
  "version" : {
    "distribution" : "opensearch",
    "number" : "1.3.2",
    "build_type" : "tar",
    "build_hash" : "6febcf7b53ff189de767e460e905e9e5aeecc8cb",
    "build_date" : "2022-05-04T03:58:48.328641Z",
    "build_snapshot" : false,
    "lucene_version" : "8.10.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

And deliberately using the wrong password gives me the expected Unauthorized:

$ curl -k -u dadmin:admin https://localhost:9200/
Unauthorized

@iamthealex-es Is that your current sg_config.yml? If yes, then JWT is not enabled and not configured. Only basic auth is enabled, that’s why curl got a response using basic authentication and admin:admin user:password.
Check the OpenSearch documentation.

Thank @pablo , but if you scroll down in the config file I pasted above
you will see that I do have JWT configured

      jwt_auth_domain:
        description: "Authenticate via JWT"


        http_enabled: false
        transport_enabled: false
        order: 0
        http_authenticator:
          type: jwt
          challenge: false
          config:
            signing_key: "base64 encoded key"
            jwt_header: "Authorization"
            jwt_url_parameter: null
            subject_key: null
            roles_key: null

        authentication_backend:
          type: noop

@pablo, and since I essentially have the config snippet already from the link you posted, I think it more likely that my attempt to refresh the security config with the JWT stanza is flawed in some way.

And possibly related with my struggles to make JWT work:

In desperation, I tried removing the Basic Auth section leaving in only the JWT section.
I can successfully run securityadmin.sh, but when I try the curl, I get an error about

curl -k -u admin:admin https://localhost:9200/
OpenSearch Security not initialized

I really think there is something wrong with the way I’m invoking securityadmin.sh, but I can’t figure out why I don’t get any error messages:

sh-4.2$ /usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh -nhnv -icl -cd /usr/share/opensearch/config/opensearch-security -cacert /usr/share/opensearch/config/root-ca.pem -cert /usr/share/opensearch/config/kirk.pem -key /usr/share/opensearch/config/kirk-key.pem
Security Admin v7
Will connect to localhost:9300 ... done
Connected as CN=kirk,OU=client,O=client,L=test,C=de
OpenSearch Version: 1.3.2
OpenSearch Security Version: 1.3.2.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Clustername: opensearch-cluster
Clusterstate: YELLOW
Number of nodes: 1
Number of data nodes: 1
.opendistro_security index already exists, so we do not need to create one.
Populate config from /usr/share/opensearch/config/opensearch-security/
Will update '_doc/config' with /usr/share/opensearch/config/opensearch-security/config.yml
   SUCC: Configuration for 'config' created or updated
Will update '_doc/roles' with /usr/share/opensearch/config/opensearch-security/roles.yml
   SUCC: Configuration for 'roles' created or updated
Will update '_doc/rolesmapping' with /usr/share/opensearch/config/opensearch-security/roles_mapping.yml
   SUCC: Configuration for 'rolesmapping' created or updated
Will update '_doc/internalusers' with /usr/share/opensearch/config/opensearch-security/internal_users.yml
   SUCC: Configuration for 'internalusers' created or updated
Will update '_doc/actiongroups' with /usr/share/opensearch/config/opensearch-security/action_groups.yml
   SUCC: Configuration for 'actiongroups' created or updated
Will update '_doc/tenants' with /usr/share/opensearch/config/opensearch-security/tenants.yml
   SUCC: Configuration for 'tenants' created or updated
Will update '_doc/nodesdn' with /usr/share/opensearch/config/opensearch-security/nodes_dn.yml
   SUCC: Configuration for 'nodesdn' created or updated
Will update '_doc/whitelist' with /usr/share/opensearch/config/opensearch-security/whitelist.yml
   SUCC: Configuration for 'whitelist' created or updated
Done with success

@iamthealex-es I’ve checked your config. The only issue I can see is the roles_key value.

Try

roles_key: roles

instead of

roles_key: null

I tried your suggestion, I first started with

roles_key: "roles"

and I got a parser complaint about a space not being allowed:

[2022-05-24T15:45:00,075][INFO ][o.o.s.s.ConfigHelper     ] [opensearch-node1] Will update 'config' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/config.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
 [2022-05-24T15:45:00,289][INFO ][o.o.s.s.ConfigHelper     ] [opensearch-node1] Index .opendistro_security already contains doc with id config, skipping update.

I then tried to do it without quotes:

roles_key: roles

But now my elasticsearch is remembering the failure and won’t let me try a new config.

[2022-05-24T15:45:00,075][INFO ][o.o.s.s.ConfigHelper     ] [opensearch-node1] Will update 'config' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/config.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
 [2022-05-24T15:45:00,289][INFO ][o.o.s.s.ConfigHelper     ] [opensearch-node1] Index .opendistro_security already contains doc with id config, skipping update.

I tried to go back to just using basic auth, but I am stuck with the non-working config.
I get this repeated forever:

opensearch-dashboards    | {"type":"log","@timestamp":"2022-05-24T15:53:37Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}
opensearch-node1         | [2022-05-24T15:53:39,684][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin)
opensearch-node1         | [2022-05-24T15:53:39,691][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin)
opensearch-node1         | [2022-05-24T15:53:39,697][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin)
opensearch-node1         | [2022-05-24T15:53:39,700][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin)

I tried to connect to the instance and re-rerun securityadmin.sh as I posted originally, but that didn’t help.

Is there a way to completely reset the security ?

@iamthealex-es Could you post your most recent config.yml?

@iamthealex-es Also please share your securityadmin.sh command and the output.

I am feeling stupid.

I have this in my docker-compose:

    volumes:
      - opensearch-data1:/usr/share/opensearch/data
      - ./root-ca.pem:/usr/share/opensearch/config/root-ca.pem
      - ./admin.pem:/usr/share/opensearch/config/admin.pem
      - ./admin-key.pem:/usr/share/opensearch/config/admin-key.pem
      - ./kirk.pem:/usr/share/opensearch/config/kirk.pem
      - ./kirk-key.pem:/usr/share/opensearch/config/kirk-key.pem
      - ./esnode.pem:/usr/share/opensearch/config/esnode.pem
      - ./esnode-key.pem:/usr/share/opensearch/config/esnode-key.pem
      - ./custom-opensearch.yml:/usr/share/opensearch/config/opensearch.yml
      - ./custom-security-config.yml:/usr/share/opensearch/config/opensearch-security/config.yml
      - ./internal_users.yml:/usr/share/opensearch/config/opensearch-security/internal_users.yml
      - ./roles_mapping.yml:/usr/share/opensearch/config/opensearch-security/roles_mapping.yml
      - ./tenants.yml:/usr/share/opensearch/config/opensearch-security/tenants.yml
      - ./roles.yml:/usr/share/opensearch/config/opensearch-security/roles.yml
      - ./action_groups.yml:/usr/share/opensearch/config/opensearch-security/action_groups.yml
      - ./nodes_dn.yml:/usr/share/opensearch/config/opensearch-security/nodes_dn.yml
      - ./whitelist.yml:/usr/share/opensearch/config/opensearch-security/whitelist.yml

And the contents of my custom-security-config.yml are:


---

_meta:
  type: "config"
  config_version: 2

config:
  dynamic:

    http:
      anonymous_auth_enabled: false

    authc:


      basic_internal_auth_domain:
        description: "Authenticate via HTTP Basic against internal users database\"
        http_enabled: true
        transport_enabled: true
        order: 4
        http_authenticator:
          type: basic
          challenge: true
        authentication_backend:
          type: intern



    authz:

But when I copy the file from the docker image back, I don’t get the same file I expected:

sudo docker cp opensearch-node1:/usr/share/opensearch/plugins/opensearch-security/securityconfig/config.yml /tmp/config.yml

and

12:00:51 es-master:~# cat /tmp/config.yml
---

# This is the main OpenSearch Security configuration file where authentication
# and authorization is defined.
#
# You need to configure at least one authentication domain in the authc of this file.
# An authentication domain is responsible for extracting the user credentials from
# the request and for validating them against an authentication backend like Active Directory for example.
#
# If more than one authentication domain is configured the first one which succeeds wins.
# If all authentication domains fail then the request is unauthenticated.
# In this case an exception is thrown and/or the HTTP status is set to 401.
#
# After authentication authorization (authz) will be applied. There can be zero or more authorizers which collect
# the roles from a given backend for the authenticated user.
#
# Both, authc and auth can be enabled/disabled separately for REST and TRANSPORT layer. Default is true for both.
#        http_enabled: true
#        transport_enabled: true
#
# For HTTP it is possible to allow anonymous authentication. If that is the case then the HTTP authenticators try to
# find user credentials in the HTTP request. If credentials are found then the user gets regularly authenticated.
# If none can be found the user will be authenticated as an "anonymous" user. This user has always the username "anonymous"
# and one role named "anonymous_backendrole".
# If you enable anonymous authentication all HTTP authenticators will not challenge.
#
#
# Note: If you define more than one HTTP authenticators make sure to put non-challenging authenticators like "proxy" or "clientcert"
# first and the challenging one last.
# Because it's not possible to challenge a client with two different authentication methods (for example
# Kerberos and Basic) only one can have the challenge flag set to true. You can cope with this situation
# by using pre-authentication, e.g. sending a HTTP Basic authentication header in the request.
#
# Default value of the challenge flag is true.
#
#
# HTTP
#   basic (challenging)
#   proxy (not challenging, needs xff)
#   kerberos (challenging)
#   clientcert (not challenging, needs https)
#   jwt (not challenging)
#   host (not challenging) #DEPRECATED, will be removed in a future version.
#                          host based authentication is configurable in roles_mapping

# Authc
#   internal
#   noop
#   ldap

# Authz
#   ldap
#   noop



_meta:
  type: "config"
  config_version: 2

config:
  dynamic:
    # Set filtered_alias_mode to 'disallow' to forbid more than 2 filtered aliases per index
    # Set filtered_alias_mode to 'warn' to allow more than 2 filtered aliases per index but warns about it (default)
    # Set filtered_alias_mode to 'nowarn' to allow more than 2 filtered aliases per index silently
    #filtered_alias_mode: warn
    #do_not_fail_on_forbidden: false
    #kibana:
    # Kibana multitenancy
    #multitenancy_enabled: true
    #server_username: kibanaserver
    #index: '.kibana'
    http:
      anonymous_auth_enabled: false
      xff:
        enabled: false
        internalProxies: '192\.168\.0\.10|192\.168\.0\.11' # regex pattern
        #internalProxies: '.*' # trust all internal proxies, regex pattern
        #remoteIpHeader:  'x-forwarded-for'
        ###### see https://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html for regex help
        ###### more information about XFF https://en.wikipedia.org/wiki/X-Forwarded-For
        ###### and here https://tools.ietf.org/html/rfc7239
        ###### and https://tomcat.apache.org/tomcat-8.0-doc/config/valve.html#Remote_IP_Valve
    authc:
      kerberos_auth_domain:
        http_enabled: false
        transport_enabled: false
        order: 6
        http_authenticator:
          type: kerberos
          challenge: true
          config:
            # If true a lot of kerberos/security related debugging output will be logged to standard out
            krb_debug: false
            # If true then the realm will be stripped from the user name
            strip_realm_from_principal: true
        authentication_backend:
          type: noop
      basic_internal_auth_domain:
        description: "Authenticate via HTTP Basic against internal users database"
        http_enabled: true
        transport_enabled: true
        order: 4
        http_authenticator:
          type: basic
          challenge: true
        authentication_backend:
          type: intern
      proxy_auth_domain:
        description: "Authenticate via proxy"
        http_enabled: false
        transport_enabled: false
        order: 3
        http_authenticator:
          type: proxy
          challenge: false
          config:
            user_header: "x-proxy-user"
            roles_header: "x-proxy-roles"
        authentication_backend:
          type: noop
      jwt_auth_domain:
        description: "Authenticate via Json Web Token"
        http_enabled: false
        transport_enabled: false
        order: 0
        http_authenticator:
          type: jwt
          challenge: false
          config:
            signing_key: "base64 encoded HMAC key or public RSA/ECDSA pem key"
            jwt_header: "Authorization"
            jwt_url_parameter: null
            roles_key: null
            subject_key: null
        authentication_backend:
          type: noop
      clientcert_auth_domain:
        description: "Authenticate via SSL client certificates"
        http_enabled: false
        transport_enabled: false
        order: 2
        http_authenticator:
          type: clientcert
          config:
            username_attribute: cn #optional, if omitted DN becomes username
          challenge: false
        authentication_backend:
          type: noop
      ldap:
        description: "Authenticate via LDAP or Active Directory"
        http_enabled: false
        transport_enabled: false
        order: 5
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          # LDAP authentication backend (authenticate users against a LDAP or Active Directory)
          type: ldap
          config:
            # enable ldaps
            enable_ssl: false
            # enable start tls, enable_ssl should be false
            enable_start_tls: false
            # send client certificate
            enable_ssl_client_auth: false
            # verify ldap hostname
            verify_hostnames: true
            hosts:
            - localhost:8389
            bind_dn: null
            password: null
            userbase: 'ou=people,dc=example,dc=com'
            # Filter to search for users (currently in the whole subtree beneath userbase)
            # {0} is substituted with the username
            usersearch: '(sAMAccountName={0})'
            # Use this attribute from the user as username (if not set then DN is used)
            username_attribute: null
    authz:
      roles_from_myldap:
        description: "Authorize via LDAP or Active Directory"
        http_enabled: false
        transport_enabled: false
        authorization_backend:
          # LDAP authorization backend (gather roles from a LDAP or Active Directory, you have to configure the above LDAP authentication backend settings too)
          type: ldap
          config:
            # enable ldaps
            enable_ssl: false
            # enable start tls, enable_ssl should be false
            enable_start_tls: false
            # send client certificate
            enable_ssl_client_auth: false
            # verify ldap hostname
            verify_hostnames: true
            hosts:
            - localhost:8389
            bind_dn: null
            password: null
            rolebase: 'ou=groups,dc=example,dc=com'
            # Filter to search for roles (currently in the whole subtree beneath rolebase)
            # {0} is substituted with the DN of the user
            # {1} is substituted with the username
            # {2} is substituted with an attribute value from user's directory entry, of the authenticated user. Use userroleattribute to specify the name of the attribute
            rolesearch: '(member={0})'
            # Specify the name of the attribute which value should be substituted with {2} above
            userroleattribute: null
            # Roles as an attribute of the user entry
            userrolename: disabled
            #userrolename: memberOf
            # The attribute in a role entry containing the name of that role, Default is "name".
            # Can also be "dn" to use the full DN as rolename.
            rolename: cn
            # Resolve nested roles transitive (roles which are members of other roles and so on ...)
            resolve_nested_roles: true
            userbase: 'ou=people,dc=example,dc=com'
            # Filter to search for users (currently in the whole subtree beneath userbase)
            # {0} is substituted with the username
            usersearch: '(uid={0})'
            # Skip users matching a user name, a wildcard or a regex pattern
            #skip_users:
            #  - 'cn=Michael Jackson,ou*people,o=TEST'
            #  - '/\S*/'
      roles_from_another_ldap:
        description: "Authorize via another Active Directory"
        http_enabled: false
        transport_enabled: false
        authorization_backend:
          type: ldap
          #config goes here ...
  #    auth_failure_listeners:
  #      ip_rate_limiting:
  #        type: ip
  #        allowed_tries: 10
  #        time_window_seconds: 3600
  #        block_expiry_seconds: 600
  #        max_blocked_clients: 100000
  #        max_tracked_clients: 100000
  #      internal_authentication_backend_limiting:
  #        type: username
  #        authentication_backend: intern
  #        allowed_tries: 10
  #        time_window_seconds: 3600
  #        block_expiry_seconds: 600
.......

@iamthealex-es You have authentication_backend: duplicated.

That was a copy-paste error on my part.
I have corrected the copy-paste error above so that the duplication is gone.

@iamthealex-es Double check your volumes mappings.

You have:

/usr/share/opensearch/config/opensearch-security/

Instead of:

/usr/share/opensearch/plugins/opensearch-security/securityconfig/

I know I’m sounding like a newbie, and I do appreciate all your help.
I have seen posts using both

/usr/share/opensearch/config/opensearch-security

and

/usr/share/opensearch/plugins/opensearch-security/securityconfig.

I am unsure which is the standard location.

This may be a mixup on my part between opendistro and opensearch.

In any case, I have the config loaded now, and I can now do basic auth again, but I still get the message about the Authorization header should be “basic” (apparently my jwt stanza is being ignored).

Here is the interactive docker shell copy-paste:

12:24:42 es-master:~# sudo docker container exec -it opensearch-node1 /bin/sh
sh-4.2$ /usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh -nhnv -icl -cacert /usr/share/opensearch/config/root-ca.pem -cert /usr/share/opensearch/config/kirk.pem -key /usr/share/opensearch/config/kirk-key.pem -f /usr/share/opensearch/plugins/opensearch-security/securityconfig/config.yml
Security Admin v7
Will connect to localhost:9300 ... done
Connected as CN=kirk,OU=client,O=client,L=test,C=de
OpenSearch Version: 1.3.2
OpenSearch Security Version: 1.3.2.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Clustername: opensearch-cluster
Clusterstate: YELLOW
Number of nodes: 1
Number of data nodes: 1
.opendistro_security index already exists, so we do not need to create one.
Populate config from /usr/share/opensearch
Will update '_doc/config' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/config.yml
   SUCC: Configuration for 'config' created or updated
Done with success

And when I docker copy that config back to the host, it looks right:

12:28:59 es-master:~# sudo docker cp opensearch-node1:/usr/share/opensearch/plugins/opensearch-security/securityconfig/config.yml /tmp/config.yml
12:29:02 es-master:~# cat !$
cat /tmp/config.yml
---

_meta:
  type: "config"
  config_version: 2

config:
  dynamic:

    http:
      anonymous_auth_enabled: false

    authc:


      basic_internal_auth_domain:
        description: "Authenticate via HTTP Basic against internal users database"
        http_enabled: true
        transport_enabled: true
        order: 4
        http_authenticator:
          type: basic
          challenge: true
        authentication_backend:
          type: intern

      jwt_auth_domain:
        description: "Authenticate via JWT"

        http_enabled: true
        transport_enabled: true
        order: 0
        http_authenticator:
          type: jwt
          challenge: false
          config:
            signing_key: "base64 encoded key"
            jwt_header: "Authorization"
            jwt_url_parameter: null
            subject_key: null
            roles_key: roles

        authentication_backend:
          type: noop

    authz:
12:29:06 es-master:~#

Here is the curl:

12:29:54 es-master:~# curl -v -k -H "Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJyb2xlcyI6ImFkbWluIiwiaXNzIjoiaHR0cHM6Ly9sb2NhbGhvc3QiLCJzdWIiOiJhZG1pbiIsImV4cCI6MTU1NDUyMjQzMH0.cFuVSeylBMHoudZjptUsuH529ipBjuxLLj6TSoghZY8"  https://localhost:9200/_cat/nodes
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 9200 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Unknown (8):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Client hello (1):
* TLSv1.3 (OUT), TLS Unknown, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS Unknown, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: DC=de; L=test; O=node; OU=node; CN=node-0.example.com
*  start date: Apr 22 03:43:47 2018 GMT
*  expire date: Apr 19 03:43:47 2028 GMT
*  issuer: DC=com; DC=example; O=Example Com Inc.; OU=Example Com Inc. Root CA; CN=Example Com Inc. Root CA
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
> GET /_cat/nodes HTTP/1.1
> Host: localhost:9200
> User-Agent: curl/7.58.0
> Accept: */*
> Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJyb2xlcyI6ImFkbWluIiwiaXNzIjoiaHR0cHM6Ly9sb2NhbGhvc3QiLCJzdWIiOiJhZG1pbiIsImV4cCI6MTU1NDUyMjQzMH0.cFuVSeylBMHoudZjptUsuH529ipBjuxLLj6TSoghZY8
>
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
< HTTP/1.1 401 Unauthorized
< WWW-Authenticate: Basic realm="OpenSearch Security"
< content-type: text/plain; charset=UTF-8
< content-length: 12
<
* Connection #0 to host localhost left intact
Unauthorized1

Here is the docker log for the curl:

 [2022-05-24T16:29:58,155][WARN ][o.o.s.h.HTTPBasicAuthenticator] [opensearch-node1] No 'Basic Authorization' header, send 401 and 'WWW-Authenticate Basic'

I tried again using quotes:
roles_key: "roles"

and again, I did the docker-compose up and again I applied the config via securityadmin.sh in the interactive shell and I still get the same from curl and the same log message from the docker log for the opensearch container.

@iamthealex-es Did you replace value of signing_key: with base64 encoded key?

i.e.
clear secret - 12345678901234567890123456789012
base64 encoded secret - MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=

Yes, for proof-of-concept purposes, I’m using

Y2hhbmdlbWU=

which is the base64 encoding of

changeme

But the error message implies that the JWT authenticator is not present.