JWT Yet Again - Trouble Setting Up JWT Authentication Method

Security
1

Hello Community,

I’ve been struggling to get the JWT authentication method to work on my system. Despite going through numerous forum topics and documentation, I haven’t had any success. Every time I attempt to authenticate, I receive the following error message:

{“statusCode”:401,“error”:“Unauthorized”,“message”:“Authentication Exception”} Perhaps you guys have any clue how to make it run.

http://192.168.1.0:5601/app/dashboards?mytoken=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJhZG1pbiIsInJvbGVzIjoiYWRtaW4iLCJpYXQiOjE1MTYyMzkwMjJ9.zeEgRjPg8oRJrfA5iLinbBW-I8-akmfkg3NZAxHIMs8

My config.yml

  jwt_auth_domain:
    description: "Authenticate via Json Web Token"
    http_enabled: true
    transport_enabled: true
    order: 1
    http_authenticator:
      type: jwt
      challenge: false
      config:
        signing_key: "MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI="
        jwt_header: "Authorization"
        jwt_url_parameter: mytoken
        #jwt_clock_skew_tolerance_seconds: 30
        roles_key: roles
        subject_key: sub

Keys

clear secret - 12345678901234567890123456789012
base64 encoded secret - MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=

jwt.io

Zrzut ekranu 2024-03-22 o 15.49.33
Zrzut ekranu 2024-03-22 o 15.49.331266×764 55.5 KB

I’ve double-checked my configuration settings and followed the recommended steps, but I’m still encountering this issue. I’m reaching out to see if anyone has encountered a similar problem or if there are any specific tips or tricks to troubleshoot this.

Any help or guidance you can provide would be greatly appreciated.

Thank you in advance!

2

Hi @m_p,

Did you set the below:

opensearch_dashboards.yml

opensearch_security.auth.type: "jwt"
opensearch_security.jwt.url_param: <your-param-name-here>

please see more here: JSON Web Token - OpenSearch Documentation

Best,
mj

3

Hi @Mantas thanks for your interest. Yes, I set that. :confused:

opensearch_dashboards.yml

opensearch.hosts: [https://localhost:9200]

opensearch.ssl.verificationMode: none

opensearch.username: kibanaserver

opensearch.password: kibanaserver

opensearch.requestHeadersWhitelist: [authorization, securitytenant]

opensearch_security.multitenancy.enabled: true

opensearch_security.multitenancy.tenants.preferred: [Private, Global]

opensearch_security.readonly_mode.roles: [kibana_read_only]

opensearch_security.cookie.secure: false

opensearch_security.auth.type: “jwt”

opensearch_security.jwt.url_param: ‘mytoken’

Best,
mp

4

Hello Everyone,

Despite going through the documentation thoroughly, I’m still facing issues, and I’m unsure about what to double-check at this point.

@pablo , since you’ve been active in similar discussions before, I was wondering if you could provide some guidance or assistance. Any suggestions or pointers you have would be greatly appreciated.

Thank you all for your help!

Best regards,
m_p

5

Hi @m_p,

Is this a full config.yml or just a JWT snippet? You should have in your config.yml the BasicAuth with a higher order number, as well i.e.: jwt_auth_domain.order: 0, basic_internal_auth_domain.order: 1

authc:
  basic_internal_auth_domain:
    description: "Authenticate via HTTP Basic against internal users database"
    http_enabled: true
    transport_enabled: true
    order: 1
    http_authenticator:
      type: basic
      challenge: true
    authentication_backend:
      type: intern
  jwt_auth_domain:
    description: "Authenticate via Json Web Token"
    http_enabled: true
    transport_enabled: true
    order: 0
    http_authenticator:
      type: jwt
      challenge: false
      config:
        signing_key: "MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI="
        jwt_header: "Authorization"
        jwt_url_parameter: mytoken
        #jwt_clock_skew_tolerance_seconds: 30
        roles_key: roles
        subject_key: sub

Best,
mj

6

It is only the JWT snippet, but here is full config perhaps that would help.

config.yml

config:
 
    http:
      anonymous_auth_enabled: false
      xff:
        enabled: false
        internalProxies: '192\.168\.0\.10|192\.168\.0\.11' # regex pattern
    authc:
      kerberos_auth_domain:
        http_enabled: false
        transport_enabled: false
        order: 6
        http_authenticator:
          type: kerberos
          challenge: true
          config:
            # If true a lot of kerberos/security related debugging output will be logged to standard out
            krb_debug: false
            # If true then the realm will be stripped from the user name
            strip_realm_from_principal: true
        authentication_backend:
          type: noop
      basic_internal_auth_domain:
        description: "Authenticate via HTTP Basic against internal users database"
        http_enabled: true
        transport_enabled: true
        order: 0
        http_authenticator:
          type: basic
          challenge: true
        authentication_backend:
          type: intern
      proxy_auth_domain:
        description: "Authenticate via proxy"
        http_enabled: false
        transport_enabled: false
        order: 3
        http_authenticator:
          type: proxy
          challenge: false
          config:
            user_header: "x-proxy-user"
            roles_header: "x-proxy-roles"
        authentication_backend:
          type: noop
      jwt_auth_domain:
        description: "Authenticate via Json Web Token"
        http_enabled: true
        transport_enabled: true
        order: 1
        http_authenticator:
          type: jwt
          challenge: false
          config:
            signing_key: "MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI="
            jwt_header: "Authorization"
            jwt_url_parameter: mytoken
            #jwt_clock_skew_tolerance_seconds: 30
            roles_key: roles
            subject_key: sub
        authentication_backend:
          type: noop
      clientcert_auth_domain:
        description: "Authenticate via SSL client certificates"
        http_enabled: false
transport_enabled: false
        order: 2
        http_authenticator:
          type: clientcert
          config:
            username_attribute: cn #optional, if omitted DN becomes username
          challenge: false
        authentication_backend:
          type: noop
      ldap:
        description: "Authenticate via LDAP or Active Directory"
        http_enabled: false
        transport_enabled: false
        order: 5
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          # LDAP authentication backend (authenticate users against a LDAP or Active Directory)
          type: ldap
          config:
            # enable ldaps
            enable_ssl: false
            # enable start tls, enable_ssl should be false
            enable_start_tls: false
            # send client certificate
            enable_ssl_client_auth: false
            # verify ldap hostname
            verify_hostnames: true
            hosts:
            - localhost:8389
            bind_dn: null
            password: null
            userbase: 'ou=people,dc=example,dc=com'
            # Filter to search for users (currently in the whole subtree beneath userbase)
            # {0} is substituted with the username
            usersearch: '(sAMAccountName={0})'
            # Use this attribute from the user as username (if not set then DN is used)
            username_attribute: null
    authz:
      roles_from_myldap:
        description: "Authorize via LDAP or Active Directory"
        http_enabled: false
        transport_enabled: false
        authorization_backend:
          # LDAP authorization backend (gather roles from a LDAP or Active Directory, you have to configure the above LDAP authentication backend settings too)
          type: ldap
          config:
            # enable ldaps
            enable_ssl: false
            # enable start tls, enable_ssl should be false
            enable_start_tls: false
            # send client certificate
            enable_ssl_client_auth: false
            # verify ldap hostname
            verify_hostnames: true
            hosts:
            - localhost:8389
            bind_dn: null
            password: null
            rolebase: 'ou=groups,dc=example,dc=com'
            # Filter to search for roles (currently in the whole subtree beneath rolebase)
            # {0} is substituted with the DN of the user
            # {1} is substituted with the username
            # {2} is substituted with an attribute value from user's directory entry, of the authenticated user. Use userroleattribute to specify the name of the attribute
            rolesearch: '(member={0})'
            # Specify the name of the attribute which value should be substituted with {2} above
            userroleattribute: null
            # Roles as an attribute of the user entry
            userrolename: disabled
            #userrolename: memberOf
            # The attribute in a role entry containing the name of that role, Default is "name".
            # Can also be "dn" to use the full DN as rolename.
            rolename: cn
            # Resolve nested roles transitive (roles which are members of other roles and so on ...)
            resolve_nested_roles: true
            userbase: 'ou=people,dc=example,dc=com'
            # Filter to search for users (currently in the whole subtree beneath userbase)
            # {0} is substituted with the username
            usersearch: '(uid={0})'
            # Skip users matching a user name, a wildcard or a regex pattern
            #skip_users:
            #  - 'cn=Michael Jackson,ou*people,o=TEST'
            #  - '/\S*/'
      roles_from_another_ldap:
        description: "Authorize via another Active Directory"
        http_enabled: false
        transport_enabled: false
        authorization_backend:
          type: ldap
7

Could you adjust the order as per my comment above, test it and let me know?

thanks,
mj

8

I’ve configured auto domain to have order 1 and JWT domain to have order 0, but unfortunately, I’m still unable to log in successfully. Every time I attempt to log in, I receive the following error message:

{"statusCode":401,"error":"Unauthorized","message":"Authentication Exception"}

I’m puzzled because I expected that if the JWT authentication method isn’t working, it should automatically transfer me to the next authentication method in line. Is it possible to use multiple login methods to log in to the dashboard?

9

Please run the below and share the output:


curl --insecure -u <admin_user>:<admin_password> -XGET https://<OS_node>:9200/_plugins/_security/api/securityconfig?pretty
10

The output:

{
  "config" : {
    "dynamic" : {
      "filtered_alias_mode" : "warn",
      "disable_rest_auth" : false,
      "disable_intertransport_auth" : false,
      "respect_request_indices_options" : false,
      "kibana" : {
        "multitenancy_enabled" : true,
        "private_tenant_enabled" : true,
        "default_tenant" : "",
        "server_username" : "kibanaserver",
        "index" : ".kibana"
      },
      "http" : {
        "anonymous_auth_enabled" : false,
        "xff" : {
          "enabled" : false,
          "internalProxies" : "192\\.168\\.0\\.10|192\\.168\\.0\\.11",
          "remoteIpHeader" : "X-Forwarded-For"
        }
      },
      "authc" : {
        "jwt_auth_domain" : {
          "http_enabled" : false,
          "transport_enabled" : false,
          "order" : 0,
          "http_authenticator" : {
            "challenge" : false,
            "type" : "jwt",
            "config" : {
              "signing_key" : "base64 encoded HMAC key or public RSA/ECDSA pem key",
              "jwt_header" : "Authorization",
              "jwt_clock_skew_tolerance_seconds" : 30
            }
          },
          "authentication_backend" : {
            "type" : "noop",
            "config" : { }
          },
          "description" : "Authenticate via Json Web Token"
        },
        "ldap" : {
          "http_enabled" : false,
          "transport_enabled" : false,
          "order" : 5,
          "http_authenticator" : {
            "challenge" : false,
            "type" : "basic",
            "config" : { }
          },
          "authentication_backend" : {
            "type" : "ldap",
            "config" : {
              "enable_ssl" : false,
              "enable_start_tls" : false,
              "enable_ssl_client_auth" : false,
              "verify_hostnames" : true,
              "hosts" : [
                "localhost:8389"
              ],
              "userbase" : "ou=people,dc=example,dc=com",
              "usersearch" : "(sAMAccountName={0})"
            }
          },
          "description" : "Authenticate via LDAP or Active Directory"
        },
        "basic_internal_auth_domain" : {
          "http_enabled" : true,
          "transport_enabled" : true,
          "order" : 4,
          "http_authenticator" : {
            "challenge" : true,
            "type" : "basic",
            "config" : { }
          },
          "authentication_backend" : {
            "type" : "intern",
            "config" : { }
          },
          "description" : "Authenticate via HTTP Basic against internal users database"
        },
        "proxy_auth_domain" : {
          "http_enabled" : false,
          "transport_enabled" : false,
          "order" : 3,
          "http_authenticator" : {
            "challenge" : false,
            "type" : "proxy",
            "config" : {
              "user_header" : "x-proxy-user",
              "roles_header" : "x-proxy-roles"
            }
          },
          "authentication_backend" : {
            "type" : "noop",
            "config" : { }
          },
          "description" : "Authenticate via proxy"
        },
        "clientcert_auth_domain" : {
          "http_enabled" : false,
          "transport_enabled" : false,
          "order" : 2,
          "http_authenticator" : {
            "challenge" : false,
            "type" : "clientcert",
            "config" : {
              "username_attribute" : "cn"
            }
          },
          "authentication_backend" : {
            "type" : "noop",
            "config" : { }
          },
          "description" : "Authenticate via SSL client certificates"
        },
        "kerberos_auth_domain" : {
          "http_enabled" : false,
          "transport_enabled" : false,
          "order" : 6,
          "http_authenticator" : {
            "challenge" : true,
            "type" : "kerberos",
            "config" : {
              "krb_debug" : false,
              "strip_realm_from_principal" : true
            }
          },
          "authentication_backend" : {
            "type" : "noop",
            "config" : { }
          }
        }
      },
      "authz" : {
        "roles_from_another_ldap" : {
          "http_enabled" : false,
          "transport_enabled" : false,
          "authorization_backend" : {
            "type" : "ldap",
            "config" : { }
          },
          "description" : "Authorize via another Active Directory"
        },
        "roles_from_myldap" : {
          "http_enabled" : false,
          "transport_enabled" : false,
          "authorization_backend" : {
            "type" : "ldap",
            "config" : {
              "enable_ssl" : false,
              "enable_start_tls" : false,
              "enable_ssl_client_auth" : false,
              "verify_hostnames" : true,
              "hosts" : [
                "localhost:8389"
              ],
              "rolebase" : "ou=groups,dc=example,dc=com",
              "rolesearch" : "(member={0})",
              "userrolename" : "disabled",
              "rolename" : "cn",
              "resolve_nested_roles" : true,
              "userbase" : "ou=people,dc=example,dc=com",
              "usersearch" : "(uid={0})"
            }
          },
          "description" : "Authorize via LDAP or Active Directory"
        }
      },
      "auth_failure_listeners" : { },
      "do_not_fail_on_forbidden" : false,
      "multi_rolespan_enabled" : true,
      "hosts_resolver_mode" : "ip-only",
      "do_not_fail_on_forbidden_empty" : false
    }
  }
}
11

Looks like JWT authentication is disabled, have you applied your configuration using securityadmin.sh?

Please see more here: Applying changes to configuration files - OpenSearch Documentation

Best,
mj

12

Yes, I always perform the security admin tool invocation after making any configuration changes.

sudo ./securityadmin.sh -f /etc/opensearch/opensearch-security/internal_users.yml -icl -nhnv -cacert /etc/opensearch/root-ca.pem -cert /etc/opensearch/kirk.pem -key /etc/opensearch/kirk-key.pem

also I see running htop a lot of java open search process is this normal?

/usr/share/opensearch/jdk/bin/java -Xshare: auto -Dopensearch.networkaddress.cache.ttl=60 -D opensearch.networkaddress.cache.negative.ttl=10
13

The above will only update your internal_users.yml as per documentation.

A full update should look something like below (or replace internal_users.yml with config.yml to only update config.yml):

sudo ./securityadmin.sh -cd /etc/opensearch/config/opensearch-security -icl -cacert /etc/opensearch/root-ca.pem -cert /etc/opensearch/kirk.pem -key /etc/opensearch/kirk-key.pem -nhnv

Best,
mj

14

Hi Mantas,

Thank you for your assistance earlier. Although the curl output now displays the expected values, I’m encountering an issue with logging in using the JWT authentication method.

Despite providing the JWT token in the URL, I’m still redirected to the login page. Do you have any further suggestions on how to resolve this?

Here’s the URL I’m using:

http://0.0.0.0:5601/app/dashboards?mytoken=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJhZG1pbiIsInJvbGVzIjoiYWRtaW4iLCJpYXQiOjE1MTYyMzkwMjJ9.zeEgRjPg8oRJrfA5iLinbBW-I8-akmfkg3NZAxHIMs8

And here’s the curl output for reference:

{
  "config" : {
    "dynamic" : {
      "filtered_alias_mode" : "warn",
      "disable_rest_auth" : false,
      "disable_intertransport_auth" : false,
      "respect_request_indices_options" : false,
      "kibana" : {
        "multitenancy_enabled" : true,
        "private_tenant_enabled" : true,
        "default_tenant" : "",
        "server_username" : "kibanaserver",
        "index" : ".kibana"
      },
      "http" : {
        "anonymous_auth_enabled" : false,
        "xff" : {
          "enabled" : false,
          "internalProxies" : "192\\.168\\.0\\.10|192\\.168\\.0\\.11",
          "remoteIpHeader" : "X-Forwarded-For"
        }
      },
      "authc" : {
        "jwt_auth_domain" : {
          "http_enabled" : true,
          "order" : 0,
          "http_authenticator" : {
            "challenge" : false,
            "type" : "jwt",
            "config" : {
              "signing_key" : "MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=",
              "jwt_header" : "Authorization",
              "jwt_url_parameter" : "mytoken",
              "roles_key" : "roles",
              "subject_key" : "sub"
            }
          },
          "authentication_backend" : {
            "type" : "noop",
            "config" : { }
          },
          "description" : "Authenticate via Json Web Token"
        },
        "ldap" : {
          "http_enabled" : false,
          "order" : 5,
          "http_authenticator" : {
            "challenge" : false,
            "type" : "basic",
            "config" : { }
          },
          "authentication_backend" : {
            "type" : "ldap",
            "config" : {
              "enable_ssl" : false,
              "enable_start_tls" : false,
              "enable_ssl_client_auth" : false,
              "verify_hostnames" : true,
              "hosts" : [
                "localhost:8389"
              ],
              "userbase" : "ou=people,dc=example,dc=com",
              "usersearch" : "(sAMAccountName={0})"
            }
          },
          "description" : "Authenticate via LDAP or Active Directory"
        },
        "basic_internal_auth_domain" : {
          "http_enabled" : true,
          "order" : 1,
          "http_authenticator" : {
            "challenge" : true,
            "type" : "basic",
            "config" : { }
          },
          "authentication_backend" : {
            "type" : "intern",
            "config" : { }
          },
          "description" : "Authenticate via HTTP Basic against internal users database"
        },
        "proxy_auth_domain" : {
          "http_enabled" : false,
          "order" : 3,
          "http_authenticator" : {
            "challenge" : false,
            "type" : "proxy",
            "config" : {
              "user_header" : "x-proxy-user",
              "roles_header" : "x-proxy-roles"
            }
          },
          "authentication_backend" : {
            "type" : "noop",
            "config" : { }
          },
          "description" : "Authenticate via proxy"
        },
        "clientcert_auth_domain" : {
          "http_enabled" : false,
          "order" : 2,
          "http_authenticator" : {
            "challenge" : false,
            "type" : "clientcert",
            "config" : {
              "username_attribute" : "cn"
            }
          },
          "authentication_backend" : {
            "type" : "noop",
            "config" : { }
          },
          "description" : "Authenticate via SSL client certificates"
        },
        "kerberos_auth_domain" : {
          "http_enabled" : false,
          "order" : 6,
          "http_authenticator" : {
            "challenge" : true,
            "type" : "kerberos",
            "config" : {
              "krb_debug" : false,
              "strip_realm_from_principal" : true
            }
          },
          "authentication_backend" : {
            "type" : "noop",
            "config" : { }
          }
        }
      },
      "authz" : {
        "roles_from_another_ldap" : {
          "http_enabled" : false,
          "authorization_backend" : {
            "type" : "ldap",
            "config" : { }
          },
          "description" : "Authorize via another Active Directory"
        },
        "roles_from_myldap" : {
          "http_enabled" : false,
          "authorization_backend" : {
            "type" : "ldap",
            "config" : {
              "enable_ssl" : false,
              "enable_start_tls" : false,
              "enable_ssl_client_auth" : false,
              "verify_hostnames" : true,
              "hosts" : [
                "localhost:8389"
              ],
              "rolebase" : "ou=groups,dc=example,dc=com",
              "rolesearch" : "(member={0})",
              "userrolename" : "disabled",
              "rolename" : "cn",
              "resolve_nested_roles" : true,
              "userbase" : "ou=people,dc=example,dc=com",
              "usersearch" : "(uid={0})"
            }
          },
          "description" : "Authorize via LDAP or Active Directory"
        }
      },
      "auth_failure_listeners" : { },
      "do_not_fail_on_forbidden" : false,
      "multi_rolespan_enabled" : true,
      "hosts_resolver_mode" : "ip-only",
      "do_not_fail_on_forbidden_empty" : false,
      "on_behalf_of" : {
        "enabled" : false
      }
    }
  }
}

Your assistance is greatly appreciated.

Thank you,
m_p

15

Alright, I’ve managed to identify the mistake I made. In the opensearch_dashboards.yml file, I mistakenly hashed out these lines after numerous attempts to resolve the issue. As a result, only the JWT authentication method is currently functional.

#JWT
opensearch_security.auth.type: "jwt"
opensearch_security.jwt.url_param: 'mytoken'

Now, I’m curious about incorporating basic authentication method into OpenSearch Dashboards alongside JWT. Could anyone provide guidance on how to achieve this?

Your assistance is much appreciated!

Best regards,
m_p

1 Like
16

Hi @m_p,

There is a configuration to set multiple authentications in opensearch_dashboards.yml ( multiple_auth_enabled), however, it will only accept "basicauth", "openid", and "saml". It does not look like JWT is supported with multiple_auth.

opensearch_security.auth.multiple_auth_enabled: true
opensearch_security.auth.type: ["..",".."]

For more info refer to: Configuring sign-in options - OpenSearch Documentation

Best,
mj

17

Hi @m_p,

Can you test with the below:


http://0.0.0.0:5601/app/dashboards?jwtToken=<valid_token>

thanks.
mj

18

Hi @Mantas,

I wanted to give you an update on the issue I was facing with the JWT token authorization. After some further investigation, I realized that the problem was related to my configuration file. I managed to fix it, and now everything seems to be working smoothly. However, I encountered a new challenge when trying to share a dashboard via iframe – it appears that the JWT token isn’t working as expected.

Here’s the iframe code I’m using:

<iframe src="http://0.0.0.0:5601/app/dashboards?mytoken=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJhZG1pbiIsInJvbGVzIjoiYWRtaW4iLCJpYXQiOjE1MTYyMzkwMjJ9.zeEgRjPg8oRJrfA5iLinbBW-I8-akmfkg3NZAxHIMs8#/view/3a1625a0-f266-11ee-b7b2-9d101a2138ff?embed=true&_g=(filters%3A!()%2CrefreshInterval%3A(pause%3A!t%2Cvalue%3A0)%2Ctime%3A(from%3Anow-2y%2Cto%3A'2022-09-28T19%3A33%3A16.116Z'))&show-time-filter=true" height="600" width="800"></iframe>

I wanted to express my gratitude for your assistance in troubleshooting the initial issue. Without your help, I wouldn’t have been able to resolve it. Thank you once again!

Additionally, if anyone else encounters similar issues with JWT authorization, I’ve included a simple Python script below that can generate all the necessary data for JWT authorization. Feel free to use it if needed.

import jwt
import base64
from datetime import datetime, timedelta

# Define the secret key
secret_key = '12345678901234567890123456789012'

# Encode secret key in base64 format
encoded_secret_key = base64.b64encode(secret_key.encode()).decode()

# Define the payload data
payload = {
    "sub": "admin",
    "roles": "admin",
    "iat": int(datetime.now().timestamp())
}

# Define the token expiration time
token_validity_days = 30  
expiry_time = datetime.utcnow() + timedelta(days=token_validity_days)

# Encode the JWT token
jwt_token = jwt.encode(payload, secret_key, algorithm='HS256')

print("----------------------------------------------------------------------------------------------")

print("Secret Key (base64 encoded):", encoded_secret_key)

print("----------------------------------------------------------------------------------------------")

print("----------------------------------------------------------------------------------------------")

print("Generated JWT token:", jwt_token)

print("----------------------------------------------------------------------------------------------")

print("----------------------------------------------------------------------------------------------")

print("Token expired in {} days".format(token_validity_days))

print("----------------------------------------------------------------------------------------------")

Once again, thank you for your help.

Best regards,
m_p

19

Hi @m_p,

You are very welcome, glad to hear I could help.
Thank you a million for sharing your solution the community will benefit greatly!

Best,
mj