Security_exception","reason":"no permissions for [cluster:monitor/main]

Security
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
opensearch-2.14.0

Describe the issue:
I am trying to create a user for fluent to connect with open search.

I get this error:

>  > 2024-11-13 11:42:12 +0000 [warn]: #0 [out_opens_logs-os-kratos] Could not communicate to OpenSearch, resetting connection and trying again. [403] {"error":{"root_cause":[{"type":"security_exception","reason":"no permissions for [cluster:monitor/main] and User [name=fluentd_internal, backend_roles=[kibana_user, fluentd], requestedTenant=null]"}],"type":"security_exception","reason":"no permissions for [cluster:monitor/main] and User [name=fluentd_internal, backend_roles=[kibana_user, fluentd], requestedTenant=null]"},"status":403}
> > 2024-11-13 11:42:12 +0000 [warn]: #0 [out_opens_logs-os-kratos] Remaining retry: 8. Retry to communicate after 128 second(s).

Configuration:
app vars: # static users for Opensearch internal_users.yml

opens_static_users:
  admin:
    _password: "{{opens_password[buildenv]}}"
    reserved: true
    backend_roles: ["admin"]
    attributes: { description: "admin user" }

  logstash_internal:
    _password: "{{ logstash_password[buildenv] }}"
    reserved: false
    backend_roles: ["logstash"] 
    attributes:
      description: "logstash internal user for writing logs"

  fluentd_internal:
    _password: "{{ fluentd_password[buildenv] }}"
    reserved: false
    backend_roles: ["fluentd", "kibana_user"]
    attributes:
      description: "Fluentd internal user for writing logs"

roles.yml: `

Task to create custom users with passwords and assign them roles

  • name: Create or update custom users with passwords and roles
    uri:
    url: “https://localhost:9200/_plugins/_security/api/internalusers/{{ item.username }}”
    user: “admin”
    password: “{{ opens_password[buildenv] }}”
    force_basic_auth: yes
    method: PUT
    validate_certs: no
    body_format: json
    body: |
    {
    “password”: “{{ item.password }}”,
    “roles”: {{ item.roles | to_json }}
    }
    headers:
    Content-Type: “application/json”
    with_items:
    • { username: “logstash_internal”, password: “{{ logstash_password[buildenv] }}”, roles: [“logstash”] }
    • { username: “fluentd_internal”, password: “{{ fluentd_password[buildenv] }}”, roles: [“fluentd”] }
      register: user_creation_response
      debug:
      msg: “Response for creating user {{ item.username }}: {{ user_creation_response }}”

Task to create custom roles for Fluentd

  • name: Create custom roles for Fluentd and Logstash
    uri:
    url: “https://localhost:9200/_plugins/_security/api/roles/{{ item.name }}”
    user: “admin”
    password: “{{ opens_password[buildenv] }}”
    force_basic_auth: yes
    method: PUT
    validate_certs: no
    body_format: json
    body: “{{ item.definition | to_json }}”
    headers:
    Content-Type: “application/json”
    with_items:

    • name: “fluentd”
      definition:
      reserved: false
      hidden: false
      cluster_permissions:
      - “cluster_monitor”
      - “cluster_composite_ops”
      - “cluster:admin/ingest/pipeline/"
      - "indices:admin/template/      ”
      - “cluster:admin/component_template/"
      index_permissions:
      - index_patterns: ["      ”]
      dls: “”
      fls:
      masked_fields:
      allowed_actions:
      - “create_index”
      - “crud”
      tenant_permissions:
      - tenant_patterns: [“global_tenant”]
      allowed_actions: [“kibana_read_only”]
      static: false
      _meta:
      type: “roles”
      config_version: 2
      register: role_creation_response

  • debug:
    msg: “Response for creating role {{ item.name }}: {{ role_creation_response }}”

Task to map fluentd_internal user to fluentd role

  • name: Update role mapping for fluentd
    uri:
    url: “https://localhost:9200/_plugins/_security/api/rolesmapping/fluentd”
    user: “admin”
    password: “{{ opens_password[buildenv] }}”
    force_basic_auth: yes
    method: PUT
    validate_certs: no
    body_format: json
    body: |
    {
    “reserved”: false,
    “hidden”: false,
    “backend_roles”: [“fluentd”],
    “hosts”: ,
    “users”: [“fluentd_internal”],
    “roles”: [“kibana_user”],
    “and_backend_roles”: ,
    “_meta”: {
    “type”: “rolesmapping”,
    “config_version”: 2
    }
    }
    headers:
    Content-Type: “application/json”
    register: complex_role_mapping_response

  • debug:
    msg: “Response for complex-role mapping: {{ complex_role_mapping_response }}”
    `

Relevant Logs or Screenshots:

2

U have a same issue

For temp solution i provide admin backend role for service user

3

Thank you so much, I mapped the roles logstash and kibana to my fluentd user. It worked but hopefully there are no further problems with ingestion

4

I was unable to write, still error 403. This is really sad. does this mean you cannot assign specific roles to connect fluentd to opensearch

5

Hi @bukola,

Could you share the output of the following:

curl --insecure -u <admin_username>:<admin_password> -XGET https://<OS_node>:9200/_plugins/_security/api/rolesmapping?pretty

thanks,
mj