Not able to create the user

Security
1

While deploying my opensearch cluster, a custom internal user for monitoring purpose with custom role by putting the user spec in internal_users.yml file. But user is not being created. Below are the details

internal_users.yml:
        prometheus:
          hash: "$2y$12$Sfi1XzL7j94vpAjRnK0wOeMzJ60XNBSeN0BLh/srmweHnfpgnhdMC"
          reserved: false
          description: "Prometheus metrics user"


roles.yml:
        # Allows prometheus users to get the metrics
        prometheus:
          reserved: false
          cluster_permissions:
            - 'cluster:monitor/health'
            - 'cluster_monitor'
roles_mapping.yml:
        prometheus:
          reserved: false
          users:
          - "prometheus"

image
image3576×1594 359 KB

Could anyone please help.

2

@kksaha Do you get any errors when you upload new settings with securityadmin.sh script?
What is the OpenSearch version?

3

Hello @pablo Thanks for your quick response. I am deploying through helm, so manually not updating those files and reload securityadmin.sh. But no suspicious error messages in logs. Opensearch version - 1.3.0

4

@kksaha Are the modified yml files present in the pod’s filesystem?
Do you see prometheus roles in the deployed environment?

5

@pablo Yes Yes, it’s present in the deployed environment.

[opensearch@observability-opensearch-data-az1-0 securityconfig]$ grep prometheus -A 6 internal_users.yml 
prometheus:
  hash: "$2y$12$Odpu44Pa.h38b2YEI.r33eXsVkQXEQic7Zw9UkcTkmCNCzaqxEvPa"
  reserved: false
  description: "Prometheus metrics user"

snapshotrestore:
  hash: "$2y$12$DpwmetHKwgYnorbgdvORCenv4NAK8cPUg8AI6pxLCuWf/ALc0.v7W"

[opensearch@observability-opensearch-data-az1-0 securityconfig]$ grep prometheus -A 6 roles.yml 
# Allows prometheus users to get the metrics
prometheus:
  reserved: false
  cluster_permissions:
    - 'cluster:monitor/health'
    - 'cluster_monitor'


[opensearch@observability-opensearch-data-az1-0 securityconfig]$ grep prometheus -A 6 roles_mapping.yml 
prometheus:
  reserved: false
  users:
  - "prometheus"

manage_snapshots:
  reserved: false
  backend_roles:
  - "snapshotrestore"
6

I am able to create the prometheus user now. But we restrict the permission to prometheus user only to get the metrics.

[opensearch@observability-opensearch-master-0 ~]$ curl -k https://0:9200/_cluster/health?pretty -u prometheus:*********
{
  "cluster_name" : "observabilty-opensearch",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 9,
  "number_of_data_nodes" : 6,
  "discovered_master" : true,
  "active_primary_shards" : 4,
  "active_shards" : 12,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

image
image3148×1370 272 KB

Given following permission but no luck


        # Allows prometheus users to get the metrics
        prometheus:
          reserved: false
          cluster_permissions:
            - "cluster:monitor/health"
            - "cluster:monitor/nodes/stats"
            - "cluster:monitor/state"
            - "cluster:monitor/nodes/info"
            - "cluster:monitor/prometheus/metrics"

[opensearch@observability-opensearch-master-0 ~]$ curl -k https://0:9200/_prometheus/metrics -u prometheus:password-a
{"error":{"root_cause":[{"type":"security_exception","reason":"no permissions for [indices:monitor/stats] and User [name=prometheus, backend_roles=[prometheus], requestedTenant=null]"}],"type":"exception","reason":"Indices stats request failed","caused_by":{"type":"security_exception","reason":"no permissions for [indices:monitor/stats] and User [name=prometheus, backend_roles=[prometheus], requestedTenant=null]"}},"status":500}

Do I need to create one permissions group and attach with the role. Any suggestion please.

7

I have added one extra permission for indices and seems to be working.

        # Allows prometheus users to get the metrics
        prometheus:
          reserved: false
          cluster_permissions:
            - "cluster:monitor/health"
            - "cluster:monitor/nodes/stats"
            - "cluster:monitor/state"
            - "cluster:monitor/nodes/info"
            - "cluster:monitor/prometheus/metrics"
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - 'indices:monitor/stats'  

[opensearch@observability-opensearch-master-0 ~]$ curl -k https://0:9200/_prometheus/metrics -u prometheus:**********

# HELP opensearch_index_merges_total_docs_count Count of documents merged

# TYPE opensearch_index_merges_total_docs_count gauge

opensearch_index_merges_total_docs_count{cluster="observabilty-opensearch",index=".opendistro_security",context="primaries",} 0.0

opensearch_index_merges_total_docs_count{cluster="observabilty-opensearch",index="security-auditlog-2022.04.12",context="total",} 40.0

opensearch_index_merges_total_docs_count{cluster="observabilty-opensearch",index="security-auditlog-2022.04.12",context="primaries",} 20.0

opensearch_index_merges_total_docs_count{cluster="observabilty-opensearch",index=".opendistro_security",context="total",} 0.0
8

@kksaha Permissions group would only group your existing permissions but wouldn’t change the behaviour. Adding indices:monitor/stats is the solution as that is what OpenSearch was requesting in the logs.
When there are missing permissions, OpenSearch should always report it in its logs.