OpenSearch Security not initialized in docker

OpenSearch
1

Versions :
OpenSearch - 2.15.0
Docker - 26.1.3
Docker Compose - 2.27.0

Describe the issue:
I installed an open source cluster in docker. Before starting docker composition, I run the command

export OPENSEARCH_INITIAL_ADMIN_PASSWORD=Qwerty12345678#

But when I check

curl https://localhost:9200 -ku 'admin:Qwerty12345678#'

it gives an error

OpenSearch Security not initialized

Relevant Logs or Screenshots:
my docker-compose.yml

version: '3'
services:
  opensearch-node1: # This is also the hostname of the container within the Docker network (i.e. https://opensearch-node1/)
    image: opensearch:2.15.0 # Specifying the latest available image - modify if you want a specific version
    container_name: opensearch-node1
    environment:
      - cluster.name=opensearch-cluster # Name the cluster
      - node.name=node-01 # Name the node that will run in this container
      - discovery.seed_hosts=node-01,node-02,node-03 # Nodes to look for when discovering the cluster
      - cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2,opensearch-node3 # Nodes eligible to serve as cluster manager
      - bootstrap.memory_lock=true # Disable JVM heap memory swapping
      - "OPENSEARCH_JAVA_OPTS=-Xms8192m -Xmx8192m" # Set min and max JVM heap sizes to at least 50% of system RAM
      - OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_INITIAL_ADMIN_PASSWORD}    # Sets the demo admin user password when using demo configuration, required for OpenSearch 2.12 and later
    ulimits:
      memlock:
        soft: -1 # Set memlock to unlimited (no soft or hard limit)
        hard: -1
      nofile:
        soft: 65536 # Maximum number of open files for the opensearch user - set to at least 65536
        hard: 65536
    volumes:
      - /data/opensearch:/usr/share/opensearch/data # Creates volume called opensearch-data1 and mounts it to the container
    ports:
      - 9200:9200 # REST API
      - 9600:9600 # Performance Analyzer
    network_mode: host

  opensearch-dashboards:
    image: opensearch-dashboards:2.15.0 # Make sure the version of opensearch-dashboards matches the version of opensearch installed on other nodes
    container_name: opensearch-dashboards
    ports:
      - 5601:5601 # Map host port 5601 to container port 5601
    expose:
      - "5601" # Expose port 5601 for web access to OpenSearch Dashboards
    environment:
      OPENSEARCH_HOSTS: '["https://node-01:9200","https://node-02:9200","https://node-03:9200"]' # Define the OpenSearch nodes that OpenSearch Dashboards will query
    network_mode: host
2

@Ascalonking If this is the full docker-compose.yml file then the message is expected.
The OpenSearch node is configured as part of the cluster. If you’re trying to deploy a single node cluster then you should use the following option.

- discovery.type=single-node

What errors do you see in the OpenSearch logs?

3

I’m installing an open source cluster in docker on three physical servers

$ docker logs "container_id"

[2024-07-11T09:01:39,705][INFO ][o.o.s.c.ConfigurationRepository] [node-01] Wait for cluster to be available ...
[2024-07-11T09:01:40,705][INFO ][o.o.s.c.ConfigurationRepository] [node-01] Wait for cluster to be available ...
[2024-07-11T09:01:41,201][ERROR][o.o.s.a.BackendRegistry  ] [node-01] Not yet initialized (you may need to run securityadmin)
[2024-07-11T09:01:41,205][ERROR][o.o.s.a.BackendRegistry  ] [node-01] Not yet initialized (you may need to run securityadmin)
[2024-07-11T09:01:41,705][INFO ][o.o.s.c.ConfigurationRepository] [node-01] Wait for cluster to be available ...
[2024-07-11T09:01:42,705][INFO ][o.o.s.c.ConfigurationRepository] [node-01] Wait for cluster to be available ...
[2024-07-11T09:01:43,706][INFO ][o.o.s.c.ConfigurationRepository] [node-01] Wait for cluster to be available ...
[2024-07-11T09:01:43,707][ERROR][o.o.s.a.BackendRegistry  ] [node-01] Not yet initialized (you may need to run securityadmin)
[2024-07-11T09:01:44,706][INFO ][o.o.s.c.ConfigurationRepository] [node-01] Wait for cluster to be available ...
[2024-07-11T09:01:45,706][INFO ][o.o.s.c.ConfigurationRepository] [node-01] Wait for cluster to be available ...
[2024-07-11T09:01:46,209][ERROR][o.o.s.a.BackendRegistry  ] [node-01] Not yet initialized (you may need to run securityadmin)
[2024-07-11T09:01:46,706][INFO ][o.o.s.c.ConfigurationRepository] [node-01] Wait for cluster to be available ...
[2024-07-11T09:01:47,706][INFO ][o.o.s.c.ConfigurationRepository] [node-01] Wait for cluster to be available ...
[2024-07-11T09:01:48,706][INFO ][o.o.s.c.ConfigurationRepository] [node-01] Wait for cluster to be available ...
[2024-07-11T09:01:48,710][ERROR][o.o.s.a.BackendRegistry  ] [node-01] Not yet initialized (you may need to run securityadmin)
[2024-07-11T09:01:48,713][ERROR][o.o.s.a.BackendRegistry  ] [node-01] Not yet initialized (you may need to run securityadmin)
[2024-07-11T09:01:49,062][WARN ][o.o.c.c.ClusterFormationFailureHelper] [node-01] cluster-manager not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover cluster-manager-eligible nodes [opensearch-node1, opensearch-node2, opensearch-node3] to bootstrap a cluster: have discovered [{node-01}{zj4riFT1QV-czKxuUQsSRQ}{2WGyQo84RriOag4_YLFs5w}{192.168.0.66}{192.168.0.66:9300}{dimr}{shard_indexing_pressure_enabled=true}, {node-02}{5RPTs2DHQIG1lJPHxRRBKA}{LxCI9gCKTm6J4osSD5PAlg}{192.168.0.67}{192.168.0.67:9300}{dimr}{shard_indexing_pressure_enabled=true}, {node-03}{qGvT8QxISguFBRdFXrOu-Q}{_SffCkrqT4S5EPFLuPWt4w}{192.168.0.68}{192.168.0.68:9300}{dimr}{shard_indexing_pressure_enabled=true}]; discovery will continue using [127.0.0.1:9300, 192.168.0.67:9300, 192.168.0.68:9300] from hosts providers and [{node-01}{zj4riFT1QV-czKxuUQsSRQ}{2WGyQo84RriOag4_YLFs5w}{192.168.0.66}{192.168.0.66:9300}{dimr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2024-07-11T09:01:49,707][INFO ][o.o.s.c.ConfigurationRepository] [node-01] Wait for cluster to be available ...
[2024-07-11T09:01:50,708][INFO ][o.o.s.c.ConfigurationRepository] [node-01] Wait for cluster to be available ...
[2024-07-11T09:01:51,216][ERROR][o.o.s.a.BackendRegistry  ] [node-01] Not yet initialized (you may need to run securityadmin)
[2024-07-11T09:01:51,708][INFO ][o.o.s.c.ConfigurationRepository] [node-01] Wait for cluster to be available ...
[2024-07-11T09:01:52,708][INFO ][o.o.s.c.ConfigurationRepository] [node-01] Wait for cluster to be available ...
[2024-07-11T09:01:53,708][INFO ][o.o.s.c.ConfigurationRepository] [node-01] Wait for cluster to be available ...
[2024-07-11T09:01:53,718][ERROR][o.o.s.a.BackendRegistry  ] [node-01] Not yet initialized (you may need to run securityadmin)
[2024-07-11T09:01:54,708][INFO ][o.o.s.c.ConfigurationRepository] [node-01] Wait for cluster to be available ...
[2024-07-11T09:01:55,708][INFO ][o.o.s.c.ConfigurationRepository] [node-01] Wait for cluster to be available ...
[2024-07-11T09:01:56,219][ERROR][o.o.s.a.BackendRegistry  ] [node-01] Not yet initialized (you may need to run securityadmin)
[2024-07-11T09:01:56,223][ERROR][o.o.s.a.BackendRegistry  ] [node-01] Not yet initialized (you may need to run securityadmin)
4

@Ascalonking It looks like your node doesn’t connect to an existing cluster. Check the logs on other nodes. Do you see any discovery errors?

5

@pablo

docker logs node-02

[2024-07-11T12:44:27,278][INFO ][o.o.s.c.ConfigurationRepository] [node-02] Wait for cluster to be available ...
[2024-07-11T12:44:27,977][ERROR][o.o.s.a.BackendRegistry  ] [node-02] Not yet initialized (you may need to run securityadmin)
[2024-07-11T12:44:28,278][INFO ][o.o.s.c.ConfigurationRepository] [node-02] Wait for cluster to be available ...
[2024-07-11T12:44:29,278][INFO ][o.o.s.c.ConfigurationRepository] [node-02] Wait for cluster to be available ...
[2024-07-11T12:44:30,278][INFO ][o.o.s.c.ConfigurationRepository] [node-02] Wait for cluster to be available ...
[2024-07-11T12:44:30,478][ERROR][o.o.s.a.BackendRegistry  ] [node-02] Not yet initialized (you may need to run securityadmin)
[2024-07-11T12:44:30,481][ERROR][o.o.s.a.BackendRegistry  ] [node-02] Not yet initialized (you may need to run securityadmin)
[2024-07-11T12:44:31,278][INFO ][o.o.s.c.ConfigurationRepository] [node-02] Wait for cluster to be available ...
[2024-07-11T12:44:32,279][INFO ][o.o.s.c.ConfigurationRepository] [node-02] Wait for cluster to be available ...
[2024-07-11T12:44:32,981][ERROR][o.o.s.a.BackendRegistry  ] [node-02] Not yet initialized (you may need to run securityadmin)
[2024-07-11T12:44:33,279][INFO ][o.o.s.c.ConfigurationRepository] [node-02] Wait for cluster to be available ...
[2024-07-11T12:44:34,279][INFO ][o.o.s.c.ConfigurationRepository] [node-02] Wait for cluster to be available ...
[2024-07-11T12:44:35,280][INFO ][o.o.s.c.ConfigurationRepository] [node-02] Wait for cluster to be available ...
[2024-07-11T12:44:35,483][ERROR][o.o.s.a.BackendRegistry  ] [node-02] Not yet initialized (you may need to run securityadmin)
[2024-07-11T12:44:36,280][INFO ][o.o.s.c.ConfigurationRepository] [node-02] Wait for cluster to be available ...
[2024-07-11T12:44:37,098][WARN ][o.o.c.c.ClusterFormationFailureHelper] [node-02] cluster-manager not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover cluster-manager-eligible nodes [opensearch-node1, opensearch-node2, opensearch-node3] to bootstrap a cluster: have discovered [{node-02}{5RPTs2DHQIG1lJPHxRRBKA}{LxCI9gCKTm6J4osSD5PAlg}{192.168.0.67}{192.168.0.67:9300}{dimr}{shard_indexing_pressure_enabled=true}, {node-01}{zj4riFT1QV-czKxuUQsSRQ}{2WGyQo84RriOag4_YLFs5w}{192.168.0.66}{192.168.0.66:9300}{dimr}{shard_indexing_pressure_enabled=true}, {node-03}{qGvT8QxISguFBRdFXrOu-Q}{_SffCkrqT4S5EPFLuPWt4w}{192.168.0.68}{192.168.0.68:9300}{dimr}{shard_indexing_pressure_enabled=true}]; discovery will continue using [192.168.0.66:9300, 127.0.0.1:9300, 192.168.0.68:9300] from hosts providers and [{node-02}{5RPTs2DHQIG1lJPHxRRBKA}{LxCI9gCKTm6J4osSD5PAlg}{192.168.0.67}{192.168.0.67:9300}{dimr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2024-07-11T12:44:37,280][INFO ][o.o.s.c.ConfigurationRepository] [node-02] Wait for cluster to be available ...
[2024-07-11T12:44:37,984][ERROR][o.o.s.a.BackendRegistry  ] [node-02] Not yet initialized (you may need to run securityadmin)
[2024-07-11T12:44:37,987][ERROR][o.o.s.a.BackendRegistry  ] [node-02] Not yet initialized (you may need to run securityadmin)
[2024-07-11T12:44:38,280][INFO ][o.o.s.c.ConfigurationRepository] [node-02] Wait for cluster to be available ...
[2024-07-11T12:44:39,280][INFO ][o.o.s.c.ConfigurationRepository] [node-02] Wait for cluster to be available ...
[2024-07-11T12:44:40,281][INFO ][o.o.s.c.ConfigurationRepository] [node-02] Wait for cluster to be available ...
[2024-07-11T12:44:40,491][ERROR][o.o.s.a.BackendRegistry  ] [node-02] Not yet initialized (you may need to run securityadmin)
[2024-07-11T12:44:41,281][INFO ][o.o.s.c.ConfigurationRepository] [node-02] Wait for cluster to be available ...
[2024-07-11T12:44:42,281][INFO ][o.o.s.c.ConfigurationRepository] [node-02] Wait for cluster to be available ...
[2024-07-11T12:44:42,992][ERROR][o.o.s.a.BackendRegistry  ] [node-02] Not yet initialized (you may need to run securityadmin)
[2024-07-11T12:44:43,281][INFO ][o.o.s.c.ConfigurationRepository] [node-02] Wait for cluster to be available ...

docker logs node-03

[2024-07-11T12:47:56,966][INFO ][o.o.s.c.ConfigurationRepository] [node-03] Wait for cluster to be available ...
[2024-07-11T12:47:57,966][INFO ][o.o.s.c.ConfigurationRepository] [node-03] Wait for cluster to be available ...
[2024-07-11T12:47:58,140][ERROR][o.o.s.a.BackendRegistry  ] [node-03] Not yet initialized (you may need to run securityadmin)
[2024-07-11T12:47:58,966][INFO ][o.o.s.c.ConfigurationRepository] [node-03] Wait for cluster to be available ...
[2024-07-11T12:47:59,966][INFO ][o.o.s.c.ConfigurationRepository] [node-03] Wait for cluster to be available ...
[2024-07-11T12:48:00,642][ERROR][o.o.s.a.BackendRegistry  ] [node-03] Not yet initialized (you may need to run securityadmin)
[2024-07-11T12:48:00,966][INFO ][o.o.s.c.ConfigurationRepository] [node-03] Wait for cluster to be available ...
[2024-07-11T12:48:01,967][INFO ][o.o.s.c.ConfigurationRepository] [node-03] Wait for cluster to be available ...
[2024-07-11T12:48:02,967][INFO ][o.o.s.c.ConfigurationRepository] [node-03] Wait for cluster to be available ...
[2024-07-11T12:48:03,143][ERROR][o.o.s.a.BackendRegistry  ] [node-03] Not yet initialized (you may need to run securityadmin)
[2024-07-11T12:48:03,145][ERROR][o.o.s.a.BackendRegistry  ] [node-03] Not yet initialized (you may need to run securityadmin)
[2024-07-11T12:48:03,967][INFO ][o.o.s.c.ConfigurationRepository] [node-03] Wait for cluster to be available ...
[2024-07-11T12:48:04,968][INFO ][o.o.s.c.ConfigurationRepository] [node-03] Wait for cluster to be available ...
[2024-07-11T12:48:05,648][ERROR][o.o.s.a.BackendRegistry  ] [node-03] Not yet initialized (you may need to run securityadmin)
[2024-07-11T12:48:05,968][INFO ][o.o.s.c.ConfigurationRepository] [node-03] Wait for cluster to be available ...
[2024-07-11T12:48:06,282][WARN ][o.o.c.c.ClusterFormationFailureHelper] [node-03] cluster-manager not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover cluster-manager-eligible nodes [opensearch-node1, opensearch-node2, opensearch-node3] to bootstrap a cluster: have discovered [{node-03}{qGvT8QxISguFBRdFXrOu-Q}{_SffCkrqT4S5EPFLuPWt4w}{192.168.0.68}{192.168.0.68:9300}{dimr}{shard_indexing_pressure_enabled=true}, {node-01}{zj4riFT1QV-czKxuUQsSRQ}{2WGyQo84RriOag4_YLFs5w}{192.168.0.66}{192.168.0.66:9300}{dimr}{shard_indexing_pressure_enabled=true}, {node-02}{5RPTs2DHQIG1lJPHxRRBKA}{LxCI9gCKTm6J4osSD5PAlg}{192.168.0.67}{192.168.0.67:9300}{dimr}{shard_indexing_pressure_enabled=true}]; discovery will continue using [192.168.0.66:9300, 192.168.0.67:9300, 127.0.0.1:9300] from hosts providers and [{node-03}{qGvT8QxISguFBRdFXrOu-Q}{_SffCkrqT4S5EPFLuPWt4w}{192.168.0.68}{192.168.0.68:9300}{dimr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2024-07-11T12:48:06,968][INFO ][o.o.s.c.ConfigurationRepository] [node-03] Wait for cluster to be available ...
[2024-07-11T12:48:07,969][INFO ][o.o.s.c.ConfigurationRepository] [node-03] Wait for cluster to be available ...
[2024-07-11T12:48:08,147][ERROR][o.o.s.a.BackendRegistry  ] [node-03] Not yet initialized (you may need to run securityadmin)
[2024-07-11T12:48:08,969][INFO ][o.o.s.c.ConfigurationRepository] [node-03] Wait for cluster to be available ...
[2024-07-11T12:48:09,969][INFO ][o.o.s.c.ConfigurationRepository] [node-03] Wait for cluster to be available ...
[2024-07-11T12:48:10,649][ERROR][o.o.s.a.BackendRegistry  ] [node-03] Not yet initialized (you may need to run securityadmin)
[2024-07-11T12:48:10,653][ERROR][o.o.s.a.BackendRegistry  ] [node-03] Not yet initialized (you may need to run securityadmin)
[2024-07-11T12:48:10,969][INFO ][o.o.s.c.ConfigurationRepository] [node-03] Wait for cluster to be available ...
[2024-07-11T12:48:11,969][INFO ][o.o.s.c.ConfigurationRepository] [node-03] Wait for cluster to be available ...
6

@Ascalonking It looks like your cluster didn’t form at all.
I’ve checked your docker-compose and test it. I couldn’t get it working until I’ve change your discovery configuration as per below

services:
  opensearch-node1: # This is also the hostname of the container within the Docker network (i.e. https://opensearch-node1/)
    image: opensearch:2.15.0 # Specifying the latest available image - modify if you want a specific version
    container_name: opensearch-node1
    environment:
      - cluster.name=opensearch-cluster # Name the cluster
      - node.name=node-01 # Name the node that will run in this container
      - discovery.seed_hosts= opensearch-node1,opensearch-node2,opensearch-node3# Nodes to look for when discovering the cluster
      - cluster.initial_cluster_manager_nodes= node-01,node-02,node-03 # Nodes eligible to serve as cluster manager
7

@pablo

The problem has not gone away, when running the command curl https://localhost:9200 -ku 'admin:Qwerty12345678#' also displays an error
OpenSearch Security not initialized

8

I found what was wrong

The problem was in the discovery.seed_hosts line, I needed to specify the host names

services:
  opensearch-node1: # This is also the hostname of the container within the Docker network (i.e. https://opensearch-node1/)
    image: opensearch:2.15.0 # Specifying the latest available image - modify if you want a specific version
    container_name: opensearch-node1
    environment:
      - cluster.name=opensearch-cluster # Name the cluster
      - node.name=node-01 # Name the node that will run in this container
      - discovery.seed_hosts=node-01,node-02,node-03 # Nodes to look for when discovering the cluster
      - cluster.initial_cluster_manager_nodes=node-01,node-02,node-03 # Nodes eligible to serve as cluster manager

Now i have problem with opensearch dashboard

Thanks @pablo for helping me

1 Like