Please define an environment variable with a strong passw If a password is not provided, the setup will quit

Hi,

I am running a Rocky Linux controller node and two managed nodes. I am using Ansible to run containers with Portainer on the managed nodes. I pull my repository to set up my Portainer stack based on the repository’s containers. In the stack task, I assign my repository’s Docker Compose file and set an environment variable for ADMIN_PASSWORD as follows:

“name”: “ADMIN_PASSWORD”,
“value”: “{{ adminpassword.stdout }}”

Then, I set the adminpassword in another task as follows:
container: opensearch-node1
command: /bin/bash -c “export OPENSEARCH_INITIAL_ADMIN_PASSWORD={{ adminpassword.stdout }} && /usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh -cd /usr/share/opensearch/config/opensearch-security -icl -key /usr/share/opensearch/config/kirk-key.pem -cert /usr/share/opensearch/config/kirk.pem -cacert /usr/share/opensearch/config/root-ca.pem -nhnv”
chdir: /root

But, my OpenSearch containers go into restart mode on the managed hosts. Upon checking the logs, I see the above error.

In my repository’s Docker Compose file, I have the following configuration for opensearch-node1:

opensearch-node1:
  image: opensearchproject/opensearch:latest
  build: ./components/opensearch
  container_name: opensearch-node1
  restart: always
  environment:
    - cluster.name=opensearch-cluster
    - node.name=opensearch-node1
    - discovery.seed_hosts=opensearch-node1,opensearch-node2
    - cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2
    - bootstrap.memory_lock=true
    - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
    - OPENSEARCH_INITIAL_ADMIN_PASSWORD=${ADMIN_PASSWORD}
  ulimits:
    memlock:
      soft: -1
      hard: -1
    nofile:
      soft: 65536
      hard: 65536
  volumes:
    - opensearch-data1:/usr/share/opensearch/data
    -/root/prodir/project/components/opensearch/config/internal_users.yml:/usr/share/opensearch/config/opensearch-security/internal_users.yml
    -/root/prodir/project/components/opensearch/config/roles.yml:/usr/share/opensearch/config/opensearch-security/roles.yml

/root/prodir/project/components/opensearch/config/roles_mapping.yml:/usr/share/opensearch/config/opensearch-security/roles_mapping.yml

-/root/prodir/project/components/opensearch/config/securityadmin_demo.sh:/usr/share/opensearch/config/securityadmin_demo.sh

and i am doing the same for opensearch-node2

volumes:
opensearch-data1:
opensearch-data2:

In my repository’s OpenSearch container Dockerfile, I have the following:

FROM opensearchproject/opensearch:latest

COPY ./config/internal_users.yml /usr/share/opensearch/config/opensearch-security/
COPY ./config/roles.yml /usr/share/opensearch/config/opensearch-security/
COPY ./config/securityadmin_demo.sh /usr/share/opensearch/config/securityadmin_demo.sh
USER root
RUN chmod +x /usr/share/opensearch/config/securityadmin_demo.sh

USER opensearch

What am I doing wrong here?

Hi @Daniyal,

The error can be caused by the password being too weak (would you mind sharing the full error message?).

please see more here: Setting up a demo configuration - OpenSearch Documentation

You might be interested in: Enhancing security in OpenSearch 2.12.0: The end of the default admin password · OpenSearch

Thanks,
mj

Hi @Mantas,

Thank you for your response. I managed to resolve the issue; I didn’t have the .env file on my host, so it was not being picked up. The second issue, as you mentioned, was with the password not being strong enough.

Now, I am facing another issue. I have defined my internal_users.yml, roles.yml, and roles_mapping.yml files. As you can see in the Docker file, this is how I am copying them. But, the changes are not being reflected when I check inside the container, and I receive a permission error.

No permissions for [indices:admin/create]

The steps i am doing:

  1. Created files inside components/opensearch/config/internal_users.yml and so on
  2. Copying them using Docker file like this
FROM opensearchproject/opensearch:latest

COPY ./config/internal_users.yml /usr/share/opensearch/config/opensearch-security/

COPY ./config/roles.yml /usr/share/opensearch/config/opensearch-security/

COPY ./config/roles_mapping.yml /usr/share/opensearch/config/opensearch-security/

COPY ./config/securityadmin_demo.sh /usr/share/opensearch/config/securityadmin_demo.sh

USER root
RUN chmod +x /usr/share/opensearch/config/securityadmin_demo.sh

USER opensearch

CMD ["/usr/share/opensearch/bin/opensearch"]

Hi @Daniyal,

Glad to read you fixed it.

for your next blocker, have you applied new configurations with ./securityadmin.sh as per:

best,
mj

Hi @Mantas

Thank you providing the resource.

I have changed my docker-compose.yml and included this

volumes:
      - opensearch-data2:/usr/share/opensearch/data
    -./components/opensearch/config/internal_users.yml:/usr/share/opensearch/config/opensearch-security/internal_users.yml
      - ./components/opensearch/config/roles.yml:/usr/share/opensearch/config/opensearch-security/roles.yml
      -./components/opensearch/config/roles_mapping.yml:/usr/share/opensearch/config/opensearch-security/roles_mapping.yml
    ports:
      - 9201:9200
      - 9601:9600
    command: >
      /bin/sh -c "/usr/share/opensearch/bin/opensearch &
      sleep 10;
      tail -f /dev/null"

I get this error https://forum.opensearch.org/t/unable-to-start-opensearch-from-command-line-error-plugins-security-ssl-transport-keystore-filepath/18054

and upon checking the opensearch.yml file

---
cluster.name: docker-cluster

# Bind to all interfaces because we don't know what IP address Docker will assign to us.
network.host: 0.0.0.0

# # minimum_master_nodes need to be explicitly set when bound on a public IP
# # set to 1 to allow single node clusters
# discovery.zen.minimum_master_nodes: 1

# Setting network.host to a non-loopback address enables the annoying bootstrap checks. "Single-node" mode disables them again.
# discovery.type: single-nodeste code here

I am also trying this but get the below error

     /bin/sh -c "/usr/share/opensearch/bin/opensearch &
      sleep 10;
      /usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh \
      -f /usr/share/opensearch/config/opensearch-security/internal_users.yml \
      -t internalusers \
      -icl \
      -nhnv \
      -cacert /usr/share/opensearch/config/root-ca.pem \
      -cert /usr/share/opensearch/config/kirk.pem \
      -key /usr/share/opensearch/config/kirk-key.pem && \
      tail -f /dev/null"
WARNING: Using incubator modules: jdk.incubator.vector
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch/lib/opensearch-2.15.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
WARNING: System::setSecurityManager will be removed in a future release
Jul 08, 2024 1:50:42 PM sun.util.locale.provider.LocaleProviderAdapter <clinit>
WARNING: COMPAT locale provider will be removed in a future release
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/opensearch/lib/opensearch-2.15.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
WARNING: System::setSecurityManager will be removed in a future release
[2024-07-08T13:50:43,988][INFO ][o.o.n.Node               ] [2e254c00edf4] version[2.15.0], pid[7], build[tar/61dbcd0795c9bfe9b81e5762175414bc38bbcadf/2024-06-20T03:26:49.193630411Z], OS[Linux/5.14.0-362.18.1.el9_3.0.1.x86_64/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/21.0.3/21.0.3+9-LTS]
[2024-07-08T13:50:43,997][INFO ][o.o.n.Node               ] [2e254c00edf4] JVM home [/usr/share/opensearch/jdk], using bundled JDK/JRE [true]
[2024-07-08T13:50:44,000][INFO ][o.o.n.Node               ] [2e254c00edf4] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.security.manager=allow, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-18361159957642623826, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.security.manager=allow, --add-modules=jdk.incubator.vector, -Djava.util.concurrent.ForkJoinPool.common.threadFactory=org.opensearch.secure_sm.SecuredForkJoinWorkerThreadFactory, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=/usr/share/opensearch/config/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -Dopensearch.cgroups.hierarchy.override=/, -Xms512m, -Xmx512m, -XX:MaxDirectMemorySize=268435456, -Dopensearch.path.home=/usr/share/opensearch, -Dopensearch.path.conf=/usr/share/opensearch/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]
Security Admin v7
Will connect to localhost:9200
ERR: Seems there is no OpenSearch running on localhost:9200 - Will exit

Because I don’t want run the securityadmin.sh command manually and needs the containers to start with the modifications i have made.

what am i doing wrong? how should i go about running the securityadmin.sh if the problem is with this?

Thank you for your time.

Hi @Daniyal,

You can use the below (in your terminal):

docker exec -it -u 0 <container name> bash

best,
mj

Hi @Mantas

I get the same error when I try to run the docker-compose.yml file with volumes. If i remove my custom files volumes it works fine.

Enabling OpenSearch Security Plugin
Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
OpenSearch 2.12.0 onwards, the OpenSearch Security Plugin a change that requires an initial password for 'admin' user.
Please define an environment variable 'OPENSEARCH_INITIAL_ADMIN_PASSWORD' with a strong password string.
If a password is not provided, the setup will quit.
 For more details, please visit: https://opensearch.org/docs/latest/install-and-configure/install-opensearch/docker/
### OpenSearch Security Demo Installer
### ** Warning: Do not use on production or public reachable systems **
OpenSearch install type: rpm/deb on Linux 5.14.0-362.18.1.el9_3.0.1.x86_64 amd64
OpenSearch config dir: /usr/share/opensearch/config/
OpenSearch config file: /usr/share/opensearch/config/opensearch.yml
OpenSearch bin dir: /usr/share/opensearch/bin/
OpenSearch plugins dir: /usr/share/opensearch/plugins/
OpenSearch lib dir: /usr/share/opensearch/lib/
Detected OpenSearch Version: 2.15.0
Detected OpenSearch Security Version: 2.15.0.0

@Daniyal, this looks like you are trying to execute the command but the cluster is not yet fully booted up.

Make sure:

*     Minimum 8 characters
*     Must contain at least one uppercase letter [A–Z]
*     One lowercase letter [a–z]
*     One digit [0–9]
*     One special character

I guess you define it in your .env file? try testing with aSupperCompleXpas12!!

best,
mj

Hi @Mantas

Yes, I am doing this and using .env file, I am satisfying the password criteria, and when i comment out my volumes it starts working but with volumes it starts to complain regarding the password.

Thanks for your time.

@Daniyal, I see, interesting - if you like share your full docker-compose.yml and dockerfile.yml files (note: blank all/any sensitive data) and I can play with it in my lab.

best,
mj

@Mantas This is my Docker-compose.yml

version: '3.8'

services:
 opensearch-node1: # This is also the hostname of the container within the Docker network (i.e. https://opensearch-node1/)
    image: opensearchproject/opensearch:latest # Specifying the latest available image - modify if you want a specific version
    container_name: opensearch-node1
    environment:
      - cluster.name=opensearch-cluster # Name the cluster
      - node.name=opensearch-node1 # Name the node that will run in this container
      - discovery.seed_hosts=opensearch-node1,opensearch-node2 # Nodes to look for when discovering the cluster
      - cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2 # Nodes eligible to serve as cluster manager
      - bootstrap.memory_lock=true # Disable JVM heap memory swapping
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # Set min and max JVM heap sizes to at least 50% of system RAM
      - OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_INITIAL_ADMIN_PASSWORD}    # Sets the demo admin user password when using demo configuration, required for OpenSearch 2.12 and later
    env_file:
      - .env
    ulimits:
      memlock:
        soft: -1 # Set memlock to unlimited (no soft or hard limit)
        hard: -1
      nofile:
        soft: 65536 # Maximum number of open files for the opensearch user - set to at least 65536
        hard: 65536
    volumes:
      - opensearch-data1:/usr/share/opensearch/data # Creates volume called opensearch-data1 and mounts it to the container
      - ./components/opensearch/config/internal_users.yml:/usr/share/opensearch/config/opensearch-security/internal_users.yml
      - ./components/opensearch/config/roles.yml:/usr/share/opensearch/config/opensearch-security/roles.yml
      - ./components/opensearch/config/roles_mapping.yml:/usr/share/opensearch/config/opensearch-security/roles_mapping.yml
#      - ./components/opensearch/config/securityadmin_demo.sh:/usr/share/opensearch/config/securityadmin_demo.sh
    ports:
      - 9200:9200 # REST API
      - 9600:9600 # Performance Analyzer
#    networks:
#      - opensearch-net # All of the containers will join the same Docker bridge network
  opensearch-node2:
    image: opensearchproject/opensearch:latest # This should be the same image used for opensearch-node1 to avoid issues
    container_name: opensearch-node2
    environment:
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node2
      - discovery.seed_hosts=opensearch-node1,opensearch-node2
      - cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2
      - bootstrap.memory_lock=true
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
      - OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_INITIAL_ADMIN_PASSWORD}
    env_file:
      - .env
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - opensearch-data2:/usr/share/opensearch/data
      - ./components/opensearch/config/internal_users.yml:/usr/share/opensearch/config/opensearch-security/internal_users.yml
      - ./components/opensearch/config/roles.yml:/usr/share/opensearch/config/opensearch-security/roles.yml
      - ./components/opensearch/config/roles_mapping.yml:/usr/share/opensearch/config/opensearch-security/roles_mapping.yml
#      - ./components/opensearch/config/securityadmin_demo.sh:/usr/share/opensearch/config/securityadmin_demo.sh
#    networks:
#      - opensearch-net
  opensearch-dashboards:
    image: opensearchproject/opensearch-dashboards:latest # Make sure the version of opensearch-dashboards matches the version of opensearch installed on other nodes
    container_name: opensearch-dashboards
    ports:
      - 5601:5601 # Map host port 5601 to container port 5601
    expose:
      - "5601" # Expose port 5601 for web access to OpenSearch Dashboards
    environment:
      OPENSEARCH_HOSTS: '["https://opensearch-node1:9200","https://opensearch-node2:9200"]' # Define the OpenSearch nodes that OpenSearch Dashboards will query

volumes:
  opensearch-data1:
  opensearch-data2:
  opensearch-dashboards:

This is Dockerfile


FROM opensearchproject/opensearch:latest

COPY ./config/internal_users.yml /usr/share/opensearch/config/opensearch-security/

COPY ./config/roles.yml /usr/share/opensearch/config/opensearch-security/

COPY ./config/roles_mapping.yml /usr/share/opensearch/config/opensearch-security/

COPY ./config/securityadmin_demo.sh /usr/share/opensearch/config/securityadmin_demo.sh

USER root
RUN chmod +x /usr/share/opensearch/config/securityadmin_demo.sh

USER opensearch

CMD ["/usr/share/opensearch/bin/opensearch"]
ENTRYPOINT ["/usr/share/opensearch/config/securityadmin_demo.sh"]

Thanks a lot

Hi @Daniyal.

The below worked for me:

docker-compose.yml:

version: '3.8'
services:
 opensearch-node1: # This is also the hostname of the container within the Docker network (i.e. https://opensearch-node1/)
    image: opensearchproject/opensearch:latest # Specifying the latest available image - modify if you want a specific version
    container_name: opensearch-node1
    environment:
      - cluster.name=opensearch-cluster # Name the cluster
      - node.name=opensearch-node1 # Name the node that will run in this container
      - discovery.seed_hosts=opensearch-node1,opensearch-node2 # Nodes to look for when discovering the cluster
      - cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2 # Nodes eligible to serve as cluster manager
      - bootstrap.memory_lock=true # Disable JVM heap memory swapping
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # Set min and max JVM heap sizes to at least 50% of system RAM
      - OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_INITIAL_ADMIN_PASSWORD}    # Sets the demo admin user password when using demo configuration, required for OpenSearch 2.12 and later
    env_file:
      - .env
    ulimits:
      memlock:
        soft: -1 # Set memlock to unlimited (no soft or hard limit)
        hard: -1
      nofile:
        soft: 65536 # Maximum number of open files for the opensearch user - set to at least 65536
        hard: 65536
    volumes:
      - opensearch-data1:/usr/share/opensearch/data # Creates volume called opensearch-data1 and mounts it to the container
      - ./opensearch-security/internal_users.yml:/usr/share/opensearch/config/opensearch-security/internal_users.yml
      - ./opensearch-security/roles.yml:/usr/share/opensearch/config/opensearch-security/roles.yml
      - ./opensearch-security/roles_mapping.yml:/usr/share/opensearch/config/opensearch-security/roles_mapping.yml
#      - ./components/opensearch/config/securityadmin_demo.sh:/usr/share/opensearch/config/securityadmin_demo.sh
    ports:
      - 9200:9200 # REST API
      - 9600:9600 # Performance Analyzer

#    networks:
#      - opensearch-net # All of the containers will join the same Docker bridge network
 opensearch-node2:
   image: opensearchproject/opensearch:latest # This should be the same image used for opensearch-node1 to avoid issues
   container_name: opensearch-node2
   environment:
     - cluster.name=opensearch-cluster
     - node.name=opensearch-node2
     - discovery.seed_hosts=opensearch-node1,opensearch-node2
     - cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2
     - bootstrap.memory_lock=true
     - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
     - OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_INITIAL_ADMIN_PASSWORD}
   env_file:
     - .env
   ulimits:
     memlock:
       soft: -1
       hard: -1
     nofile:
       soft: 65536
       hard: 65536
   volumes:
     - opensearch-data2:/usr/share/opensearch/data
     - ./opensearch-security/internal_users.yml:/usr/share/opensearch/config/opensearch-security/internal_users.yml
     - ./opensearch-security/roles.yml:/usr/share/opensearch/config/opensearch-security/roles.yml
     - ./opensearch-security/roles_mapping.yml:/usr/share/opensearch/config/opensearch-security/roles_mapping.yml
 #    - ./components/opensearch/config/securityadmin_demo.sh:/usr/share/opensearch/config/securityadmin_demo.sh
 #   networks:
 #     - opensearch-net
 opensearch-dashboards:
   image: opensearchproject/opensearch-dashboards:latest # Make sure the version of opensearch-dashboards matches the version of opensearch installed on other nodes
   container_name: opensearch-dashboards
   ports:
     - 5601:5601 # Map host port 5601 to container port 5601
   expose:
     - "5601" # Expose port 5601 for web access to OpenSearch Dashboards
   environment:
     OPENSEARCH_HOSTS: '["https://opensearch-node1:9200","https://opensearch-node2:9200"]' # Define the OpenSearch nodes that OpenSearch Dashboards will query
volumes:
  opensearch-data1:
  opensearch-data2:
  opensearch-dashboards:

dockerfile:


FROM opensearchproject/opensearch:latest

COPY ./config/internal_users.yml /usr/share/opensearch/config/opensearch-security/

COPY ./config/roles.yml /usr/share/opensearch/config/opensearch-security/

COPY ./config/roles_mapping.yml /usr/share/opensearch/config/opensearch-security/

COPY ./config/securityadmin_demo.sh /usr/share/opensearch/config/securityadmin_demo.sh

USER root

RUN chmod +x /usr/share/opensearch/config/securityadmin_demo.sh

USER opensearch

CMD ["/usr/share/opensearch/bin/opensearch"]

ENTRYPOINT ["/usr/share/opensearch/config/securityadmin_demo.sh"]

.env:

OPENSEARCH_INITIAL_ADMIN_PASSWORD=aSupperCompleXpas12#

internal_users.yml:


---
_meta:
  type: "internalusers"
  config_version: 2


admin:
  hash: "$2y$12$EXIMN5K9KR/1dcyrVQyL4.V4vlJ/Lsdxa.5josYRgXQvI5fNSi5te" 
 # hash = aSupperCompleXpas12#
  reserved: true
  backend_roles:
  - "admin"
  description: "Demo admin user"
.
.
.
.

let me know if you need any clarification.

Best,
mj

Hi @Mantas

Oh, I was not generating the hash of my password and placing it my internal_users.yml.
Thanks a lot for your help.

But I notice that it gives me the error again, and after some time i get this.

Will connect to localhost:9200 ... done

Which of the files gets precedence or the OpenSearch is configured with if there is Docker file and docker-compose.yml?

e.g

Docker file
COPY ./config/internal_users.yml /usr/share/opensearch/config/opensearch-security/

Docker compose
- ./opensearch-security/internal_users.yml:/usr/share/opensearch/config/opensearch-security/internal_users.yml

Hi @Daniyal,

Would you mind sharing that error message?

as you are not specifying the build in your Docker-compose.yml the Dockerfile ( The contents of a Dockerfile describe how to create and build a Docker image) is not used (sorry for the confusion, it overspilled from my lab) please refer to docker docs for more info:

i.e:

    build:
      context: <path to context/Dockerfile>
      args:

best,
mj