Running odfe culster with another user using docker-compose

[GezimSejdiu]

Hi odfe community,

I have been playing around with odfe and was trying to setup it using the official docker-compose example taken from here: Docker - Open Distro Documentation. It work well as it should be but then moved one level up and trying to setup the security and mounting the data and custom-conf* files via mounted volume on docker-compose. While doing so (even chenging the chmod to the one requested by the odfe) I’m getting an issue that it can;t access the '/usr/share/supervisor/supervisord.log'.

Note: I’m also running it with a different user (pid) which doesn;t have root access and isn’t; mounted to default user elasticsearch which mean the only changes to the docker-compose file above are the

--user ${PID}

and

volumes: 
      - ($pwd)/data:/usr/share/elasticsearch/data

I did follow the docker-compose.yml.j2 template from opendistro-for-elasticsearch/opendistro-build repo for these changes.

See the error below:

Traceback (most recent call last):
  File "/usr/bin/supervisord", line 9, in <module>
    load_entry_point('supervisor==4.2.0', 'console_scripts', 'supervisord')()
  File "/usr/lib/python2.7/site-packages/supervisor-4.2.0-py2.7.egg/supervisor/supervisord.py", line 361, in main
    go(options)
  File "/usr/lib/python2.7/site-packages/supervisor-4.2.0-py2.7.egg/supervisor/supervisord.py", line 371, in go
    d.main()
  File "/usr/lib/python2.7/site-packages/supervisor-4.2.0-py2.7.egg/supervisor/supervisord.py", line 72, in main
    self.options.make_logger()
  File "/usr/lib/python2.7/site-packages/supervisor-4.2.0-py2.7.egg/supervisor/options.py", line 1476, in make_logger
    backups=self.logfile_backups,
  File "/usr/lib/python2.7/site-packages/supervisor-4.2.0-py2.7.egg/supervisor/loggers.py", line 417, in handle_file
    handler = RotatingFileHandler(filename, 'a', maxbytes, backups)
  File "/usr/lib/python2.7/site-packages/supervisor-4.2.0-py2.7.egg/supervisor/loggers.py", line 213, in __init__
    FileHandler.__init__(self, filename, mode)
  File "/usr/lib/python2.7/site-packages/supervisor-4.2.0-py2.7.egg/supervisor/loggers.py", line 160, in __init__
    self.stream = open(filename, mode)
IOError: [Errno 13] Permission denied: '/usr/share/supervisor/supervisord.log'

Any help would be more than welcome, any if you were able to setup the odfe using a different user without following the instructions here: Docker - Open Distro Documentation (this works as soon as we use named volumes for the data and pass the configurations files).

Best regards,

[sayaligaikawad]

Hi @GezimSejdiu,

Thank you for approaching. Can you tell us what version of ODFE are you using, which user are you trying to run docker with and how are you trying to access the logs?

Thanks!

[GezimSejdiu]

Hi @sayaligaikawad, thanks a lot for your prompt reply.

Sure, I’m using odfe version 1.9.0 and a user can be any AD user which does have r/w access to that particular volume. We can take an arbitrary user i.e. 555:555 (passed via PROCESS_UID) which isn’t in a group or root i.e. 555:0. Let me post a slightly modified version of docker-compose I’m trying to use:

version: '3'
services:
  odfe-node1:
    image: amazon/opendistro-for-elasticsearch:1.9.0
    container_name: odfe-node1
    user: ${PROCESS_UID}
    environment:
      - cluster.name=odfe-cluster
      - node.name=odfe-node1
      - discovery.seed_hosts=odfe-node1
      - cluster.initial_master_nodes=odfe-node1
      - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
      - network.host=0.0.0.0 # required if not using the demo security configuration
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536 # maximum number of open files for the Elasticsearch user, set to at least 65536 on modern systems
        hard: 65536
    volumes:
      - ./odfe-data:/usr/share/elasticsearch/data
      - ./root-ca.pem:/usr/share/elasticsearch/config/root-ca.pem
      - ./node.pem:/usr/share/elasticsearch/config/node.pem
      - ./node-key.pem:/usr/share/elasticsearch/config/node-key.pem
      - ./admin.pem:/usr/share/elasticsearch/config/admin.pem
      - ./admin-key.pem:/usr/share/elasticsearch/config/admin-key.pem
      - ./custom-elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
      - ./internal_users.yml:/usr/share/elasticsearch/plugins/opendistro_security/securityconfig/internal_users.yml
    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer
    network_mode: "host"
  kibana:
    image: amazon/opendistro-for-elasticsearch-kibana:1.9.0
    container_name: odfe-kibana
    user: ${PROCESS_UID}
    ports:
      - 5601:5601
    expose:
      - "5601"
    environment:
      ELASTICSEARCH_URL: https://odfe-node1:9200
      ELASTICSEARCH_HOSTS: https://odfe-node1:9200
    volumes:
      - ./custom-kibana.yml:/usr/share/kibana/config/kibana.yml
    depends_on:
      - odfe-node1
    network_mode: "host"

Please, see that the data and configuration volumes are bounded to the docker container and not using named volumes.

Reg . logs, I’m not trying to access them, but that is the message I do get when I do docker-compose up -d, specifically after /usr/share/elasticsearch/config/elasticsearch.yml seems to be already configured for Security. Quit.

I also came across this: opendistro-build/docker-entrypoint.sh at main · opendistro-for-elasticsearch/opendistro-build · GitHub but that implies only when running it as root. Another option could be that we will have to write some customized cmd (opendistro-build/docker-entrypoint.sh at main · opendistro-for-elasticsearch/opendistro-build · GitHub) commands.

Many thanks once more for your help and looking forward to any suggestions on how this can be solved. Maybe, indeed I’m missing something between the lines but as of now, I’m not able to figure it out how to do it .

Best regards,

[zhujiaxi]

Hi @GezimSejdiu we have a PR merged for allowing elasticsearch user to access here:

Move PA/RCA files to subdir owned by elasticsearch user (Support)

This change is not in 1.9.0 release yet, we plan to add it in the upcoming release.

Thanks.

[GezimSejdiu]

Hi @zhujiaxi , this is great news. Many thanks for your support. Looking forward to it.

Any change (or is there a concept of a SNAPSHOT at odfe) that I can build from sources and generate that version already? or the whole module integration is a rather complex process which needs a careful plan (as the PR mentioned already as it has to be tested the behavior of the other plugins) and it is better to wait for the official release of the docker image?

Once more, thanks a lot for your answer.

Best regards,

[zhujiaxi]

Hi @GezimSejdiu as of now our release build process have several coupling with other scripts and our internal aws account credentials. We do have plans to decoupling some of those and allows communities to build locally later on.

As of now, the best option is to wait for the next release, as we are already working on ODFE 1.10.0 which you can see on the roadmap:

Release ODFE 1.10.0 based on ES 7.9.0

Thanks.

[GezimSejdiu]

Hi @zhujiaxi,

many thanks for your reply. Appreciate it and thanks a lot for referring to the roadmap about the upcoming release. Looking forward to it and will let you know how it goes with the current setup.

Have a nice day ahead.

Best regards,

[GezimSejdiu]

Hi team,

reopening this thread – but more related to OpenSearch setup.

This issue was resolved previously with the ODFE and now (to my surprise) it is happening with OpenSearch setup.

So I’m constantly getting this:

...   | Killing opensearch process 9
...   | OpenSearch exited with code 143
...   | Performance analyzer exited with code 1
...   | ./opensearch-docker-entrypoint.sh: line 105: /usr/share/opensearch/logs/performance-analyzer.log: Permission denied
o

The permission of logs and data is set to 755 (read and execute access for that user testuser)

drwxr-xr-x 2 testuser testuser 6 Sep  6 13:44 logs

stat -c %a logs data
755
755

but as soon as you move these folders to 777 mode – the setup starts. I still think this is related to the predefined userID : https://github.com/opensearch-project/opensearch-build/blob/main/release/docker/dockerfiles/opensearch.al2.dockerfile#L63-L64 and often that ID is reserved for different users on the cluster.

With ODFE setup this was resolved by changing the mode to 775 and group was change to root via:

mkdir logs
chmod g+rwx -R logs
chgrp 0 -R logs

and it was working just fine.

So, I’m a bit confused and was hard to really debug/look into build-scripts: https://github.com/opensearch-project/opensearch-build to see what permission are reserved to PA.

If anyone else did face the same issue and wants to share the fix, would be great to do so.

PS: I do not want to add this user to sudoers.

Best regards,