Hi there, I’m running a cluster of 6 containerized nodes opensearch 1.3.13 in 3 different VMs using docker-ce, each VM has 1 master + 1 data,ingest, for a total of 6 nodes.
I ran into some space issue, after solving the space issue I now have one of the masters container that is starting but not joining the cluster. In the logs I see opensearch added the data role to this master-only container and its now saying to run the repurpose command to get back to master only role.
According to guidelines I shall stop the node, run the the repurpose command within the container and restart the node; problem is I don’t see a way to stop the node without stopping the container itself, the actual entrypoint of the container doesn’t allow it. Am I missing something? Docker “commit” is an option? Thanks for anyone who want to help.
@lucas72 I don’t think you can do repurpose inside the container. The container runs as long as the OpenSearch service is alive.
How did you solve the storage problem? The repurpose command is used to adjust the role of the node. Have you modified the node’s roles?
Hi Pablo, thanks for your reply. To solve the space issue I simply deleted some indices I didn’t need anymore. I did not touch the role mode, my best guess is that the internal opensearch mechanism deliberately changed the master role to master+data the moment the space issue was there.
@lucas72 As far as I’m aware OpenSearch doesn’t change node roles on the fly. You need to restart (recreate container) to change the role.
If you don’t specify the node roles then OpenSearch will apply default roles which include master and data. However, if you used node.roles: [ ] in opensearch.yml then OpenSearch must respect this setting above the default ones.
There shouldn’t be any issue with changing master to master+data, however reversing to master from master+data would be problematic as data (indices) already exist in the node’s volume.
This is only a suggestion that you should test before applying. Since this node was a data and master for a moment, there is a indices folder in the node’s volume (_data/nodes/0/indices).
As long as this folder exists the container will fail to start with master role only.
If there are no data that are crucial for you, you could delete it and start the container then.
Again, you need double or triple check that node doesn’t contain any critical data.