I’m running OpenSearch in a container and not (yet) as a cluster. The logs are sent from the hosts to OpenSearch via Fluent Bit. At the moment, the entire system is still a proof of concept. As a result, the configurations change very frequently. Unfortunately, this often means I can no longer run securityadmin.sh because OpenSearch throws too many errors.
For example: I had a typo in the username. This caused all hosts running Fluent Bit to send invalid requests to OpenSearch. I wanted to apply the updated internal_users.yml file via securityadmin.sh, but the request failed. I either got a timeout or a message that shards were failing. Even restarting the containers didn’t help, as the number of requests remained high.
As mentioned above, there are currently many changes being made to the files. During normal operation, this probably happens less frequently. But even then, in the event of an error, I need to be able to prioritize running securityadmin.sh without OpenSearch blocking me.
So my question is:
Is there something like a maintenance mode where I can ignore external requests for a short time and run updates/securityadmin.sh without interruption?
Or will the problem resolve itself once I start running OpenSearch as a cluster? Since this is still a POC, I’m not yet running OpenSearch as a cluster. However, I would prefer to take that step if it would solve the problem with updates and securityadmin.sh.
Configuration: OpenSearch running with Docker and singlenode
Could you please share the full securityadmin.sh command you are running, along with the complete error or output from the command and the relevant OpenSearch logs from around the same time?
Also, could you confirm the cluster health when the issue occurs?
I added a new user to my Fluent Bit configuration who was supposed to send data to OpenSearch. A playbook handles this. It was supposed to change the login credentials on all hosts and, in the same process, update the internal_users.yml and roles.yml files in OpenSearch. However, the order was incorrect. First, the Fluent Bit configuration was updated, and requests from over 100 hosts were sent to OpenSearch before OpenSearch had created the new user.For hours, there were requests with incorrect credentials from over 100 hosts:
[timestamp][WARN ][o.o.s.a.BackendRegistry ] [d447c7bbb9c5] Authentication finally failed for fluentbit from IP-Adress
When I noticed the problem, I logged in on the OpenSearch host and first tried to manually import the config.yml file (which had also changed).
Security Admin v7
Will connect to localhost:9200 … done
Connected as “CN=kirk,OU=client,O=client,L=test,C=de”
OpenSearch Version: 3.7.0
Contacting opensearch cluster ‘opensearch’ and wait for YELLOW clusterstate …
Clustername: docker-cluster
Clusterstate: YELLOW
Number of nodes: 1
Number of data nodes: 1
ERR: An unexpected SocketTimeoutException occurred: 1000 MILLISECONDS
Trace:
java.net.SocketTimeoutException: 1000 MILLISECONDS
at org.opensearch.client.RestClient.extractAndWrapCause(RestClient.java:1319)
at org.opensearch.client.RestClient.performRequest(RestClient.java:370)
at org.opensearch.client.RestClient.performRequest(RestClient.java:358)
at org.opensearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1918)
at org.opensearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1901)
at org.opensearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1865)
at org.opensearch.client.IndicesClient.get(IndicesClient.java:768)
at org.opensearch.security.tools.SecurityAdmin.execute(SecurityAdmin.java:771)
at org.opensearch.security.tools.SecurityAdmin.main(SecurityAdmin.java:152)
Caused by: java.net.SocketTimeoutException: 1000 MILLISECONDS
at org.apache.hc.core5.io.SocketTimeoutExceptionFactory.create(SocketTimeoutExceptionFactory.java:50)
at org.apache.hc.core5.reactor.ssl.SSLIOSession$1.timeout(SSLIOSession.java:223)
at org.apache.hc.core5.reactor.InternalDataChannel.onTimeout(InternalDataChannel.java:166)
at org.apache.hc.core5.reactor.InternalChannel.checkTimeout(InternalChannel.java:67)
at org.apache.hc.core5.reactor.SingleCoreIOReactor.checkTimeout(SingleCoreIOReactor.java:261)
at org.apache.hc.core5.reactor.SingleCoreIOReactor.validateActiveChannels(SingleCoreIOReactor.java:183)
at org.apache.hc.core5.reactor.SingleCoreIOReactor.doExecute(SingleCoreIOReactor.java:143)
at org.apache.hc.core5.reactor.AbstractSingleCoreIOReactor.execute(AbstractSingleCoreIOReactor.java:92)
at org.apache.hc.core5.reactor.IOReactorWorker.run(IOReactorWorker.java:44)
at java.base/java.lang.Thread.run(Thread.java:1474)
And I got this error message every time I tried to import a different configuration—for example, to enable internal_users.yml and roles.yml so that the user “fluentbit” could connect.
So I wasn’t able to fix my mistake.
That’s why I wanted to ask if there is a maintenance mode that, in this case, would have ignored external requests or prioritized the execution of securityadmin.sh.
Thanks for the additional information.
OpenSearch does not have a dedicated maintenance mode. Since there were continuous requests from more than 100 Fluent Bit hosts with invalid credentials, could you please temporarily stop the Fluent Bit ingestion and then run securityadmin.sh again to see whether the issue still occurs?