Adinternal/cron is unauthorized for user _system

Plugins Machine Learning
1

Hello,

i have setup a normal 7.10.0 elasticsearch cluster, kibana and beats (metric- and filebeat on clusterserver) with multiple nodes which all communicate SSL-encrypted.

On every node of the elasticsearch cluster i installed open-distro-anomaly-detection and opendistro-job-scheduler in the version 1.12.0.0 and i installed opendistroAnomalyDetectionKibana@1.12.0.0 on kibana.
This works fine, but every hour i get the following error. Have i forgotten to install or configure something wrong?

[2021-01-29T00:49:46,203][ERROR][c.a.o.a.c.HourlyCron     ] [bdp-node-12] Hourly maintenance has exception.
org.elasticsearch.ElasticsearchSecurityException: action [cluster:admin/opendistro/adinternal/cron] is unauthorized for user [_system]
        at org.elasticsearch.xpack.core.security.support.Exceptions.authorizationError(Exceptions.java:34) ~[x-pack-core-7.10.0.jar:7.10.0]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.denialException(AuthorizationService.java:613) ~[x-pack-security-7.10.0.jar:7.10.0]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeSystemUser(AuthorizationService.java:404) [x-pack-security-7.10.0.jar:7.10.0]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:205) [x-pack-security-7.10.0.jar:7.10.0]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.authorizeRequest(SecurityActionFilter.java:173) [x-pack-security-7.10.0.jar:7.10.0]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:159) [x-pack-security-7.10.0.jar:7.10.0]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.10.0.jar:7.10.0]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:323) [x-pack-security-7.10.0.jar:7.10.0]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:384) [x-pack-security-7.10.0.jar:7.10.0]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:395) [x-pack-security-7.10.0.jar:7.10.0]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:320) [x-pack-security-7.10.0.jar:7.10.0]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:261) [x-pack-security-7.10.0.jar:7.10.0]
        at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:156) [x-pack-security-7.10.0.jar:7.10.0]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:156) [x-pack-security-7.10.0.jar:7.10.0]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$1(SecurityActionFilter.java:93) [x-pack-security-7.10.0.jar:7.10.0]
        at org.elasticsearch.xpack.core.security.SecurityContext.executeAsUser(SecurityContext.java:138) [x-pack-core-7.10.0.jar:7.10.0]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:91) [x-pack-security-7.10.0.jar:7.10.0]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177) [elasticsearch-7.10.0.jar:7.10.0]
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:155) [elasticsearch-7.10.0.jar:7.10.0]
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:83) [elasticsearch-7.10.0.jar:7.10.0]
        at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:86) [elasticsearch-7.10.0.jar:7.10.0]
        at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:75) [elasticsearch-7.10.0.jar:7.10.0]
        at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:412) [elasticsearch-7.10.0.jar:7.10.0]
        at com.amazon.opendistroforelasticsearch.ad.cluster.HourlyCron.run(HourlyCron.java:49) [opendistro-anomaly-detection-1.12.0.0.jar:1.12.0.0]
        at org.elasticsearch.threadpool.Scheduler$ReschedulingRunnable.doRun(Scheduler.java:213) [elasticsearch-7.10.0.jar:7.10.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:737) [elasticsearch-7.10.0.jar:7.10.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.10.0.jar:7.10.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]

I have seen, that Open Distro alert plugin elasticsearch exception has a similar exception and thats it not possible with xpack, but kibana - Enable xpack features on Open Distro For Elasticsearch - Stack Overflow stated, that some xpack features are included in BASIC version.

So doesn’t it definitely work with xpack installed or can these ERRORs be safely ignored, because the installation and kibana visualisation seems to work.

2

Thanks @vielfarbig for reaching out.
All opendistro plugins are not compatible with x-pack enabled features, in this case x-pack enabled security.
That said, we haven’t tested it with x-pack and cron in ad is a frequent job which runs to take care of some critical pieces of Anomaly Detection where we manage cleaning up of checkpoints, models etc. If this user does not have enough permissions to run the cron, Anomaly Detection plugin might be impaired.

3

Thanks @vemsarat for your reply.
If i actually try to use the ad plugin i get Elasticsearch security config - #4 by bpavani as well. So is there a way around to somehow “only use” the security plugin for opendistro plugins and xpack for the other elasticsearch components? As i understand
https://github.com/opendistro-for-elasticsearch/security/issues/37 is an issue regarding this and #issuecomment-747491504 says that its on the way. Is there a time plan? Is there a current somehow “hacky” solution?

If there is no option to get ad plugin with xpack woring on the same cluster, can one setup a second opendistro cluster besides a normal elastiseach cluster and use the ad plugin on the opendistro cluster and let the opendistro and elasticsearch cluster exchange data. So are opendistro and elasticsearch cluster fully integrated with each other?

4

Thats a great question. I did look into the post from bpavani@.
There is no official plan on ODFE to work with x-pack enabled security. ODFE community members might have out of the box solutions, but we haven’t seen any so far.

To answer the second part of the question, from my intuition i believe it will be tricky to get things working with 1 elasticsearch node and other as odfe node. Again as I said, we haven’t tested it so cannot confirm if it can really work but I believe there are going to be some challenges to get it work especially with security.