Can't start 1.0.0 securityadmin.sh fails

Hello,

I’ve been running with 0.9.0 for several weeks.

Can’t seem to run securityadmin.sh.

Also, note that security/roles.yml at 436fdd97dcbf0f6be9fc9808bfb0df24e31dad67 · opendistro-for-elasticsearch/security · GitHub has roles defined, but security/roles.yml at main · opendistro-for-elasticsearch/security · GitHub does not ? (perhaps this is related)

Trying to start up a fresh instance with 1.0.0 is failing:

$ dc exec elastic-service plugins/opendistro_security/tools/securityadmin.sh -f plugins/opendistro_security/securityconfig/config.yml -icl -nhnv -cert config/kirk.pem -cacert config/root-ca.pem -key config/kirk-key.pem -t config
Open Distro Security Admin v7
Will connect to localhost:9300 ... done
Connected as CN=kirk,OU=client,O=client,L=test,C=de
Elasticsearch Version: 7.0.1
Open Distro Security Version: 1.0.0.0
Contacting elasticsearch cluster 'elasticsearch' and wait for YELLOW clusterstate ...
Clustername: elasticsearch
Clusterstate: GREEN
Number of nodes: 1
Number of data nodes: 1
.opendistro_security index already exists, so we do not need to create one.
Populate config from /usr/share/elasticsearch
Force type: config
Will update '_doc/config' with plugins/opendistro_security/securityconfig/config.yml
   SUCC: Configuration for 'config' created or updated
FAIL: 1 nodes reported failures. First failure is FailedNodeException[Failed node [RFE6L_ccQ4GSbj6QtS0_gQ]]; nested: RemoteTransportException[[node-1][172.28.0.2:9300][cluster:admin/opendistro_security/config/update[n]]]; nested: NotSerializableExceptionWrapper[static_resource_exception: Unable to load static roles];
FAIL: Expected 1 nodes to return response, but got 0
Done with failures
1 Like

I also met. But I found that it seems to have no effect…

To clarify, kibana cannot connect

kibana-service | {"type":"log","@timestamp":"2019-07-08T14:06:37Z","tags":["status","plugin:elasticsearch@undefined","error"],"pid":1,"state":"red","message":"Status changed from yellow to red - Service Unavailable","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}

$ curl -k https://admin:admin@esproxy-service:9200/
Open Distro Security not initialized.

Same here:
Populate config from /root/elastic-backup/v7
Will update ‘_doc/config’ with ./config.yml
SUCC: Configuration for ‘config’ created or updated
Will update ‘_doc/roles’ with ./roles.yml
SUCC: Configuration for ‘roles’ created or updated
Will update ‘_doc/rolesmapping’ with ./roles_mapping.yml
SUCC: Configuration for ‘rolesmapping’ created or updated
Will update ‘_doc/internalusers’ with ./internal_users.yml
SUCC: Configuration for ‘internalusers’ created or updated
Will update ‘_doc/actiongroups’ with ./action_groups.yml
SUCC: Configuration for ‘actiongroups’ created or updated
Will update ‘_doc/tenants’ with ./tenants.yml
SUCC: Configuration for ‘tenants’ created or updated
FAIL: 1 nodes reported failures. First failure is FailedNodeException[Failed node [JLj4F0J2RK6Zhp2ysEqCBw]]; nested: RemoteTransportException[[tech1][z.b.c.d:9300][cluster:admin/opendistro_security/config/update[n]]]; nested: NotSerializableExceptionWrapper[static_resource_exception: Cannot override static roles];

similar issue here, won’t let me migrate.
elasticsearch, kibana and cerebro is finally running, but the securityadmins script is unable to connect

[root@odfe ~]# /usr/share/elasticsearch/plugins/opendistro_security/tools/securityadmin.sh -migrate ~/my-backup-dir -nhnv -cacert /etc/elasticsearch/root-ca.pem -cert /etc/elasticsearch/kirk.pem -key /etc/elasticsearch/kirk-key.pem
WARNING: JAVA_HOME not set, will use /bin/java
Open Distro Security Admin v7
Will connect to localhost:9300 ... done
ERR: Cannot connect to Elasticsearch. Please refer to elasticsearch logfile for more information
Trace:
NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{P0gK28WxTTi-JW6PhB9MxA}{localhost}{127.0.0.1:9300}]]
	at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:352)
	at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:248)
	at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:57)
	at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:386)
	at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:393)
	at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:382)
	at com.amazon.opendistroforelasticsearch.security.tools.OpenDistroSecurityAdmin.execute(OpenDistroSecurityAdmin.java:519)
	at com.amazon.opendistroforelasticsearch.security.tools.OpenDistroSecurityAdmin.main(OpenDistroSecurityAdmin.java:152)

UPDATE:

since I could not figure out why the securityadmin.sh just fail on migration and it was not possible to move forward with either way because of stating legacy format , I simple deleted the whole ‘.opendistro_security’ and created a new one from scratch, which fortunately worked with securityadmin.sh

So i came to the conclusion that securityadmin.sh has issues to migrate .opendistro_security from 0.x.x to 1.x.x !

Legacy index '.opendistro_security' (ES 6) detected (or forced). You should migrate the configuration!

delete ‘.opendistro_security’ index


[root@odfe esbackup]# /usr/share/elasticsearch/plugins/opendistro_security/tools/securityadmin.sh -dci -icl -nhnv -cacert /etc/elasticsearch/root-ca.pem -cert /etc/elasticsearch/kirk.pem -key /etc/elasticsearch/kirk-key.pem

WARNING: JAVA_HOME not set, will use /bin/java

Open Distro Security Admin v7

Will connect to localhost:9300 ... done

Connected as CN=kirk,OU=client,O=client,L=test,C=de

Elasticsearch Version: 7.0.1

Open Distro Security Version: 1.0.0.0

Contacting elasticsearch cluster 'elasticsearch' and wait for YELLOW clusterstate ...

Clustername: odfe

Clusterstate: YELLOW

Number of nodes: 1

Number of data nodes: 1

Deleted index '.opendistro_security'

recreate ‘.opendistro_security’ index


[root@odfe esbackup]# "/usr/share/elasticsearch/plugins/opendistro_security/tools/securityadmin.sh" -cd "/usr/share/elasticsearch/plugins/opendistro_security/securityconfig" -icl -key "/etc/elasticsearch/kirk-key.pem" -cert "/etc/elasticsearch/kirk.pem" -cacert "/etc/elasticsearch/root-ca.pem" -nhnv

WARNING: JAVA_HOME not set, will use /bin/java

Open Distro Security Admin v7

Will connect to localhost:9300 ... done

Connected as CN=kirk,OU=client,O=client,L=test,C=de

Elasticsearch Version: 7.0.1

Open Distro Security Version: 1.0.0.0

Contacting elasticsearch cluster 'elasticsearch' and wait for YELLOW clusterstate ...

Clustername: odfe

Clusterstate: YELLOW

Number of nodes: 1

Number of data nodes: 1

.opendistro_security index does not exists, attempt to create it ... done (0-all replicas)

Populate config from /usr/share/elasticsearch/plugins/opendistro_security/securityconfig/

Will update '_doc/config' with /usr/share/elasticsearch/plugins/opendistro_security/securityconfig/config.yml

SUCC: Configuration for 'config' created or updated

Will update '_doc/roles' with /usr/share/elasticsearch/plugins/opendistro_security/securityconfig/roles.yml

SUCC: Configuration for 'roles' created or updated

Will update '_doc/rolesmapping' with /usr/share/elasticsearch/plugins/opendistro_security/securityconfig/roles_mapping.yml

SUCC: Configuration for 'rolesmapping' created or updated

Will update '_doc/internalusers' with /usr/share/elasticsearch/plugins/opendistro_security/securityconfig/internal_users.yml

SUCC: Configuration for 'internalusers' created or updated

Will update '_doc/actiongroups' with /usr/share/elasticsearch/plugins/opendistro_security/securityconfig/action_groups.yml

SUCC: Configuration for 'actiongroups' created or updated

Will update '_doc/tenants' with /usr/share/elasticsearch/plugins/opendistro_security/securityconfig/tenants.yml

SUCC: Configuration for 'tenants' created or updated

Done with success

Hi @giomac and @bpwalsh, see https://github.com/opendistro-for-elasticsearch/security/issues/87#issuecomment-511042550 for a discussion of what’s going on with roles.yml.

Unless you’re adding custom roles to that file in 0.x, you can delete everything except the new metadata and let the Security plugin add in the static roles automatically. I hope that does the trick.