Including opendistro_security index in snapshots when TLS on REST layer is disabled

Security
1

Hi,

Acc to Take and Restore Snapshots - Open Distro Documentation, if one needs to restore .opendistro_security index from a snapshot, we must include an admin certificate in the request to elasticsearch REST API. Ex:

curl -k --cert chain.pem --key kirk.key.pem -XPOST 'https://localhost:9200/_snapshot/my-repository/3/_restore?pretty'

However, when opendistro_security.ssl.http.enabled is disabled, TLS is disabled on the REST layer. Only when TLS client authentication is enabled, REST clients can send a TLS certificate with the HTTP request to provide identity information to the security plugin.

My use-case: I use elasticsearch in k8s environment with istio mtls enabled. With that, as pod-to-pod mtls would be handled by istio, tls on REST layer for elasticsearch has been disabled. With this, I can connect to the REST api with user credentials on http (like http://localhost:9200 -u < uname>:< pwd>), but I cannot use certificate-based authentication.
Now, lets say, we want to periodically backup all data in ES (including security configurations, roles, users etc) using elasticsearch snapshots and restore from it in case of disaster recovery. In the current situation, I can only restore other indices while there would be data loss as opendistro_security index cannot be restored.

Is there a way to mitigate this issue? Can an admin user in internal_users.yml (having unlimited permissions like " all_access") be used to restore .opendistro_security index instead of admin-certificates?

Any pointers would be appreciated.
Thanks!

2

Any ideas how we can achieve this?
I found opendistro_security.unsupported.restore.securityindex.enabled parameter with which we can restore the security index without requiring admin certs, but I still have issues in using this restored index.

This is the scenario -

  1. Install opendistro es with default security configurations. (In istio enabled case, tls on rest layer is disabled)
  2. Make changes to security index using the rest api /kibana UI.
  3. Enable backup of elasticsearch data to be taken periodically (creation of snapshots from all indices)
  4. Disaster happens - opendistro-es cluster and the data get deleted.
  5. Re-install elasticsearch
  6. Restore all the data from the snapshot.

Now, in step 4, all the elasticsearch indices were deleted, so all the data including security configurations in opendistro_security index are lost. Upon reinstall in step 5, securityadmin.sh is run with default security configs creating .opendistro_security index.
Now, in step 6, I can’t restore the security index as-is because index with that name already exists. I can rename the index while restoring - like .opendistro_security-restored.
Now, the cluster would have two such indices -
opendistro_security & .opendistro_security-restored

Securityadmin would still use opendistro_security index only. So if i want to make use of the security configurations made in step 2, the data in opendistro_security-restored should be either reindexed/renamed as opendistro_security index.
But upon attempt to delete/reindex this index, I get this error -

[c.a.o.s.p.OpenDistroSecurityIndexAccessEvaluator] indices:data/write/reindex for ‘.opendistro_security’ index is not allowed for a regular user

Any suggestions on how we can achieve this without using admin certs?

3

@shivani The option that you found (opendistro_security.unsupported.restore.securityindex.enabled) is not supported for a number of reasons, above being one of them. The recommended way to back up the security index is by using the securityadmin.sh script with --retrieve option. This will dump all the configs in the relevant files, which can later be used to update or even recreate security index. Hope this helps