Hi,
I’m upgrading our production Elasticsearch OpenDistro from version 1.9.0 to 1.13.0.
Our cluster is writing audit data to another es cluster using opendistro_security plugin.
I’m doing it via rolling upgrade which means adding upgraded hosts to the cluster and migrating to them.
I’ve already migrated successfully to the new version data nodes, and now I’m trying to upgrade the coordinating nodes.
The issue I’ve encountered is when I point clients to the new coordinating nodes, I don’t see the writes to the query log in the audit cluster.
in the logs I see the following, the current coordinating node log:
[2021-05-10T13:27:46,065][INFO ][c.a.o.s.OpenDistroSecurityPlugin] [es-od-research-upgrade-coordinating-2] 3 Open Distro Security modules loaded so far: [Module [type=REST_MANAGEMENT_API, implementing class=com.amazon.opendistroforelasticsearch.security.dlic.rest.api.OpenDistroSecurityRestApiActions], Module [type=MULTITENANCY, implementing class=com.amazon.opendistroforelasticsearch.security.configuration.PrivilegesInterceptorImpl], Module [type=AUDITLOG, implementing class=com.amazon.opendistroforelasticsearch.security.auditlog.impl.AuditLogImpl]]
[2021-05-10T13:27:46,066][INFO ][c.a.o.s.c.ConfigurationRepository] [es-od-research-upgrade-coordinating-2] Background init thread started. Install default config?: true
[2021-05-10T13:27:46,271][INFO ][c.a.o.s.c.ConfigurationRepository] [es-od-research-upgrade-coordinating-2] Index .opendistro_security already exists
[2021-05-10T13:27:46,460][INFO ][stdout ] [es-od-research-upgrade-coordinating-2] [FINE] No subscribers registered for event class com.amazon.opendistroforelasticsearch.security.securityconf.DynamicConfigFactory$NodesDnModelImpl
the upgraded coordinating node:
[2021-05-10T13:04:31,773][INFO ][c.a.o.s.c.ConfigurationRepository] [es-od-research-upgrade-coordinating-3] Will attempt to create index .opendistro_security and default configs if they are absent
[2021-05-10T13:04:31,774][INFO ][c.a.o.s.OpenDistroSecurityPlugin] [es-od-research-upgrade-coordinating-3] 0 Open Distro Security modules loaded so far:
[2021-05-10T13:04:31,774][INFO ][c.a.o.s.c.ConfigurationRepository] [es-od-research-upgrade-coordinating-3] Background init thread started. Install default config?: true
[2021-05-10T13:04:31,789][INFO ][c.a.o.s.c.ConfigurationRepository] [es-od-research-upgrade-coordinating-3] Index .opendistro_security already exists
[2021-05-10T13:04:31,791][INFO ][c.a.o.s.s.ConfigHelper ] [es-od-research-upgrade-coordinating-3] Will update ‘audit’ with /usr/share/elasticsearch/plugins/opendistro_security/securityconfig/audit.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2021-05-10T13:04:31,840][INFO ][c.a.o.s.s.ConfigHelper ] [es-od-research-upgrade-coordinating-3] Index .opendistro_security already contains doc with id audit, skipping update.
the plugins in the current es nodes:
bash # docker exec es bin/elasticsearch-plugin list
opendistro-anomaly-detection
opendistro-job-scheduler
opendistro-knn
opendistro_alerting
opendistro_index_management
opendistro_performance_analyzer
opendistro_security
opendistro_sql
repository-gcs
the plugins in the current es nodes:
bash # docker exec es bin/elasticsearch-plugin list
opendistro-alerting
opendistro-anomaly-detection
opendistro-asynchronous-search
opendistro-index-management
opendistro-job-scheduler
opendistro-knn
opendistro-performance-analyzer
opendistro-reports-scheduler
opendistro-sql
opendistro_security
repository-gcs
this is the opendistro configs in both versions:
bash # grep opendistro /etc/elasticsearch/elasticsearch.yml
opendistro_security.ssl.transport.pemcert_filepath: node.pem
opendistro_security.ssl.transport.pemkey_filepath: node-key.pem
opendistro_security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
opendistro_security.ssl.transport.enforce_hostname_verification: false
opendistro_security.ssl.http.enabled: false
opendistro_security.ssl.http.pemcert_filepath: node.pem
opendistro_security.ssl.http.pemkey_filepath: node-key.pem
opendistro_security.ssl.http.pemtrustedcas_filepath: root-ca.pem
opendistro_security.allow_default_init_securityindex: true
opendistro_security.nodes_dn:
opendistro_security.authcz.admin_dn:
opendistro_security.audit.config.disabled_rest_categories: OPENDISTRO_SECURITY_INDEX_ATTEMPT
opendistro_security.audit.type: external_elasticsearch
opendistro_security.audit.config.http_endpoints: [ 10.240.1.46:9200,10.240.1.64:9200 ]
#opendistro_security.audit.ignore_users:
#- opendistro_security_anonymous
opendistro_security.audit.ignore_requests: [ “indices:data/write/*”, “/_bulk” ]
opendistro_security.enable_snapshot_restore_privilege: true
opendistro_security.check_snapshot_restore_write_privileges: true
opendistro_security.restapi.roles_enabled: [“all_access”, “security_rest_api_access”]
What is the right way to return the writing to the audit cluster and to load the modules that not loaded in the upgraded version?
Open Distro Security modules loaded so far:
thanks!