Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpenSearch: 2.17.0
opensearch-k8s-operator: 2.7.0
Describe the issue:
I have an OpenSearch cluster deployed via the opensearch-k8s-operator, with Remote-backed Storage (Azure) configured. I am trying to test and document the disaster recovery steps. i.e. After catastrophic loss and replacement of the Kubernetes cluster, what are the steps to restore the OpenSearch cluster from its Remote-backed Storage?
I (perhaps naively) hoped that deploying a new OpenSearch cluster with the same Remote-backed Storage configuration would allow it to simply start and pick up the latest data. Similar to recovery from an unexpected reboot.
But the recovered OpenSeach cluster’s bootstrap node fails to finish running. (Logs below.)
Are there other steps required to recover a cluster from its Remote-backed Storage?
Configuration:
Data is being written to the Azure Storage Account Blob container before the simulated disaster. Relevant configuration:
spec.general.additionalConfig:
node.attr.remote_store.repository.remote-backed-storage.type: "azure"
node.attr.remote_store.repository.remote-backed-storage.settings.endpoint: "..."
node.attr.remote_store.repository.remote-backed-storage.settings.container: "remote-backed-storage"
cluster.indices.replication.strategy: 'SEGMENT'
cluster.remote_store.state.enabled: "true"
node.attr.remote_store.segment.repository: "remote-backed-storage"
node.attr.remote_store.translog.repository: "remote-backed-storage"
node.attr.remote_store.state.repository: "remote-backed-storage"
References:
- Azure repository | Elasticsearch Guide [8.16] | Elastic
- OpenSearch docs:
- Availability and recovery / Remote-backed storage
- Availability and recovery / Segment replication
- Availability and recovery / Remote-backed storage / Remote cluster state
- Availability and recovery / Remote-backed storage / Migrating to remote-backed storage
Relevant Logs or Screenshots:
Bootstrap node before simulated disaster, UUID-related log entries:
[INFO ][o.o.g.r.RemoteClusterStateService] [cluster-bootstrap-0] There is no valid previous cluster UUID. All cluster UUIDs evaluated are: []
[INFO ][o.o.g.r.RemoteManifestManager] [cluster-bootstrap-0] No manifest file present in remote store for cluster name: cluster, cluster UUID: 5v7pz2y8TGyIGbw9MoSX5w
[ERROR][o.o.g.G.RemotePersistedState] [cluster-bootstrap-0] Latest manifest is not present in remote store for cluster UUID: 5v7pz2y8TGyIGbw9MoSX5w
[INFO ][o.o.c.c.CoordinationState] [cluster-bootstrap-0] cluster UUID set to [5v7pz2y8TGyIGbw9MoSX5w]
Bootstrap node on the attempted recovery cluster (after disaster), UUID-related log entries:
[INFO ][o.o.g.r.RemoteClusterStateService] [cluster-bootstrap-0] Known UUIDs found in remote store : [[5v7pz2y8TGyIGbw9MoSX5w]]
[INFO ][o.o.i.r.RemoteStoreRestoreService] [cluster-bootstrap-0] Restoring cluster state from remote store from cluster UUID : [5v7pz2y8TGyIGbw9MoSX5w]
[INFO ][o.o.g.r.RemoteManifestManager] [cluster-bootstrap-0] No manifest file present in remote store for cluster name: cluster, cluster UUID: 7GS3z_h9Tn2Waf-MnXoemQ
[ERROR][o.o.g.G.RemotePersistedState] [cluster-bootstrap-0] Latest manifest is not present in remote store for cluster UUID: 7GS3z_h9Tn2Waf-MnXoemQ
[INFO ][o.o.c.c.CoordinationState] [cluster-bootstrap-0] cluster UUID set to [7GS3z_h9Tn2Waf-MnXoemQ]
Bootstrap node on the attempted recovery cluster (after disaster), full log:
-> Installing repository-azure
-> Downloading repository-azure from opensearch
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: plugin requires additional permissions @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.lang.RuntimePermission accessDeclaredMembers
* java.lang.RuntimePermission setContextClassLoader
* java.lang.RuntimePermission setFactory
* java.lang.RuntimePermission shutdownHooks
* java.lang.reflect.ReflectPermission suppressAccessChecks
* java.net.NetPermission getProxySelector
* java.net.NetPermission setDefaultAuthenticator
* java.net.SocketPermission * connect,resolve
* java.util.PropertyPermission os.name read
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.
-> Installed repository-azure with folder name repository-azure
Enabling OpenSearch Security Plugin
Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
OpenSearch 2.12.0 onwards, the OpenSearch Security Plugin a change that requires an initial password for 'admin' user.
Please define an environment variable 'OPENSEARCH_INITIAL_ADMIN_PASSWORD' with a strong password string.
If a password is not provided, the setup will quit.
For more details, please visit: https://opensearch.org/docs/latest/install-and-configure/install-opensearch/docker/
### OpenSearch Security Demo Installer
### ** Warning: Do not use on production or public reachable systems **
OpenSearch install type: rpm/deb on Linux 6.5.0-1025-azure amd64
OpenSearch config dir: /usr/share/opensearch/config/
OpenSearch config file: /usr/share/opensearch/config/opensearch.yml
OpenSearch bin dir: /usr/share/opensearch/bin/
OpenSearch plugins dir: /usr/share/opensearch/plugins/
OpenSearch lib dir: /usr/share/opensearch/lib/
Detected OpenSearch Version: 2.17.0
Detected OpenSearch Security Version: 2.17.0.0
/usr/share/opensearch/config/opensearch.yml seems to be already configured for Security. Quit.
Enabling execution of OPENSEARCH_HOME/bin/opensearch-performance-analyzer/performance-analyzer-agent-cli for OpenSearch Performance Analyzer Plugin
WARNING: Using incubator modules: jdk.incubator.vector
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch/lib/opensearch-2.17.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
WARNING: System::setSecurityManager will be removed in a future release
Nov 18, 2024 10:18:49 PM sun.util.locale.provider.LocaleProviderAdapter <clinit>
WARNING: COMPAT locale provider will be removed in a future release
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/opensearch/lib/opensearch-2.17.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
WARNING: System::setSecurityManager will be removed in a future release
[INFO ][o.o.n.Node ] [cluster-bootstrap-0] version[2.17.0], pid[1], build[tar/8586481dc99b1740ca3c7c966aee15ad0fc7b412/2024-09-13T01:04:14.707418737Z], OS[Linux/6.5.0-1025-azure/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/21.0.4/21.0.4+7-LTS]
[INFO ][o.o.n.Node ] [cluster-bootstrap-0] JVM home [/usr/share/opensearch/jdk], using bundled JDK/JRE [true]
[INFO ][o.o.n.Node ] [cluster-bootstrap-0] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.security.manager=allow, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-15627832101566755657, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.security.manager=allow, --add-modules=jdk.incubator.vector, -Djava.util.concurrent.ForkJoinPool.common.threadFactory=org.opensearch.secure_sm.SecuredForkJoinWorkerThreadFactory, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=/usr/share/opensearch/config/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -Dopensearch.cgroups.hierarchy.override=/, -Xmx512M, -Xms512M, -XX:MaxDirectMemorySize=268435456, -Dopensearch.path.home=/usr/share/opensearch, -Dopensearch.path.conf=/usr/share/opensearch/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]
[INFO ][o.a.l.i.v.PanamaVectorizationProvider] [cluster-bootstrap-0] Java vector incubator API enabled; uses preferredBitSize=256; FMA enabled
[INFO ][o.o.s.s.t.SSLConfig ] [cluster-bootstrap-0] SSL dual mode is disabled
[INFO ][o.o.s.OpenSearchSecurityPlugin] [cluster-bootstrap-0] OpenSearch Config path is /usr/share/opensearch/config
[INFO ][o.o.s.s.DefaultSecurityKeyStore] [cluster-bootstrap-0] JVM supports TLSv1.3
[INFO ][o.o.s.s.DefaultSecurityKeyStore] [cluster-bootstrap-0] Config directory is /usr/share/opensearch/config/, from there the key- and truststore files are resolved relatively
[INFO ][o.o.s.s.DefaultSecurityKeyStore] [cluster-bootstrap-0] TLS Transport Client Provider : JDK
[INFO ][o.o.s.s.DefaultSecurityKeyStore] [cluster-bootstrap-0] TLS Transport Server Provider : JDK
[INFO ][o.o.s.s.DefaultSecurityKeyStore] [cluster-bootstrap-0] TLS HTTP Provider : JDK
[INFO ][o.o.s.s.DefaultSecurityKeyStore] [cluster-bootstrap-0] Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2]
[INFO ][o.o.s.s.DefaultSecurityKeyStore] [cluster-bootstrap-0] Enabled TLS protocols for HTTP layer : [TLSv1.3, TLSv1.2]
[INFO ][o.o.s.OpenSearchSecurityPlugin] [cluster-bootstrap-0] Clustername: cluster
[WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-bootstrap-0] Directory /usr/share/opensearch/config has insecure file permissions (should be 0700)
[WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-bootstrap-0] File /usr/share/opensearch/config/opensearch.yml has insecure file permissions (should be 0600)
[WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-bootstrap-0] File /usr/share/opensearch/config/opensearch-security/roles.yml has insecure file permissions (should be 0600)
[WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-bootstrap-0] File /usr/share/opensearch/config/opensearch-security/roles_mapping.yml has insecure file permissions (should be 0600)
[WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-bootstrap-0] File /usr/share/opensearch/config/opensearch-security/nodes_dn.yml has insecure file permissions (should be 0600)
[WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-bootstrap-0] File /usr/share/opensearch/config/opensearch-security/tenants.yml has insecure file permissions (should be 0600)
[WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-bootstrap-0] File /usr/share/opensearch/config/opensearch-security/config.yml has insecure file permissions (should be 0600)
[WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-bootstrap-0] File /usr/share/opensearch/config/opensearch-security/action_groups.yml has insecure file permissions (should be 0600)
[WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-bootstrap-0] File /usr/share/opensearch/config/opensearch-security/allowlist.yml has insecure file permissions (should be 0600)
[WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-bootstrap-0] File /usr/share/opensearch/config/opensearch-security/internal_users.yml has insecure file permissions (should be 0600)
[WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-bootstrap-0] Directory /usr/share/opensearch/config/tls-http has insecure file permissions (should be 0700)
[WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-bootstrap-0] File /usr/share/opensearch/config/tls-http/ca.crt has insecure file permissions (should be 0600)
[WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-bootstrap-0] File /usr/share/opensearch/config/tls-http/tls.key has insecure file permissions (should be 0600)
[WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-bootstrap-0] File /usr/share/opensearch/config/tls-http/tls.crt has insecure file permissions (should be 0600)
[WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-bootstrap-0] File /usr/share/opensearch/config/tls-http/..data has insecure file permissions (should be 0600)
[WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-bootstrap-0] Directory /usr/share/opensearch/config/tls-http/..2024_11_18_22_16_52.1454163967 has insecure file permissions (should be 0700)
[WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-bootstrap-0] File /usr/share/opensearch/config/tls-http/..2024_11_18_22_16_52.1454163967/tls.key has insecure file permissions (should be 0600)
[WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-bootstrap-0] File /usr/share/opensearch/config/tls-http/..2024_11_18_22_16_52.1454163967/tls.crt has insecure file permissions (should be 0600)
[WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-bootstrap-0] File /usr/share/opensearch/config/tls-http/..2024_11_18_22_16_52.1454163967/ca.crt has insecure file permissions (should be 0600)
[WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-bootstrap-0] Directory /usr/share/opensearch/config/tls-transport has insecure file permissions (should be 0700)
[WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-bootstrap-0] File /usr/share/opensearch/config/tls-transport/cluster-bootstrap-0.key has insecure file permissions (should be 0600)
[WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-bootstrap-0] File /usr/share/opensearch/config/tls-transport/cluster-managers-1.crt has insecure file permissions (should be 0600)
[WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-bootstrap-0] File /usr/share/opensearch/config/tls-transport/cluster-managers-0.crt has insecure file permissions (should be 0600)
[WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-bootstrap-0] File /usr/share/opensearch/config/tls-transport/cluster-managers-2.key has insecure file permissions (should be 0600)
[WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-bootstrap-0] File /usr/share/opensearch/config/tls-transport/cluster-managers-2.crt has insecure file permissions (should be 0600)
[WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-bootstrap-0] File /usr/share/opensearch/config/tls-transport/cluster-bootstrap-0.crt has insecure file permissions (should be 0600)
[WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-bootstrap-0] File /usr/share/opensearch/config/tls-transport/ca.crt has insecure file permissions (should be 0600)
[WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-bootstrap-0] File /usr/share/opensearch/config/tls-transport/cluster-managers-0.key has insecure file permissions (should be 0600)
[WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-bootstrap-0] File /usr/share/opensearch/config/tls-transport/cluster-managers-1.key has insecure file permissions (should be 0600)
[WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-bootstrap-0] File /usr/share/opensearch/config/tls-transport/..data has insecure file permissions (should be 0600)
[WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-bootstrap-0] Directory /usr/share/opensearch/config/tls-transport/..2024_11_18_22_16_52.3669037653 has insecure file permissions (should be 0700)
[WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-bootstrap-0] File /usr/share/opensearch/config/tls-transport/..2024_11_18_22_16_52.3669037653/cluster-managers-1.key has insecure file permissions (should be 0600)
[WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-bootstrap-0] File /usr/share/opensearch/config/tls-transport/..2024_11_18_22_16_52.3669037653/cluster-bootstrap-0.crt has insecure file permissions (should be 0600)
[WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-bootstrap-0] File /usr/share/opensearch/config/tls-transport/..2024_11_18_22_16_52.3669037653/ca.crt has insecure file permissions (should be 0600)
[WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-bootstrap-0] File /usr/share/opensearch/config/tls-transport/..2024_11_18_22_16_52.3669037653/cluster-managers-0.key has insecure file permissions (should be 0600)
[WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-bootstrap-0] File /usr/share/opensearch/config/tls-transport/..2024_11_18_22_16_52.3669037653/cluster-managers-2.crt has insecure file permissions (should be 0600)
[WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-bootstrap-0] File /usr/share/opensearch/config/tls-transport/..2024_11_18_22_16_52.3669037653/cluster-bootstrap-0.key has insecure file permissions (should be 0600)
[WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-bootstrap-0] File /usr/share/opensearch/config/tls-transport/..2024_11_18_22_16_52.3669037653/cluster-managers-1.crt has insecure file permissions (should be 0600)
[WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-bootstrap-0] File /usr/share/opensearch/config/tls-transport/..2024_11_18_22_16_52.3669037653/cluster-managers-0.crt has insecure file permissions (should be 0600)
[WARN ][o.o.s.OpenSearchSecurityPlugin] [cluster-bootstrap-0] File /usr/share/opensearch/config/tls-transport/..2024_11_18_22_16_52.3669037653/cluster-managers-2.key has insecure file permissions (should be 0600)
[INFO ][o.o.p.c.c.PluginSettings ] [cluster-bootstrap-0] Config: metricsLocation: /dev/shm/performanceanalyzer/, metricsDeletionInterval: 1, httpsEnabled: false, cleanup-metrics-db-files: true, batch-metrics-retention-period-minutes: 7, rpc-port: 9650, webservice-port 9600
[WARN ][stderr ] [cluster-bootstrap-0] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[WARN ][stderr ] [cluster-bootstrap-0] SLF4J: Defaulting to no-operation (NOP) logger implementation
[WARN ][stderr ] [cluster-bootstrap-0] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[INFO ][o.o.i.r.ReindexPlugin ] [cluster-bootstrap-0] ReindexPlugin reloadSPI called
[INFO ][o.o.i.r.ReindexPlugin ] [cluster-bootstrap-0] Unable to find any implementation for RemoteReindexExtension
[INFO ][o.o.j.JobSchedulerPlugin ] [cluster-bootstrap-0] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions
[INFO ][o.o.j.JobSchedulerPlugin ] [cluster-bootstrap-0] Loaded scheduler extension: opensearch_time_series_analytics, index: .opendistro-anomaly-detector-jobs
[INFO ][o.o.j.JobSchedulerPlugin ] [cluster-bootstrap-0] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
[INFO ][o.o.j.JobSchedulerPlugin ] [cluster-bootstrap-0] Loaded scheduler extension: scheduler_geospatial_ip2geo_datasource, index: .scheduler-geospatial-ip2geo-datasource
[INFO ][o.o.j.JobSchedulerPlugin ] [cluster-bootstrap-0] Loaded scheduler extension: opensearch_sap_job, index: .opensearch-sap--job
[INFO ][o.o.j.JobSchedulerPlugin ] [cluster-bootstrap-0] Loaded scheduler extension: async-query-scheduler, index: .async-query-scheduler
[INFO ][o.o.p.PluginsService ] [cluster-bootstrap-0] loaded module [aggs-matrix-stats]
[INFO ][o.o.p.PluginsService ] [cluster-bootstrap-0] loaded module [analysis-common]
[INFO ][o.o.p.PluginsService ] [cluster-bootstrap-0] loaded module [cache-common]
[INFO ][o.o.p.PluginsService ] [cluster-bootstrap-0] loaded module [geo]
[INFO ][o.o.p.PluginsService ] [cluster-bootstrap-0] loaded module [ingest-common]
[INFO ][o.o.p.PluginsService ] [cluster-bootstrap-0] loaded module [ingest-geoip]
[INFO ][o.o.p.PluginsService ] [cluster-bootstrap-0] loaded module [ingest-user-agent]
[INFO ][o.o.p.PluginsService ] [cluster-bootstrap-0] loaded module [lang-expression]
[INFO ][o.o.p.PluginsService ] [cluster-bootstrap-0] loaded module [lang-mustache]
[INFO ][o.o.p.PluginsService ] [cluster-bootstrap-0] loaded module [lang-painless]
[INFO ][o.o.p.PluginsService ] [cluster-bootstrap-0] loaded module [mapper-extras]
[INFO ][o.o.p.PluginsService ] [cluster-bootstrap-0] loaded module [opensearch-dashboards]
[INFO ][o.o.p.PluginsService ] [cluster-bootstrap-0] loaded module [parent-join]
[INFO ][o.o.p.PluginsService ] [cluster-bootstrap-0] loaded module [percolator]
[INFO ][o.o.p.PluginsService ] [cluster-bootstrap-0] loaded module [rank-eval]
[INFO ][o.o.p.PluginsService ] [cluster-bootstrap-0] loaded module [reindex]
[INFO ][o.o.p.PluginsService ] [cluster-bootstrap-0] loaded module [repository-url]
[INFO ][o.o.p.PluginsService ] [cluster-bootstrap-0] loaded module [search-pipeline-common]
[INFO ][o.o.p.PluginsService ] [cluster-bootstrap-0] loaded module [systemd]
[INFO ][o.o.p.PluginsService ] [cluster-bootstrap-0] loaded module [transport-netty4]
[INFO ][o.o.p.PluginsService ] [cluster-bootstrap-0] loaded plugin [opensearch-alerting]
[INFO ][o.o.p.PluginsService ] [cluster-bootstrap-0] loaded plugin [opensearch-anomaly-detection]
[INFO ][o.o.p.PluginsService ] [cluster-bootstrap-0] loaded plugin [opensearch-asynchronous-search]
[INFO ][o.o.p.PluginsService ] [cluster-bootstrap-0] loaded plugin [opensearch-cross-cluster-replication]
[INFO ][o.o.p.PluginsService ] [cluster-bootstrap-0] loaded plugin [opensearch-custom-codecs]
[INFO ][o.o.p.PluginsService ] [cluster-bootstrap-0] loaded plugin [opensearch-flow-framework]
[INFO ][o.o.p.PluginsService ] [cluster-bootstrap-0] loaded plugin [opensearch-geospatial]
[INFO ][o.o.p.PluginsService ] [cluster-bootstrap-0] loaded plugin [opensearch-index-management]
[INFO ][o.o.p.PluginsService ] [cluster-bootstrap-0] loaded plugin [opensearch-job-scheduler]
[INFO ][o.o.p.PluginsService ] [cluster-bootstrap-0] loaded plugin [opensearch-knn]
[INFO ][o.o.p.PluginsService ] [cluster-bootstrap-0] loaded plugin [opensearch-ml]
[INFO ][o.o.p.PluginsService ] [cluster-bootstrap-0] loaded plugin [opensearch-neural-search]
[INFO ][o.o.p.PluginsService ] [cluster-bootstrap-0] loaded plugin [opensearch-notifications]
[INFO ][o.o.p.PluginsService ] [cluster-bootstrap-0] loaded plugin [opensearch-notifications-core]
[INFO ][o.o.p.PluginsService ] [cluster-bootstrap-0] loaded plugin [opensearch-observability]
[INFO ][o.o.p.PluginsService ] [cluster-bootstrap-0] loaded plugin [opensearch-performance-analyzer]
[INFO ][o.o.p.PluginsService ] [cluster-bootstrap-0] loaded plugin [opensearch-reports-scheduler]
[INFO ][o.o.p.PluginsService ] [cluster-bootstrap-0] loaded plugin [opensearch-security]
[INFO ][o.o.p.PluginsService ] [cluster-bootstrap-0] loaded plugin [opensearch-security-analytics]
[INFO ][o.o.p.PluginsService ] [cluster-bootstrap-0] loaded plugin [opensearch-skills]
[INFO ][o.o.p.PluginsService ] [cluster-bootstrap-0] loaded plugin [opensearch-sql]
[INFO ][o.o.p.PluginsService ] [cluster-bootstrap-0] loaded plugin [opensearch-system-templates]
[INFO ][o.o.p.PluginsService ] [cluster-bootstrap-0] loaded plugin [query-insights]
[INFO ][o.o.p.PluginsService ] [cluster-bootstrap-0] loaded plugin [repository-azure]
[INFO ][o.o.s.OpenSearchSecurityPlugin] [cluster-bootstrap-0] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting 'http.compression: true' in opensearch.yml
[WARN ][stderr ] [cluster-bootstrap-0] WARNING: A restricted method in java.lang.foreign.Linker has been called
[WARN ][stderr ] [cluster-bootstrap-0] WARNING: java.lang.foreign.Linker::downcallHandle has been called by the unnamed module
[WARN ][stderr ] [cluster-bootstrap-0] WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning for this module
[INFO ][o.a.l.s.MemorySegmentIndexInputProvider] [cluster-bootstrap-0] Using MemorySegmentIndexInput and native madvise support with Java 21 or later; to disable start with -Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false
[INFO ][o.o.e.NodeEnvironment ] [cluster-bootstrap-0] using [1] data paths, mounts [[/usr/share/opensearch/data (/dev/root)]], net usable_space [20.5gb], net total_space [28.8gb], types [ext4]
[INFO ][o.o.e.NodeEnvironment ] [cluster-bootstrap-0] heap size [512mb], compressed ordinary object pointers [true]
[INFO ][o.o.n.Node ] [cluster-bootstrap-0] node name [cluster-bootstrap-0], node ID [zc21QCcJSoWFURx_cZ1Odg], cluster name [cluster], roles [cluster_manager]
[INFO ][o.o.e.ExtensionsManager ] [cluster-bootstrap-0] ExtensionsManager initialized
[DEPRECATION][o.o.d.c.s.Settings ] [cluster-bootstrap-0] [index.store.hybrid.mmap.extensions] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[DEPRECATION][o.o.d.c.s.Settings ] [cluster-bootstrap-0] [cluster.initial_master_nodes] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[DEPRECATION][o.o.d.c.s.Settings ] [cluster-bootstrap-0] [plugins.security.ssl.transport.enforce_hostname_verification] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[INFO ][o.o.n.p.NeuralSearch ] [cluster-bootstrap-0] Registering hybrid query phase searcher with feature flag [plugins.neural_search.hybrid_search_disabled]
[WARN ][o.o.s.c.Salt ] [cluster-bootstrap-0] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
[INFO ][o.o.s.a.i.AuditLogImpl ] [cluster-bootstrap-0] Message routing enabled: true
[INFO ][o.o.s.f.SecurityFilter ] [cluster-bootstrap-0] <NONE> indices are made immutable.
[INFO ][o.o.t.b.CircuitBreakerService] [cluster-bootstrap-0] Registered memory breaker.
[INFO ][o.o.m.b.MLCircuitBreakerService] [cluster-bootstrap-0] Registered ML memory breaker.
[INFO ][o.o.m.b.MLCircuitBreakerService] [cluster-bootstrap-0] Registered ML disk breaker.
[INFO ][o.o.m.b.MLCircuitBreakerService] [cluster-bootstrap-0] Registered ML native memory breaker.
[INFO ][o.r.Reflections ] [cluster-bootstrap-0] Reflections took 56 ms to scan 1 urls, producing 26 keys and 67 values
[INFO ][o.r.Reflections ] [cluster-bootstrap-0] Reflections took 2 ms to scan 1 urls, producing 3 keys and 5 values
[WARN ][o.o.s.p.SQLPlugin ] [cluster-bootstrap-0] Master key is a required config for using create and update datasource APIs. Please set plugins.query.datasources.encryption.masterkey config in opensearch.yml in all the cluster nodes. More details can be found here: https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/admin/datasources.rst#master-key-config-for-encrypting-credential-information
[INFO ][o.o.t.NettyAllocator ] [cluster-bootstrap-0] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={opensearch.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=512mb}]
[INFO ][o.o.s.s.t.SSLConfig ] [cluster-bootstrap-0] SSL dual mode is disabled
[INFO ][o.o.d.DiscoveryModule ] [cluster-bootstrap-0] using discovery type [zen] and seed hosts providers [settings]
[WARN ][o.o.g.DanglingIndicesState] [cluster-bootstrap-0] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[INFO ][o.o.p.h.c.PerformanceAnalyzerConfigAction] [cluster-bootstrap-0] PerformanceAnalyzer Enabled: false
[INFO ][o.o.n.Node ] [cluster-bootstrap-0] initialized
[INFO ][o.o.n.Node ] [cluster-bootstrap-0] starting ...
[ERROR][o.o.s.l.BuiltinLogTypeLoader] [cluster-bootstrap-0] Failed loading builtin log types from disk!
otFoundException: null
nio.zipfs.ZipFileSystemProvider.getFileSystem(ZipFileSystemProvider.java:156) ~[?:?]
nio.zipfs.ZipFileSystemProvider.getPath(ZipFileSystemProvider.java:142) ~[?:?]
le.Path.of(Path.java:209) ~[?:?]
le.Paths.get(Paths.java:98) ~[?:?]
tyanalytics.logtype.BuiltinLogTypeLoader.loadBuiltinLogTypes(BuiltinLogTypeLoader.java:73) ~[opensearch-security-analytics-2.17.0.0.jar:2.17.0.0]
tyanalytics.logtype.BuiltinLogTypeLoader.ensureLogTypesLoaded(BuiltinLogTypeLoader.java:62) [opensearch-security-analytics-2.17.0.0.jar:2.17.0.0]
tyanalytics.logtype.BuiltinLogTypeLoader.doStart(BuiltinLogTypeLoader.java:146) [opensearch-security-analytics-2.17.0.0.jar:2.17.0.0]
.lifecycle.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:77) [opensearch-common-2.17.0.jar:2.17.0]
rrayList.forEach(ArrayList.java:1596) [?:?]
ollections$UnmodifiableCollection.forEach(Collections.java:1116) [?:?]
ode.start(Node.java:1564) [opensearch-2.17.0.jar:2.17.0]
rap.Bootstrap.start(Bootstrap.java:339) [opensearch-2.17.0.jar:2.17.0]
rap.Bootstrap.init(Bootstrap.java:413) [opensearch-2.17.0.jar:2.17.0]
rap.OpenSearch.init(OpenSearch.java:181) [opensearch-2.17.0.jar:2.17.0]
rap.OpenSearch.execute(OpenSearch.java:172) [opensearch-2.17.0.jar:2.17.0]
vironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) [opensearch-2.17.0.jar:2.17.0]
mmand.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-2.17.0.jar:2.17.0]
mmand.main(Command.java:101) [opensearch-cli-2.17.0.jar:2.17.0]
rap.OpenSearch.main(OpenSearch.java:138) [opensearch-2.17.0.jar:2.17.0]
rap.OpenSearch.main(OpenSearch.java:104) [opensearch-2.17.0.jar:2.17.0]
[INFO ][o.o.t.TransportService ] [cluster-bootstrap-0] publish_address {cluster-bootstrap-0/10.42.0.18:9300}, bound_addresses {[::]:9300}
[INFO ][o.o.n.r.RemoteStoreNodeService] [cluster-bootstrap-0] remote backed storage repository with name [remote-backed-storage] and type [azure] created
[INFO ][o.o.n.r.RemoteStoreNodeService] [cluster-bootstrap-0] successfully verified [remote-backed-storage] repository
[INFO ][o.o.g.r.RemoteClusterStateService] [cluster-bootstrap-0] Known UUIDs found in remote store : [[5v7pz2y8TGyIGbw9MoSX5w]]
[INFO ][o.o.i.r.RemoteStoreRestoreService] [cluster-bootstrap-0] Restoring cluster state from remote store from cluster UUID : [5v7pz2y8TGyIGbw9MoSX5w]
[INFO ][o.o.i.r.RemoteStoreRestoreService] [cluster-bootstrap-0] Restoring ClusterState with Remote State version [67]
[INFO ][o.o.b.BootstrapChecks ] [cluster-bootstrap-0] bound or publishing to a non-loopback address, enforcing bootstrap checks
[INFO ][o.o.c.c.Coordinator ] [cluster-bootstrap-0] setting initial configuration to VotingConfiguration{zc21QCcJSoWFURx_cZ1Odg}
[INFO ][o.o.c.s.MasterService ] [cluster-bootstrap-0] Tasks batched with key: org.opensearch.cluster.coordination.JoinHelper and count: 3, term: 1, version: 68, delta: cluster-manager node changed {previous [], current [{cluster-bootstrap-0}{zc21QCcJSoWFURx_cZ1Odg}{O63iEVovT3ajz_cVeBatdA}{cluster-bootstrap-0}{10.42.0.18:9300}{m}{shard_indexing_pressure_enabled=true}]}
[INFO ][o.o.g.G.RemotePersistedState] [cluster-bootstrap-0] codec version is 4
[INFO ][o.o.g.r.RemoteManifestManager] [cluster-bootstrap-0] No manifest file present in remote store for cluster name: cluster, cluster UUID: 7GS3z_h9Tn2Waf-MnXoemQ
[ERROR][o.o.g.G.RemotePersistedState] [cluster-bootstrap-0] Latest manifest is not present in remote store for cluster UUID: 7GS3z_h9Tn2Waf-MnXoemQ
[INFO ][o.o.c.c.CoordinationState] [cluster-bootstrap-0] cluster UUID set to [7GS3z_h9Tn2Waf-MnXoemQ]
[INFO ][o.o.c.s.ClusterApplierService] [cluster-bootstrap-0] cluster-manager node changed {previous [], current [{cluster-bootstrap-0}{zc21QCcJSoWFURx_cZ1Odg}{O63iEVovT3ajz_cVeBatdA}{cluster-bootstrap-0}{10.42.0.18:9300}{m}{shard_indexing_pressure_enabled=true}]}, term: 1, version: 68, reason: Publication{term=1, version=68}
[INFO ][o.o.t.i.IndexManagement ] [cluster-bootstrap-0] Candidate custom result indices are empty.
[INFO ][o.o.t.i.IndexManagement ] [cluster-bootstrap-0] Candidate custom result indices are empty.
[INFO ][o.o.t.c.ClusterEventListener] [cluster-bootstrap-0] Cluster is not recovered yet.
[INFO ][o.o.i.i.ManagedIndexCoordinator] [cluster-bootstrap-0] Cache cluster manager node onClusterManager time: 1731968345520
[WARN ][o.o.p.c.s.h.ConfigOverridesClusterSettingHandler] [cluster-bootstrap-0] Config override setting update called with empty string. Ignoring.
[INFO ][o.o.d.PeerFinder ] [cluster-bootstrap-0] setting findPeersInterval to [1s] as node commission status = [true] for local node [{cluster-bootstrap-0}{zc21QCcJSoWFURx_cZ1Odg}{O63iEVovT3ajz_cVeBatdA}{cluster-bootstrap-0}{10.42.0.18:9300}{m}{shard_indexing_pressure_enabled=true}]
[WARN ][o.o.c.r.a.AllocationService] [cluster-bootstrap-0] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
[INFO ][o.o.h.AbstractHttpServerTransport] [cluster-bootstrap-0] publish_address {cluster-bootstrap-0/10.42.0.18:9200}, bound_addresses {[::]:9200}
[INFO ][o.o.n.Node ] [cluster-bootstrap-0] started
[INFO ][o.o.s.c.ConfigurationRepository] [cluster-bootstrap-0] Will not attempt to create index .opendistro_security and default configs if they are absent. Use securityadmin to initialize cluster
[INFO ][o.o.s.c.ConfigurationRepository] [cluster-bootstrap-0] Background init thread started. Install default config?: false
[INFO ][o.o.s.OpenSearchSecurityPlugin] [cluster-bootstrap-0] 0 OpenSearch Security modules loaded so far: []
[INFO ][o.o.s.c.ConfigurationRepository] [cluster-bootstrap-0] Wait for cluster to be available ...
[WARN ][o.o.c.r.a.AllocationService] [cluster-bootstrap-0] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
[INFO ][o.o.g.G.RemotePersistedState] [cluster-bootstrap-0] codec version is 4
[INFO ][o.o.c.s.ClusterSettings ] [cluster-bootstrap-0] updating [plugins.index_state_management.template_migration.control] from [0] to [-1]
[INFO ][o.o.t.c.HashRing ] [cluster-bootstrap-0] Node added: [zc21QCcJSoWFURx_cZ1Odg]
[INFO ][o.o.t.c.HashRing ] [cluster-bootstrap-0] All nodes with known version: {zc21QCcJSoWFURx_cZ1Odg=ADNodeInfo{version=2.17.0, isEligibleDataNode=false}}
[INFO ][o.o.t.c.HashRing ] [cluster-bootstrap-0] Rebuild hash ring for realtime with cooldown, nodeChangeEvents size 0
[INFO ][o.o.t.c.HashRing ] [cluster-bootstrap-0] Build version hash ring successfully
[INFO ][o.o.t.c.ClusterEventListener] [cluster-bootstrap-0] Init version hash ring successfully
[INFO ][o.o.g.GatewayService ] [cluster-bootstrap-0] recovered [7] indices into cluster_state
[INFO ][o.o.s.l.LogTypeService ] [cluster-bootstrap-0] Loading builtin types!
[WARN ][o.o.o.i.ObservabilityIndex] [cluster-bootstrap-0] message: index [.opensearch-observability/ktY-1R7KQ-iud-Eg-esoWQ] already exists
[INFO ][o.o.s.l.BuiltinLogTypeLoader] [cluster-bootstrap-0] Loaded [windows_logtype.json] log type
[INFO ][o.o.s.l.BuiltinLogTypeLoader] [cluster-bootstrap-0] Loaded [waf_logtype.json] log type
[INFO ][o.o.s.l.BuiltinLogTypeLoader] [cluster-bootstrap-0] Loaded [vpcflow_logtype.json] log type
[INFO ][o.o.s.l.BuiltinLogTypeLoader] [cluster-bootstrap-0] Loaded [test_windows_logtype.json] log type
[INFO ][o.o.s.l.BuiltinLogTypeLoader] [cluster-bootstrap-0] Loaded [s3_logtype.json] log type
[INFO ][o.o.s.l.BuiltinLogTypeLoader] [cluster-bootstrap-0] Loaded [others_web_logtype.json] log type
[INFO ][o.o.s.l.BuiltinLogTypeLoader] [cluster-bootstrap-0] Loaded [others_proxy_logtype.json] log type
[INFO ][o.o.s.l.BuiltinLogTypeLoader] [cluster-bootstrap-0] Loaded [others_macos_logtype.json] log type
[INFO ][o.o.s.l.BuiltinLogTypeLoader] [cluster-bootstrap-0] Loaded [others_compliance_logtype.json] log type
[INFO ][o.o.s.l.BuiltinLogTypeLoader] [cluster-bootstrap-0] Loaded [others_cloud_logtype.json] log type
[INFO ][o.o.s.l.BuiltinLogTypeLoader] [cluster-bootstrap-0] Loaded [others_apt_logtype.json] log type
[INFO ][o.o.s.l.BuiltinLogTypeLoader] [cluster-bootstrap-0] Loaded [others_application_logtype.json] log type
[INFO ][o.o.s.l.BuiltinLogTypeLoader] [cluster-bootstrap-0] Loaded [okta_logtype.json] log type
[INFO ][o.o.s.l.BuiltinLogTypeLoader] [cluster-bootstrap-0] Loaded [network_logtype.json] log type
[INFO ][o.o.s.l.BuiltinLogTypeLoader] [cluster-bootstrap-0] Loaded [netflow_logtype.json] log type
[INFO ][o.o.s.l.BuiltinLogTypeLoader] [cluster-bootstrap-0] Loaded [m365_logtype.json] log type
[INFO ][o.o.s.l.BuiltinLogTypeLoader] [cluster-bootstrap-0] Loaded [linux_logtype.json] log type
[INFO ][o.o.s.l.BuiltinLogTypeLoader] [cluster-bootstrap-0] Loaded [gworkspace_logtype.json] log type
[INFO ][o.o.s.l.BuiltinLogTypeLoader] [cluster-bootstrap-0] Loaded [github_logtype.json] log type
[INFO ][o.o.s.l.BuiltinLogTypeLoader] [cluster-bootstrap-0] Loaded [dns_logtype.json] log type
[INFO ][o.o.s.l.BuiltinLogTypeLoader] [cluster-bootstrap-0] Loaded [cloudtrail_logtype.json] log type
[INFO ][o.o.s.l.BuiltinLogTypeLoader] [cluster-bootstrap-0] Loaded [azure_logtype.json] log type
[INFO ][o.o.s.l.BuiltinLogTypeLoader] [cluster-bootstrap-0] Loaded [apache_access_logtype.json] log type
[INFO ][o.o.s.l.BuiltinLogTypeLoader] [cluster-bootstrap-0] Loaded [ad_ldap_logtype.json] log type
[INFO ][o.o.s.l.LogTypeService ] [cluster-bootstrap-0] Indexing [429] fieldMappingDocs from logTypes: 24
[WARN ][o.o.s.SecurityAnalyticsPlugin] [cluster-bootstrap-0] Failed to initialize LogType config index and builtin log types
[INFO ][o.o.s.l.LogTypeService ] [cluster-bootstrap-0] Loading builtin types!
[INFO ][o.o.s.l.LogTypeService ] [cluster-bootstrap-0] Indexing [429] fieldMappingDocs from logTypes: 24
[INFO ][o.o.s.l.LogTypeService ] [cluster-bootstrap-0] Loading builtin types!
[INFO ][o.o.s.l.LogTypeService ] [cluster-bootstrap-0] Indexing [429] fieldMappingDocs from logTypes: 24
[INFO ][o.o.s.i.DetectorIndexManagementService] [cluster-bootstrap-0] info deleteOldIndices
[INFO ][o.o.s.i.DetectorIndexManagementService] [cluster-bootstrap-0] No Old Correlation Indices to delete
[INFO ][o.o.s.i.DetectorIndexManagementService] [cluster-bootstrap-0] info deleteOldIndices
[INFO ][o.o.s.i.DetectorIndexManagementService] [cluster-bootstrap-0] No Old IOC Findings Indices to delete
[ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [cluster-bootstrap-0] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@7bf63828] retrieving configuration for [ACTIONGROUPS, ALLOWLIST, AUDIT, CONFIG, INTERNALUSERS, NODESDN, ROLES, ROLESMAPPING, TENANTS, WHITELIST] (index=.opendistro_security)
[ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [cluster-bootstrap-0] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@7bf63828] retrieving configuration for [ACTIONGROUPS, ALLOWLIST, AUDIT, CONFIG, INTERNALUSERS, NODESDN, ROLES, ROLESMAPPING, TENANTS, WHITELIST] (index=.opendistro_security)
...
(Timestamps removed.)