Hi!
I’m following the procedure to enable the Metrics Framework in a Docker-based deployment of a Single-node OpenSearch 2.19.2 cluster and noticed a null WARN exception (java.lang.ExceptionInInitializerError: null).
It begins at start-up of the node and then emits approximately once every 60 seconds:
test-cluster.log:
2025-05-23 15:57:15.158 | [2025-05-23T19:57:15,158][INFO ][o.o.p.PluginsService ] [osnode1] loaded plugin [query-insights]
2025-05-23 15:57:15.158 | [2025-05-23T19:57:15,158][INFO ][o.o.p.PluginsService ] [osnode1] loaded plugin [telemetry-otel]
2025-05-23 15:57:15.221 | [2025-05-23T19:57:15,220][WARN ][stderr ] [osnode1] WARNING: A restricted method in java.lang.foreign.Linker has been called
2025-05-23 15:57:15.221 | [2025-05-23T19:57:15,221][WARN ][stderr ] [osnode1] WARNING: java.lang.foreign.Linker::downcallHandle has been called by the unnamed module
2025-05-23 15:57:15.221 | [2025-05-23T19:57:15,221][WARN ][stderr ] [osnode1] WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning for this module
2025-05-23 15:57:15.251 | [2025-05-23T19:57:15,251][INFO ][o.a.l.s.MemorySegmentIndexInputProvider] [osnode1] Using MemorySegmentIndexInput and native madvise support with Java 21 or later; to disable start with -Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false
2025-05-23 15:57:15.255 | [2025-05-23T19:57:15,255][INFO ][o.o.e.NodeEnvironment ] [osnode1] using [1] data paths, mounts [[/usr/share/opensearch/data (/run/host_mark/Users)]], net usable_space [340.7gb], net total_space [926.3gb], types [fakeowner]
2025-05-23 15:57:15.255 | [2025-05-23T19:57:15,255][INFO ][o.o.e.NodeEnvironment ] [osnode1] heap size [1gb], compressed ordinary object pointers [true]
2025-05-23 15:57:15.403 | [2025-05-23T19:57:15,403][INFO ][o.o.n.Node ] [osnode1] node name [osnode1], node ID [MGWd3MpVQSaSTtQeS_7Wfg], cluster name [test-cluster], roles [ingest, remote_cluster_client, data, cluster_manager]
2025-05-23 15:57:15.427 | [2025-05-23T19:57:15,426][INFO ][o.o.e.ExtensionsManager ] [osnode1] ExtensionsManager initialized
2025-05-23 15:57:16.085 | [2025-05-23T19:57:16,085][DEPRECATION][o.o.d.c.s.Settings ] [osnode1] [index.store.hybrid.mmap.extensions] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
2025-05-23 15:57:16.086 | [2025-05-23T19:57:16,086][DEPRECATION][o.o.d.c.s.Settings ] [osnode1] [cluster.initial_master_nodes] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
2025-05-23 15:57:16.090 | [2025-05-23T19:57:16,090][INFO ][o.o.t.t.e.OTelSpanExporterFactory] [osnode1] Successfully instantiated the SpanExporter class class io.opentelemetry.exporter.logging.LoggingSpanExporter
2025-05-23 15:57:16.100 | [2025-05-23T19:57:16,100][WARN ][i.o.s.t.i.JcTools ] [osnode1] Cannot create high-performance queue, reverting to ArrayBlockingQueue (java.lang.ExceptionInInitializerError)
2025-05-23 15:57:16.180 | [2025-05-23T19:57:16,180][INFO ][o.o.t.m.e.OTelMetricsExporterFactory] [osnode1] Successfully instantiated the Metrics MetricExporter class class io.opentelemetry.exporter.otlp.metrics.OtlpGrpcMetricExporter
2025-05-23 15:57:16.465 | [2025-05-23T19:57:16,465][INFO ][o.o.n.p.NeuralSearch ] [osnode1] Registering hybrid query phase searcher with feature flag [plugins.neural_search.hybrid_search_disabled]
2025-05-23 15:57:16.670 | [2025-05-23T19:57:16,670][INFO ][o.o.l.b.LTRCircuitBreakerService] [osnode1] Registered memory breaker.
2025-05-23 15:57:16.675 | [2025-05-23T19:57:16,674][INFO ][o.o.r.m.c.i.SdkClientFactory] [osnode1] Using local opensearch cluster as metadata store.
2025-05-23 15:57:16.742 | [2025-05-23T19:57:16,742][INFO ][o.o.t.b.CircuitBreakerService] [osnode1] Registered memory breaker.
2025-05-23 15:57:16.952 | [2025-05-23T19:57:16,951][INFO ][o.o.r.m.c.i.SdkClientFactory] [osnode1] Using local opensearch cluster as metadata store.
2025-05-23 15:57:16.959 | [2025-05-23T19:57:16,959][INFO ][o.o.m.b.MLCircuitBreakerService] [osnode1] Registered ML memory breaker.
2025-05-23 15:57:16.960 | [2025-05-23T19:57:16,960][INFO ][o.o.m.b.MLCircuitBreakerService] [osnode1] Registered ML disk breaker.
2025-05-23 15:57:16.960 | [2025-05-23T19:57:16,960][INFO ][o.o.m.b.MLCircuitBreakerService] [osnode1] Registered ML native memory breaker.
2025-05-23 15:57:17.002 | [2025-05-23T19:57:17,002][INFO ][o.r.Reflections ] [osnode1] Reflections took 21 ms to scan 1 urls, producing 27 keys and 67 values
2025-05-23 15:57:17.013 | [2025-05-23T19:57:17,012][INFO ][o.r.Reflections ] [osnode1] Reflections took 0 ms to scan 1 urls, producing 3 keys and 5 values
2025-05-23 15:57:17.032 | [2025-05-23T19:57:17,032][WARN ][o.o.s.p.SQLPlugin ] [osnode1] Master key is a required config for using create and update datasource APIs. Please set plugins.query.datasources.encryption.masterkey config in opensearch.yml in all the cluster nodes. More details can be found here: https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/admin/datasources.rst#master-key-config-for-encrypting-credential-information
2025-05-23 15:57:17.147 | [2025-05-23T19:57:17,147][INFO ][o.o.p.i.c.s.QueryInsightsService] [osnode1] Setting query insights index template priority to [1847]
2025-05-23 15:57:17.372 | [2025-05-23T19:57:17,372][INFO ][o.o.t.NettyAllocator ] [osnode1] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={opensearch.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=1gb}]
2025-05-23 15:57:17.416 | [2025-05-23T19:57:17,416][INFO ][o.o.d.DiscoveryModule ] [osnode1] using discovery type [zen] and seed hosts providers [settings]
2025-05-23 15:57:17.670 | [2025-05-23T19:57:17,670][WARN ][o.o.g.DanglingIndicesState] [osnode1] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
2025-05-23 15:57:17.966 | [2025-05-23T19:57:17,966][INFO ][o.o.p.h.c.PerformanceAnalyzerConfigAction] [osnode1] PerformanceAnalyzer Enabled: false
2025-05-23 15:57:17.978 | [2025-05-23T19:57:17,978][INFO ][o.o.n.Node ] [osnode1] initialized
2025-05-23 15:57:17.978 | [2025-05-23T19:57:17,978][INFO ][o.o.n.Node ] [osnode1] starting ...
2025-05-23 15:57:17.995 | [2025-05-23T19:57:17,995][INFO ][o.o.s.l.BuiltinLogTypeLoader] [osnode1] Loaded [windows_logtype.json] log type
2025-05-23 15:57:17.995 | [2025-05-23T19:57:17,995][INFO ][o.o.s.l.BuiltinLogTypeLoader] [osnode1] Loaded [waf_logtype.json] log type
2025-05-23 15:57:17.995 | [2025-05-23T19:57:17,995][INFO ][o.o.s.l.BuiltinLogTypeLoader] [osnode1] Loaded [vpcflow_logtype.json] log type
2025-05-23 15:57:17.996 | [2025-05-23T19:57:17,995][INFO ][o.o.s.l.BuiltinLogTypeLoader] [osnode1] Loaded [test_windows_logtype.json] log type
2025-05-23 15:57:17.997 | [2025-05-23T19:57:17,996][INFO ][o.o.s.l.BuiltinLogTypeLoader] [osnode1] Loaded [s3_logtype.json] log type
2025-05-23 15:57:17.997 | [2025-05-23T19:57:17,997][INFO ][o.o.s.l.BuiltinLogTypeLoader] [osnode1] Loaded [others_web_logtype.json] log type
2025-05-23 15:57:17.998 | [2025-05-23T19:57:17,997][INFO ][o.o.s.l.BuiltinLogTypeLoader] [osnode1] Loaded [others_proxy_logtype.json] log type
2025-05-23 15:57:17.998 | [2025-05-23T19:57:17,998][INFO ][o.o.s.l.BuiltinLogTypeLoader] [osnode1] Loaded [others_macos_logtype.json] log type
2025-05-23 15:57:17.998 | [2025-05-23T19:57:17,998][INFO ][o.o.s.l.BuiltinLogTypeLoader] [osnode1] Loaded [others_compliance_logtype.json] log type
2025-05-23 15:57:17.998 | [2025-05-23T19:57:17,998][INFO ][o.o.s.l.BuiltinLogTypeLoader] [osnode1] Loaded [others_cloud_logtype.json] log type
2025-05-23 15:57:17.998 | [2025-05-23T19:57:17,998][INFO ][o.o.s.l.BuiltinLogTypeLoader] [osnode1] Loaded [others_apt_logtype.json] log type
2025-05-23 15:57:17.999 | [2025-05-23T19:57:17,998][INFO ][o.o.s.l.BuiltinLogTypeLoader] [osnode1] Loaded [others_application_logtype.json] log type
2025-05-23 15:57:17.999 | [2025-05-23T19:57:17,999][INFO ][o.o.s.l.BuiltinLogTypeLoader] [osnode1] Loaded [okta_logtype.json] log type
2025-05-23 15:57:17.999 | [2025-05-23T19:57:17,999][INFO ][o.o.s.l.BuiltinLogTypeLoader] [osnode1] Loaded [network_logtype.json] log type
2025-05-23 15:57:17.999 | [2025-05-23T19:57:17,999][INFO ][o.o.s.l.BuiltinLogTypeLoader] [osnode1] Loaded [netflow_logtype.json] log type
2025-05-23 15:57:17.999 | [2025-05-23T19:57:17,999][INFO ][o.o.s.l.BuiltinLogTypeLoader] [osnode1] Loaded [m365_logtype.json] log type
2025-05-23 15:57:17.999 | [2025-05-23T19:57:17,999][INFO ][o.o.s.l.BuiltinLogTypeLoader] [osnode1] Loaded [linux_logtype.json] log type
2025-05-23 15:57:18.000 | [2025-05-23T19:57:17,999][INFO ][o.o.s.l.BuiltinLogTypeLoader] [osnode1] Loaded [gworkspace_logtype.json] log type
2025-05-23 15:57:18.000 | [2025-05-23T19:57:18,000][INFO ][o.o.s.l.BuiltinLogTypeLoader] [osnode1] Loaded [github_logtype.json] log type
2025-05-23 15:57:18.000 | [2025-05-23T19:57:18,000][INFO ][o.o.s.l.BuiltinLogTypeLoader] [osnode1] Loaded [dns_logtype.json] log type
2025-05-23 15:57:18.000 | [2025-05-23T19:57:18,000][INFO ][o.o.s.l.BuiltinLogTypeLoader] [osnode1] Loaded [cloudtrail_logtype.json] log type
2025-05-23 15:57:18.000 | [2025-05-23T19:57:18,000][INFO ][o.o.s.l.BuiltinLogTypeLoader] [osnode1] Loaded [azure_logtype.json] log type
2025-05-23 15:57:18.000 | [2025-05-23T19:57:18,000][INFO ][o.o.s.l.BuiltinLogTypeLoader] [osnode1] Loaded [apache_access_logtype.json] log type
2025-05-23 15:57:18.001 | [2025-05-23T19:57:18,001][INFO ][o.o.s.l.BuiltinLogTypeLoader] [osnode1] Loaded [ad_ldap_logtype.json] log type
2025-05-23 15:57:18.065 | [2025-05-23T19:57:18,065][INFO ][o.o.t.TransportService ] [osnode1] publish_address {172.18.0.2:9300}, bound_addresses {[::]:9300}
2025-05-23 15:57:18.066 | [2025-05-23T19:57:18,065][INFO ][o.o.t.TransportService ] [osnode1] Remote clusters initialized successfully.
2025-05-23 15:57:18.120 | [2025-05-23T19:57:18,119][DEPRECATION][o.o.d.c.s.Settings ] [osnode1] [opendistro.index_state_management.history.number_of_replicas] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
2025-05-23 15:57:18.227 | [2025-05-23T19:57:18,227][INFO ][o.o.b.BootstrapChecks ] [osnode1] bound or publishing to a non-loopback address, enforcing bootstrap checks
2025-05-23 15:57:18.230 | [2025-05-23T19:57:18,230][INFO ][o.o.c.c.Coordinator ] [osnode1] cluster UUID [QojmlEMNRf2tiZJqZaiVcg]
2025-05-23 15:57:18.294 | [2025-05-23T19:57:18,293][INFO ][o.o.c.s.MasterService ] [osnode1] Tasks batched with key: org.opensearch.cluster.coordination.JoinHelper, count:3 and sample tasks: elected-as-cluster-manager ([1] nodes joined)[{osnode1}{MGWd3MpVQSaSTtQeS_7Wfg}{nUOZzDwZSZ2oYWxqgCr3Sg}{172.18.0.2}{172.18.0.2:9300}{dimr}{shard_indexing_pressure_enabled=true} elect leader, _BECOME_CLUSTER_MANAGER_TASK_, _FINISH_ELECTION_], term: 25, version: 457, delta: cluster-manager node changed {previous [], current [{osnode1}{MGWd3MpVQSaSTtQeS_7Wfg}{nUOZzDwZSZ2oYWxqgCr3Sg}{172.18.0.2}{172.18.0.2:9300}{dimr}{shard_indexing_pressure_enabled=true}]}
2025-05-23 15:57:18.359 | [2025-05-23T19:57:18,359][INFO ][o.o.c.s.ClusterApplierService] [osnode1] cluster-manager node changed {previous [], current [{osnode1}{MGWd3MpVQSaSTtQeS_7Wfg}{nUOZzDwZSZ2oYWxqgCr3Sg}{172.18.0.2}{172.18.0.2:9300}{dimr}{shard_indexing_pressure_enabled=true}]}, term: 25, version: 457, reason: Publication{term=25, version=457}
2025-05-23 15:57:18.363 | [2025-05-23T19:57:18,363][INFO ][o.o.t.i.IndexManagement ] [osnode1] Candidate custom result indices are empty.
2025-05-23 15:57:18.364 | [2025-05-23T19:57:18,363][INFO ][o.o.t.i.IndexManagement ] [osnode1] Candidate custom result indices are empty.
2025-05-23 15:57:18.364 | [2025-05-23T19:57:18,363][INFO ][o.o.t.c.ClusterEventListener] [osnode1] Cluster is not recovered yet.
2025-05-23 15:57:18.365 | [2025-05-23T19:57:18,365][INFO ][o.o.i.i.ManagedIndexCoordinator] [osnode1] Cache cluster manager node onClusterManager time: 1748030238365
2025-05-23 15:57:18.382 | [2025-05-23T19:57:18,381][WARN ][o.o.p.c.s.h.ConfigOverridesClusterSettingHandler] [osnode1] Config override setting update called with empty string. Ignoring.
2025-05-23 15:57:18.386 | [2025-05-23T19:57:18,385][INFO ][o.o.d.PeerFinder ] [osnode1] setting findPeersInterval to [1s] as node commission status = [true] for local node [{osnode1}{MGWd3MpVQSaSTtQeS_7Wfg}{nUOZzDwZSZ2oYWxqgCr3Sg}{172.18.0.2}{172.18.0.2:9300}{dimr}{shard_indexing_pressure_enabled=true}]
2025-05-23 15:57:18.387 | [2025-05-23T19:57:18,387][WARN ][o.o.c.r.a.AllocationService] [osnode1] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
2025-05-23 15:57:18.388 | [2025-05-23T19:57:18,388][INFO ][o.o.h.AbstractHttpServerTransport] [osnode1] publish_address {172.18.0.2:9200}, bound_addresses {[::]:9200}
2025-05-23 15:57:18.388 | [2025-05-23T19:57:18,388][INFO ][o.o.n.Node ] [osnode1] started
2025-05-23 15:57:18.389 | [2025-05-23T19:57:18,389][INFO ][o.o.s.OpenSearchSecurityPlugin] [osnode1] 0 OpenSearch Security modules loaded so far: []
2025-05-23 15:57:18.395 | [2025-05-23T19:57:18,395][WARN ][o.o.c.r.a.AllocationService] [osnode1] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
2025-05-23 15:57:18.455 | [2025-05-23T19:57:18,455][INFO ][o.o.c.s.ClusterSettings ] [osnode1] updating [plugins.index_state_management.history.number_of_replicas] from [1] to [0]
2025-05-23 15:57:18.456 | [2025-05-23T19:57:18,455][INFO ][o.o.c.s.ClusterSettings ] [osnode1] updating [plugins.index_state_management.history.number_of_replicas] from [1] to [0]
2025-05-23 15:57:18.456 | [2025-05-23T19:57:18,455][INFO ][o.o.c.s.ClusterSettings ] [osnode1] updating [plugins.index_state_management.metadata_migration.status] from [0] to [1]
2025-05-23 15:57:18.456 | [2025-05-23T19:57:18,455][INFO ][o.o.i.i.ManagedIndexCoordinator] [osnode1] Canceling metadata moving job because of cluster setting update.
2025-05-23 15:57:18.456 | [2025-05-23T19:57:18,455][INFO ][o.o.c.s.ClusterSettings ] [osnode1] updating [plugins.index_state_management.template_migration.control] from [0] to [-1]
2025-05-23 15:57:18.459 | [2025-05-23T19:57:18,459][INFO ][o.o.t.c.HashRing ] [osnode1] Node added: [MGWd3MpVQSaSTtQeS_7Wfg]
2025-05-23 15:57:18.462 | [2025-05-23T19:57:18,462][INFO ][o.o.t.c.HashRing ] [osnode1] Add data node to version hash ring: MGWd3MpVQSaSTtQeS_7Wfg
2025-05-23 15:57:18.463 | [2025-05-23T19:57:18,462][INFO ][o.o.t.c.HashRing ] [osnode1] All nodes with known version: {MGWd3MpVQSaSTtQeS_7Wfg=ADNodeInfo{version=2.19.2, isEligibleDataNode=true}}
2025-05-23 15:57:18.463 | [2025-05-23T19:57:18,463][INFO ][o.o.t.c.HashRing ] [osnode1] Rebuild hash ring for realtime with cooldown, nodeChangeEvents size 0
2025-05-23 15:57:18.463 | [2025-05-23T19:57:18,463][INFO ][o.o.t.c.HashRing ] [osnode1] Build version hash ring successfully
2025-05-23 15:57:18.463 | [2025-05-23T19:57:18,463][INFO ][o.o.t.c.ADDataMigrator ] [osnode1] Start migrating AD data
2025-05-23 15:57:18.463 | [2025-05-23T19:57:18,463][INFO ][o.o.t.c.ADDataMigrator ] [osnode1] AD job index doesn't exist, no need to migrate
2025-05-23 15:57:18.463 | [2025-05-23T19:57:18,463][INFO ][o.o.t.c.ClusterEventListener] [osnode1] Init version hash ring successfully
2025-05-23 15:57:18.473 | [2025-05-23T19:57:18,473][INFO ][o.o.g.GatewayService ] [osnode1] recovered [17] indices into cluster_state
2025-05-23 15:57:18.475 | [2025-05-23T19:57:18,474][WARN ][o.o.o.i.ObservabilityIndex] [osnode1] message: index [.opensearch-observability/PItWLtKdTEKZdUrP0-EWIg] already exists
2025-05-23 15:57:18.475 | [2025-05-23T19:57:18,474][INFO ][o.o.s.l.LogTypeService ] [osnode1] Loading builtin types!
2025-05-23 15:57:18.477 | [2025-05-23T19:57:18,476][INFO ][o.o.s.l.LogTypeService ] [osnode1] Indexing [432] fieldMappingDocs from logTypes: 24
2025-05-23 15:57:18.490 | [2025-05-23T19:57:18,490][WARN ][o.o.c.r.a.AllocationService] [osnode1] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
2025-05-23 15:57:18.494 | [2025-05-23T19:57:18,493][WARN ][o.o.s.SecurityAnalyticsPlugin] [osnode1] Failed to initialize LogType config index and builtin log types
2025-05-23 15:57:18.494 | [2025-05-23T19:57:18,494][ERROR][o.o.i.i.ManagedIndexCoordinator] [osnode1] Failed to get ISM policies with templates: Failed to execute phase [query], all shards failed
2025-05-23 15:57:18.567 | [2025-05-23T19:57:18,567][INFO ][o.o.p.PluginsService ] [osnode1] PluginService:onIndexModule index:[.opendistro-job-scheduler-lock/sLbqJ5BtTs6iV-_AzkZ-iw]
2025-05-23 15:57:18.570 | [2025-05-23T19:57:18,569][WARN ][i.o.s.m.i.s.MetricStorageRegistry] [osnode1] Found duplicate metric definition: cpu_utilization
2025-05-23 15:57:18.570 | at unknown source
2025-05-23 15:57:18.570 | To enable better debugging, run your JVM with -Dotel.experimental.sdk.metrics.debug=true
2025-05-23 15:57:18.570 | Causes
2025-05-23 15:57:18.570 | - Description [CPU Utilization per shard for a search phase] does not match [CPU Utilization per shard for an operation]
2025-05-23 15:57:18.570 | - InstrumentDescription [CPU Utilization per shard for a search phase] does not match [CPU Utilization per shard for an operation]
2025-05-23 15:57:18.570 | Original instrument registered with same name but is incompatible.
2025-05-23 15:57:18.570 | at unknown source
2025-05-23 15:57:18.570 | To enable better debugging, run your JVM with -Dotel.experimental.sdk.metrics.debug=true
2025-05-23 15:57:18.570 |
2025-05-23 15:57:18.623 | [2025-05-23T19:57:18,623][INFO ][o.o.p.PluginsService ] [osnode1] PluginService:onIndexModule index:[.opendistro-ism-config/GM5zNX8JSNadyBHpFrMgRw]
2025-05-23 15:57:18.623 | [2025-05-23T19:57:18,623][INFO ][o.o.j.JobSchedulerPlugin ] [osnode1] JobSweeper started listening to operations on index .opendistro-ism-config
2025-05-23 15:57:18.646 | [2025-05-23T19:57:18,646][INFO ][o.o.p.PluginsService ] [osnode1] PluginService:onIndexModule index:[.plugins-ml-config/IM8Jjx32TFWmQXKiboOJlA]
2025-05-23 15:57:18.652 | [2025-05-23T19:57:18,652][INFO ][o.o.p.PluginsService ] [osnode1] PluginService:onIndexModule index:[.opensearch-observability/PItWLtKdTEKZdUrP0-EWIg]
2025-05-23 15:57:19.094 | [2025-05-23T19:57:19,093][WARN ][o.o.c.r.a.AllocationService] [osnode1] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
2025-05-23 15:57:19.114 | [2025-05-23T19:57:19,113][DEPRECATION][o.o.d.i.m.IdFieldMapper ] [osnode1] Loading the fielddata on the _id field is deprecated and will be removed in future versions. If you require sorting or aggregating on this field you should also include the id in the body of your documents, and map this field as a keyword field that has [doc_values] enabled
2025-05-23 15:57:19.155 | [2025-05-23T19:57:19,154][INFO ][o.o.j.s.JobScheduler ] [osnode1] Scheduling job id MvfO9MViRAGRYhLhazXG-Q for index .opendistro-ism-config .
2025-05-23 15:57:19.155 | [2025-05-23T19:57:19,155][INFO ][o.o.j.s.JobScheduler ] [osnode1] Will delay 163734 miliseconds for next execution of job .ds-gl-security-investigations-metrics-000001
2025-05-23 15:57:19.155 | [2025-05-23T19:57:19,155][INFO ][o.o.j.s.JobScheduler ] [osnode1] Scheduling job id gW1PMSIjQGCblJ7qvBqZYQ for index .opendistro-ism-config .
2025-05-23 15:57:19.156 | [2025-05-23T19:57:19,155][INFO ][o.o.j.s.JobScheduler ] [osnode1] Will delay 93433 miliseconds for next execution of job .ds-gl-security-events-metrics-000001
2025-05-23 15:57:19.338 | [2025-05-23T19:57:19,338][INFO ][o.o.p.PluginsService ] [osnode1] PluginService:onIndexModule index:[top_queries-2025.05.23-81277/6UpZAfSLReOrq3CfBYENmg]
2025-05-23 15:57:19.344 | [2025-05-23T19:57:19,344][INFO ][o.o.p.PluginsService ] [osnode1] PluginService:onIndexModule index:[gl-system-events_0/kFnU7d0iSfuwSVbITe7QzQ]
2025-05-23 15:57:19.349 | [2025-05-23T19:57:19,349][INFO ][o.o.p.PluginsService ] [osnode1] PluginService:onIndexModule index:[.opendistro-ism-managed-index-history-2025.05.23-1/PjCH-okkSpW130bOPKb8jg]
2025-05-23 15:57:19.353 | [2025-05-23T19:57:19,352][INFO ][o.o.p.PluginsService ] [osnode1] PluginService:onIndexModule index:[gl_sonicwall_0/lonOZMxrT7Cth03wbaq6JA]
2025-05-23 15:57:19.384 | [2025-05-23T19:57:19,384][INFO ][o.o.s.l.LogTypeService ] [osnode1] Loading builtin types!
2025-05-23 15:57:19.385 | [2025-05-23T19:57:19,385][INFO ][o.o.s.l.LogTypeService ] [osnode1] Indexing [432] fieldMappingDocs from logTypes: 24
2025-05-23 15:57:19.387 | [2025-05-23T19:57:19,387][INFO ][o.o.s.l.LogTypeService ] [osnode1] Loading builtin types!
2025-05-23 15:57:19.387 | [2025-05-23T19:57:19,387][INFO ][o.o.s.l.LogTypeService ] [osnode1] Indexing [432] fieldMappingDocs from logTypes: 24
2025-05-23 15:57:19.389 | [2025-05-23T19:57:19,389][INFO ][o.o.s.i.DetectorIndexManagementService] [osnode1] info deleteOldIndices
2025-05-23 15:57:19.391 | [2025-05-23T19:57:19,391][INFO ][o.o.s.i.DetectorIndexManagementService] [osnode1] No Old Correlation Indices to delete
2025-05-23 15:57:19.392 | [2025-05-23T19:57:19,392][INFO ][o.o.s.i.DetectorIndexManagementService] [osnode1] info deleteOldIndices
2025-05-23 15:57:19.392 | [2025-05-23T19:57:19,392][INFO ][o.o.s.i.DetectorIndexManagementService] [osnode1] No Old IOC Findings Indices to delete
2025-05-23 15:57:20.114 | [2025-05-23T19:57:20,114][WARN ][o.o.c.r.a.AllocationService] [osnode1] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
2025-05-23 15:57:20.144 | [2025-05-23T19:57:20,144][DEPRECATION][o.o.d.c.m.IndexNameExpressionResolver] [osnode1] this request accesses system indices: [.opendistro-ism-config], but in a future major version, direct access to system indices will be prevented by default
2025-05-23 15:57:20.144 | [2025-05-23T19:57:20,144][DEPRECATION][o.o.d.c.m.IndexNameExpressionResolver] [osnode1] this request accesses system indices: [.opendistro-job-scheduler-lock], but in a future major version, direct access to system indices will be prevented by default
2025-05-23 15:57:20.144 | [2025-05-23T19:57:20,144][DEPRECATION][o.o.d.c.m.IndexNameExpressionResolver] [osnode1] this request accesses system indices: [.opensearch-observability], but in a future major version, direct access to system indices will be prevented by default
2025-05-23 15:57:20.144 | [2025-05-23T19:57:20,144][DEPRECATION][o.o.d.c.m.IndexNameExpressionResolver] [osnode1] this request accesses system indices: [.plugins-ml-config], but in a future major version, direct access to system indices will be prevented by default
2025-05-23 15:57:20.214 | [2025-05-23T19:57:20,213][INFO ][o.o.p.PluginsService ] [osnode1] PluginService:onIndexModule index:[investigation_message_index_0/l3KzqYPaR4ac62bsQgXe2w]
2025-05-23 15:57:20.216 | [2025-05-23T19:57:20,216][INFO ][o.o.p.PluginsService ] [osnode1] PluginService:onIndexModule index:[gl-events_0/0dtpk8wNRWSX-IIoQiOTgg]
2025-05-23 15:57:20.220 | [2025-05-23T19:57:20,220][INFO ][o.o.p.PluginsService ] [osnode1] PluginService:onIndexModule index:[gl-failures_0/_2rO0PZeQk-MKJ7I-sfDqQ]
2025-05-23 15:57:20.223 | [2025-05-23T19:57:20,223][INFO ][o.o.p.PluginsService ] [osnode1] PluginService:onIndexModule index:[investigation_event_index_0/IBUBXc2PSiWz5pr7gCVreg]
2025-05-23 15:57:20.709 | [2025-05-23T19:57:20,709][WARN ][o.o.c.r.a.AllocationService] [osnode1] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
2025-05-23 15:57:20.776 | [2025-05-23T19:57:20,776][INFO ][o.o.p.PluginsService ] [osnode1] PluginService:onIndexModule index:[gl_apache2_0/Bz_iqxe7T9i-eedW5HtbLg]
2025-05-23 15:57:20.782 | [2025-05-23T19:57:20,782][INFO ][o.o.p.PluginsService ] [osnode1] PluginService:onIndexModule index:[graylog_0/Cpo28QyJT0y0noIXOgOhsA]
2025-05-23 15:57:20.786 | [2025-05-23T19:57:20,786][INFO ][o.o.p.PluginsService ] [osnode1] PluginService:onIndexModule index:[.ds-gl-security-investigations-metrics-000001/MvfO9MViRAGRYhLhazXG-Q]
2025-05-23 15:57:20.788 | [2025-05-23T19:57:20,788][INFO ][o.o.p.PluginsService ] [osnode1] PluginService:onIndexModule index:[.ds-gl-security-events-metrics-000001/gW1PMSIjQGCblJ7qvBqZYQ]
2025-05-23 15:57:21.033 | [2025-05-23T19:57:21,033][WARN ][o.o.c.r.a.AllocationService] [osnode1] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
2025-05-23 15:57:21.102 | [2025-05-23T19:57:21,102][INFO ][o.o.p.PluginsService ] [osnode1] PluginService:onIndexModule index:[.opensearch-sap-log-types-config/QfuYCcIlT1CPUs848khD8Q]
2025-05-23 15:57:21.140 | [2025-05-23T19:57:21,140][INFO ][o.o.c.r.a.AllocationService] [osnode1] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.opensearch-sap-log-types-config][0]]]).
2025-05-23 15:57:21.308 | [2025-05-23T19:57:21,308][WARN ][o.o.c.r.a.AllocationService] [osnode1] Falling back to single shard assignment since batch mode disable or multiple custom allocators set
2025-05-23 15:57:28.387 | [2025-05-23T19:57:28,386][INFO ][o.o.m.a.MLModelAutoReDeployer] [osnode1] Index not found, not performing auto reloading!
2025-05-23 15:57:28.387 | [2025-05-23T19:57:28,387][INFO ][o.o.m.c.MLCommonsClusterManagerEventListener] [osnode1] Starting ML sync up job...
2025-05-23 15:57:38.414 | [2025-05-23T19:57:38,414][INFO ][o.o.m.c.MLSyncUpCron ] [osnode1] ML configuration already initialized, no action needed
2025-05-23 15:58:16.218 | [2025-05-23T19:58:16,213][WARN ][i.o.s.m.e.PeriodicMetricReader] [osnode1] Exporter threw an Exception
2025-05-23 15:58:16.218 | java.lang.ExceptionInInitializerError: null
2025-05-23 15:58:16.218 | at io.opentelemetry.exporter.internal.marshal.UnsafeString.getStringFieldOffset(UnsafeString.java:35) ~[opentelemetry-exporter-common-1.46.0.jar:1.46.0]
2025-05-23 15:58:16.218 | at io.opentelemetry.exporter.internal.marshal.UnsafeString.<clinit>(UnsafeString.java:11) ~[opentelemetry-exporter-common-1.46.0.jar:1.46.0]
2025-05-23 15:58:16.218 | at io.opentelemetry.exporter.internal.marshal.StatelessMarshalerUtil.getUtf8Size(StatelessMarshalerUtil.java:308) ~[opentelemetry-exporter-common-1.46.0.jar:1.46.0]
2025-05-23 15:58:16.218 | at io.opentelemetry.exporter.internal.marshal.StatelessMarshalerUtil.getUtf8Size(StatelessMarshalerUtil.java:303) ~[opentelemetry-exporter-common-1.46.0.jar:1.46.0]
2025-05-23 15:58:16.218 | at io.opentelemetry.exporter.internal.marshal.StatelessMarshalerUtil.sizeStringWithContext(StatelessMarshalerUtil.java:98) ~[opentelemetry-exporter-common-1.46.0.jar:1.46.0]
2025-05-23 15:58:16.218 | at io.opentelemetry.exporter.internal.otlp.metrics.MetricStatelessMarshaler.getBinarySerializedSize(MetricStatelessMarshaler.java:202) ~[opentelemetry-exporter-otlp-common-1.46.0.jar:1.46.0]
2025-05-23 15:58:16.218 | at io.opentelemetry.exporter.internal.otlp.metrics.MetricStatelessMarshaler.getBinarySerializedSize(MetricStatelessMarshaler.java:28) ~[opentelemetry-exporter-otlp-common-1.46.0.jar:1.46.0]
2025-05-23 15:58:16.218 | at io.opentelemetry.exporter.internal.marshal.StatelessMarshalerUtil.sizeRepeatedMessageWithContext(StatelessMarshalerUtil.java:135) ~[opentelemetry-exporter-common-1.46.0.jar:1.46.0]
2025-05-23 15:58:16.218 | at io.opentelemetry.exporter.internal.otlp.metrics.InstrumentationScopeMetricsStatelessMarshaler.getBinarySerializedSize(InstrumentationScopeMetricsStatelessMarshaler.java:58) ~[opentelemetry-exporter-otlp-common-1.46.0.jar:1.46.0]
2025-05-23 15:58:16.218 | at io.opentelemetry.exporter.internal.otlp.metrics.InstrumentationScopeMetricsStatelessMarshaler.getBinarySerializedSize(InstrumentationScopeMetricsStatelessMarshaler.java:22) ~[opentelemetry-exporter-otlp-common-1.46.0.jar:1.46.0]
2025-05-23 15:58:16.218 | at io.opentelemetry.exporter.internal.marshal.StatelessMarshalerUtil$RepeatedElementPairSizeCalculator.accept(StatelessMarshalerUtil.java:263) ~[opentelemetry-exporter-common-1.46.0.jar:1.46.0]
2025-05-23 15:58:16.218 | at java.base/java.util.IdentityHashMap.forEach(IdentityHashMap.java:1375) ~[?:?]
2025-05-23 15:58:16.218 | at io.opentelemetry.exporter.internal.marshal.StatelessMarshalerUtil.sizeRepeatedMessageWithContext(StatelessMarshalerUtil.java:188) ~[opentelemetry-exporter-common-1.46.0.jar:1.46.0]
2025-05-23 15:58:16.218 | at io.opentelemetry.exporter.internal.otlp.metrics.ResourceMetricsStatelessMarshaler.getBinarySerializedSize(ResourceMetricsStatelessMarshaler.java:70) ~[opentelemetry-exporter-otlp-common-1.46.0.jar:1.46.0]
2025-05-23 15:58:16.218 | at io.opentelemetry.exporter.internal.otlp.metrics.ResourceMetricsStatelessMarshaler.getBinarySerializedSize(ResourceMetricsStatelessMarshaler.java:28) ~[opentelemetry-exporter-otlp-common-1.46.0.jar:1.46.0]
2025-05-23 15:58:16.218 | at io.opentelemetry.exporter.internal.marshal.StatelessMarshalerUtil$RepeatedElementPairSizeCalculator.accept(StatelessMarshalerUtil.java:263) ~[opentelemetry-exporter-common-1.46.0.jar:1.46.0]
2025-05-23 15:58:16.218 | at java.base/java.util.IdentityHashMap.forEach(IdentityHashMap.java:1375) ~[?:?]
2025-05-23 15:58:16.218 | at io.opentelemetry.exporter.internal.marshal.StatelessMarshalerUtil.sizeRepeatedMessageWithContext(StatelessMarshalerUtil.java:188) ~[opentelemetry-exporter-common-1.46.0.jar:1.46.0]
2025-05-23 15:58:16.218 | at io.opentelemetry.exporter.internal.otlp.metrics.LowAllocationMetricsRequestMarshaler.calculateSize(LowAllocationMetricsRequestMarshaler.java:84) ~[opentelemetry-exporter-otlp-common-1.46.0.jar:1.46.0]
2025-05-23 15:58:16.218 | at io.opentelemetry.exporter.internal.otlp.metrics.LowAllocationMetricsRequestMarshaler.initialize(LowAllocationMetricsRequestMarshaler.java:57) ~[opentelemetry-exporter-otlp-common-1.46.0.jar:1.46.0]
2025-05-23 15:58:16.218 | at io.opentelemetry.exporter.internal.otlp.metrics.MetricReusableDataMarshaler.export(MetricReusableDataMarshaler.java:45) ~[opentelemetry-exporter-otlp-common-1.46.0.jar:1.46.0]
2025-05-23 15:58:16.218 | at io.opentelemetry.exporter.otlp.metrics.OtlpGrpcMetricExporter.export(OtlpGrpcMetricExporter.java:107) ~[opentelemetry-exporter-otlp-1.46.0.jar:1.46.0]
2025-05-23 15:58:16.218 | at io.opentelemetry.sdk.metrics.export.PeriodicMetricReader$Scheduled.doRun(PeriodicMetricReader.java:167) [opentelemetry-sdk-metrics-1.46.0.jar:1.46.0]
2025-05-23 15:58:16.218 | at io.opentelemetry.sdk.metrics.export.PeriodicMetricReader$Scheduled.run(PeriodicMetricReader.java:153) [opentelemetry-sdk-metrics-1.46.0.jar:1.46.0]
2025-05-23 15:58:16.218 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) [?:?]
2025-05-23 15:58:16.218 | at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358) [?:?]
2025-05-23 15:58:16.218 | at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?]
2025-05-23 15:58:16.218 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
2025-05-23 15:58:16.218 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
2025-05-23 15:58:16.218 | at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
2025-05-23 15:58:16.218 | Caused by: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "accessClassInPackage.sun.misc")
2025-05-23 15:58:16.218 | at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:488) ~[?:?]
2025-05-23 15:58:16.218 | at java.base/java.security.AccessController.checkPermission(AccessController.java:1071) ~[?:?]
2025-05-23 15:58:16.218 | at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:411) ~[?:?]
2025-05-23 15:58:16.218 | at java.base/java.lang.SecurityManager.checkPackageAccess(SecurityManager.java:1371) ~[?:?]
2025-05-23 15:58:16.218 | at java.base/java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:869) ~[?:?]
2025-05-23 15:58:16.218 | at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526) ~[?:?]
2025-05-23 15:58:16.218 | at java.base/java.lang.Class.forName0(Native Method) ~[?:?]
2025-05-23 15:58:16.218 | at java.base/java.lang.Class.forName(Class.java:534) ~[?:?]
2025-05-23 15:58:16.218 | at java.base/java.lang.Class.forName(Class.java:513) ~[?:?]
2025-05-23 15:58:16.218 | at io.opentelemetry.exporter.internal.marshal.UnsafeAccess.checkUnsafe(UnsafeAccess.java:32) ~[?:?]
2025-05-23 15:58:16.218 | at io.opentelemetry.exporter.internal.marshal.UnsafeAccess.<clinit>(UnsafeAccess.java:14) ~[?:?]
2025-05-23 15:58:16.218 | ... 30 more
Here is the opensearch.yml file in-use:
cluster.name: test-cluster
# Bind to all interfaces because we don't know what IP address Docker will assign to us.
network.host: 0.0.0.0
# Setting network.host to a non-loopback address enables the annoying bootstrap checks. "Single-node" mode disables them again.
# discovery.type: single-node
discovery.seed_hosts: osnode1
opensearch.experimental.feature.telemetry.enabled: true
telemetry.feature.metrics.enabled: true
# telemetry.feature.tracer.enabled: true
# telemetry.tracer.enabled: true
# search.query.metrics.enabled: true
telemetry.otel.metrics.exporter.class: io.opentelemetry.exporter.otlp.metrics.OtlpGrpcMetricExporter
# telemetry.otel.tracer.span.exporter.class: io.opentelemetry.exporter.otlp.trace.OtlpGrpcSpanExporter
There does not appear to be a TCP Socket listener on Port 4317, but it-is not certain whether the WARN exception is related:
~/docker > docker exec -it osnode1 /bin/bash
[opensearch@osnode1 ~]$ curl "http://localhost:4317"
curl: (7) Failed to connect to localhost port 4317 after 0 ms: Couldn't connect to server
[opensearch@osnode1 ~]$