Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Opensearch v2.5.0
Server OS: Redhat linux 8.6
Describe the issue:
Trying to use performance analyzer, in a 3 node cluster ( 2 data 1 master ),
Opensearch installed with RPM,
performance-analyzer service runs only on 1 data node.
Try to run metric query with curl:
curl -k --user admin:admin -XGET ‘https://hostipxxxx:9600/_plugins/_performanceanalyzer/metrics/units’
curl -k --user admin:admin -XGET ‘http://hostipxxxx:9600/_plugins/_performanceanalyzer/metrics/units’
getting: curl: (52) Empty reply from server
Dev-tools:
“error”: “no handler found for uri [/hostipxxxx:9600/_plugins/_performanceanalyzer/rca] and method [GET]”
hostipxxxx = data node ip address
Any idea what could be wrong?
Configuration:
======================== OpenSearch performance analyzer plugin config =========================
NOTE: this is an example for Linux. Please modify the config accordingly if you are using it under other OS.
Metrics data location
metrics-location = /dev/shm/performanceanalyzer/
Metrics deletion interval (minutes) for metrics data.
Interval should be between 1 to 60.
metrics-deletion-interval = 1
If set to true, the system cleans up the files behind it. So at any point, we should expect only 2
metrics-db-file-prefix-path files. If set to false, no files are cleaned up. This can be useful, if you are archiving
the files and wouldn’t like for them to be cleaned up.
cleanup-metrics-db-files = true
WebService exposed by App’s port
webservice-listener-port = 9600
Port for RPC Communication
rpc-port = 9650
Metric DB File Prefix Path location
metrics-db-file-prefix-path = /data/opensearch/metricsdb/metricsdb_
https-enabled = true
Setup the correct path for server certificates
certificate-file-path = /etc/opensearch/esnode.pem
private-key-file-path = /etc/opensearch/esnode-key.pem
trusted-cas-file-path = /etc/opensearch/root-ca.pem
Setup the correct path for client certificates (by default, the client will just use the server certificates)
client-certificate-file-path = specify_path
client-private-key-file-path = specify_path
client-trusted-cas-file-path = specify_path
WebService bind host; default only to local interface
webservice-bind-host = hostipxxxx
Plugin Stats Metadata file name, expected to be in the same location
plugin-stats-metadata = plugin-stats-metadata
Agent Stats Metadata file name, expected to be in the same location
agent-stats-metadata = agent-stats-metadata
Relevant Logs or Screenshots:
ERROR org.opensearch.performanceanalyzer.rca.store.metric.AggregateMetric - RCA: Caught an exception while getting the DB SQL [select sum(max) SUM_of_max from Shard_Size_In_Bytes]; [SQLITE_ERROR] SQL error or missing database (no such table: Shard_Size_In_Bytes)
ERROR org.opensearch.performanceanalyzer.rca.store.metric.AggregateMetric - RCA: Caught an exception while getting the DB SQL [select avg(sum_max) shard_avg from (select IndexName, ShardID, sum(max) sum_max from Shard_Size_In_Bytes where ShardID is not null group by IndexName, ShardID order by sum_max desc) alias_97600446]; [SQLITE_ERROR] SQL error or missing database (no such table: Shard_Size_In_Bytes)
ERROR org.opensearch.performanceanalyzer.rca.store.metric.AggregateMetric - RCA: Caught an exception while getting the DB SQL [select IndexName, ShardID, sum(max) from Shard_Size_In_Bytes where ShardID is not null group by IndexName, ShardID order by sum(max) desc]; [SQLITE_ERROR] SQL error or missing database (no such table: Shard_Size_In_Bytes)
ERROR org.opensearch.performanceanalyzer.rca.framework.api.persist.SQLParsingUtil - Eden fails to match any row in field MemType.
ERROR org.opensearch.performanceanalyzer.rca.framework.api.persist.SQLParsingUtil - Survivor fails to match any row in field MemType.
ERROR org.opensearch.performanceanalyzer.rca.store.collector.NodeConfigCollector - Metric value is NaN for resource:resource_enum: YOUNG_GEN