All,
Testing out observability section and it seams creating a dashboard fails silently. I was unable to save my Visualization from log querying.
Ubuntu-22.0.4 single node installation ( 4GB ram, 4cores, 500Gb drive)
Opensearch 2.7.0
Opensearch-Dashboards 2.7.0
Logstash–8.6.1.x
Three devices sending logs using nxlog, packetbeat, winlogbeat.
Opensearch_config
root@ansible:/etc/opensearch# cat opensearch.yml | egrep -v "^\s*(#|$)"
cluster.name: openlab
path.data: /var/lib/opensearch
path.logs: /var/log/opensearch
bootstrap.memory_lock: true
network.host: 192.168.1.100
http.port: 9200
discovery.type: single-node
plugins.alerting.alert_history_retention_period: 1d
plugins.security.nodes_dn_dynamic_config_enabled: true
indices.query.bool.max_clause_count: 10240 <--tried this but it seamed to not work.
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opensearch-notifications-*", ".opensearch-notebooks", ".opensearch-observability", ".opendistro-asynchronous-search-response*", ".replication-metadata-store"]
node.max_local_storage_nodes: 3
plugins.security.ssl.transport.pemcert_filepath: /etc/opensearch/node1.pem
plugins.security.ssl.transport.pemkey_filepath: /etc/opensearch/node1-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: /etc/opensearch/root-ca.pem
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: /etc/opensearch/node1.pem
plugins.security.ssl.http.pemkey_filepath: /etc/opensearch/node1-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: /etc/opensearch/root-ca.pem
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
- 'CN=ansible.domain.com,OU=admin,O=enseva,L=cedar,ST=iowa,C=us'
plugins.security.nodes_dn:
- 'CN=aansible.domain.com,OU=admin,O=enseva,L=ceda,ST=iowa,C=us'
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
root@ansible:/etc/opensearch#
Opensearch-Dashboard_config
root@ansible:/etc/opensearch# cat /etc/opensearch-dashboards/opensearch_dashboards.yml | egrep -v "^\s*(#|$)"
---
server.port: 5601
server.host: ansible.domain.com
server.name: "ansible"
opensearchDashboards.index: ".kibana"
opensearch.username: "admin"
opensearch.password: "changeit"
logging.dest: /var/log/opensearch-dashboards/opensearch-dashboards.log
data.search.usageTelemetry.enabled: true
ml_commons_dashboards.enabled: true
opensearch_security.ui.openid.login.buttonname: openid
opensearch_security.ui.saml.login.buttonname: Enseva-Labs
opensearch.hosts: ["https://ansible.domain.com:9200"]
opensearch.ssl.verificationMode: none
opensearch.requestHeadersAllowlist: [ authorization,securitytenant ]
server.ssl.enabled: true
server.ssl.certificate: "/etc/opensearch-dashboards/node1.pem"
server.ssl.key: "/etc/opensearch-dashboards/node1-key.pem"
opensearch.ssl.certificateAuthorities: "/etc/opensearch-dashboards/root-ca.pem"
opensearch_security.multitenancy.enabled: true
opensearch_security.multitenancy.tenants.preferred: ["Private", "Global"]
opensearch_security.readonly_mode.roles: ["kibana_read_only"]
opensearch_security.cookie.secure: true
opensearch_security.auth.type: ["basicauth","saml"]
opensearch_security.auth.multiple_auth_enabled: true
server.xsrf.allowlist: ["/_opendistro/_security/saml/acs/idpinitiated", "/_opendistro/_security/saml/acs", "/_opendistro/_security/saml/logout"]
root@ansible:/etc/opensearch#
Everything else works with our , Policys( rollover, delete) templatse Overview dashboards, notifications. Opensearch plugin section works great. I decided to test Observability like Logs dashboards, etc…
I was able to install example Samples i.e., Logs.
I followed the instructions here.
When I try to create a dashboards I click save nothing happens.
Nothing in the logs
As for creating a Visualization in Observability using the link above.
Click on Visualization and select pie. NOTE: I replaced my FQDN.
All good, saved the visualization created to the example LOG dashboard . Click save.
Error shown below.
Error: Bad Request
at Fetch._callee3$ (https://ansible.domain.com:5601/6100/bundles/core/core.entry.js:15:585214)
at tryCatch (https://anansible.domain.com:5601/6100/bundles/plugin/observabilityDashboards/observabilityDashboards.chunk.3.js:625:70467)
at Generator.invoke [as _invoke] (https://ansible.domain.com:5601/6100/bundles/plugin/observabilityDashboards/observabilityDashboards.chunk.3.js:625:74483)
at Generator.next (https://ansible.domain.com:5601/6100/bundles/plugin/observabilityDashboards/observabilityDashboards.chunk.3.js:625:71662)
at fetch_asyncGeneratorStep (https://ansible.domain.com:5601/6100/bundles/core/core.entry.js:15:578126)
at _next (https://ansible.domain.com:5601/6100/bundles/core/core.entry.js:15:578442)
Log file
[2023-05-24T20:20:31,870][INFO ][o.o.p.PluginsService ] [ansible.domain.com] PluginService:onIndexModule index:[.opensearch-sap-windows-detectors-queries-000003/q2FSLMP4SeCcav6lp-igZQ]
[2023-05-24T20:20:38,176][INFO ][o.o.o.r.ObservabilityRestHandler] [ansible.domain.com] observability:executeGetRequest idList:[] types:[operationalPanel], from:0, maxItems:10000, sortField:null, sortOrder=null, filters={}
[2023-05-24T20:20:38,176][INFO ][o.o.o.a.ObservabilityActions] [ansible.domain.com] observability:ObservabilityObject-get []
[2023-05-24T20:20:38,176][INFO ][o.o.o.a.ObservabilityActions] [ansible.domain.com] observability:ObservabilityObject-getAll
[2023-05-24T20:20:38,179][INFO ][o.o.o.i.ObservabilityIndex] [ansible.domain.com] observability:getAllObservabilityObjects types:[operationalPanel] from:0, maxItems:10000, sortField:null, sortOrder=null, filters={} retCount:18, totalCount:18
[2023-05-24T20:20:39,200][INFO ][o.o.o.r.ObservabilityRestHandler] [ansible.domain.com] observability:executeGetRequest idList:[] types:[operationalPanel], from:0, maxItems:10000, sortField:null, sortOrder=null, filters={}
[2023-05-24T20:20:39,201][INFO ][o.o.o.a.ObservabilityActions] [ansible.domain.com] observability:ObservabilityObject-get []
[2023-05-24T20:20:39,201][INFO ][o.o.o.a.ObservabilityActions] [ansible.domain.com] observability:ObservabilityObject-getAll
[2023-05-24T20:20:39,203][INFO ][o.o.o.i.ObservabilityIndex] [ansible.domain.com] observability:getAllObservabilityObjects types:[operationalPanel] from:0, maxItems:10000, sortField:null, sortOrder=null, filters={} retCount:18, totalCount:18
[2023-05-24T20:20:41,097][INFO ][o.o.j.s.JobScheduler ] [ansible.domain.com] Will delay 30248 miliseconds for next execution of job winlogbeat-2023.05.23
[2023-05-24T20:20:41,915][INFO ][o.o.i.i.ManagedIndexRunner] [ansible.domain.com] Executing attempt_transition_step for winlogbeat-2023.05.23
[2023-05-24T20:20:41,915][INFO ][o.o.i.i.ManagedIndexRunner] [ansible.domain.com] Finished executing attempt_transition_step for winlogbeat-2023.05.23
[2023-05-24T20:20:42,631][INFO ][o.o.o.r.ObservabilityRestHandler] [ansible.domain.com] observability:executeGetRequest idList:[] types:[operationalPanel], from:0, maxItems:10000, sortField:null, sortOrder=null, filters={}
[2023-05-24T20:20:42,632][INFO ][o.o.o.a.ObservabilityActions] [ansible.domain.com] observability:ObservabilityObject-get []
[2023-05-24T20:20:42,632][INFO ][o.o.o.a.ObservabilityActions] [ansible.domain.com] observability:ObservabilityObject-getAll
[2023-05-24T20:20:42,635][INFO ][o.o.o.i.ObservabilityIndex] [ansible.domain.com] observability:getAllObservabilityObjects types:[operationalPanel] from:0, maxItems:10000, sortField:null, sortOrder=null, filters={} retCount:18, totalCount:18
[2023-05-24T20:21:10,650][INFO ][o.o.j.s.JobScheduler ] [ansible.domain.com] Will delay 20969 miliseconds for next execution of job winlogbeat-2023.05.25
[2023-05-24T20:21:11,075][INFO ][o.o.i.i.ManagedIndexRunner] [ansible.domain.com] Executing attempt_transition_step for winlogbeat-2023.05.25
[2023-05-24T20:21:11,075][INFO ][o.o.i.i.ManagedIndexRunner] [ansible.domain.com] Finished executing attempt_transition_step for winlogbeat-2023.05.25
[2023-05-24T20:22:12,085][ERROR][o.o.s.p.r.RestPPLQueryAction] [ansible.domain .com] Error happened during query handling
java.lang.IllegalArgumentException: Two sibling aggregations cannot have the same name: [composite_buckets]
at org.opensearch.search.aggregations.AggregatorFactories$Builder.addAggregator(AggregatorFactories.java:357) ~[opensearch -2.7.0.jar:2.7.0]
at org.opensearch.search.builder.SearchSourceBuilder.aggregation(SearchSourceBuilder.java:657) ~[opensearch-2.7.0.jar:2.7. 0]
at org.opensearch.sql.opensearch.request.OpenSearchRequestBuilder.lambda$pushDownAggregation$0(OpenSearchRequestBuilder.ja va:163) ~[opensearch-2.7.0.0.jar:?]
at java.util.Collections$SingletonList.forEach(Collections.java:4966) ~[?:?]
at org.opensearch.sql.opensearch.request.OpenSearchRequestBuilder.pushDownAggregation(OpenSearchRequestBuilder.java:163) ~ [opensearch-2.7.0.0.jar:?]
at org.opensearch.sql.opensearch.storage.scan.OpenSearchIndexScanAggregationBuilder.build(OpenSearchIndexScanAggregationBu ilder.java:59) ~[opensearch-2.7.0.0.jar:?]
at org.opensearch.sql.opensearch.storage.scan.OpenSearchIndexScanBuilder.build(OpenSearchIndexScanBuilder.java:54) ~[opens earch-2.7.0.0.jar:?]
at org.opensearch.sql.planner.DefaultImplementor.visitTableScanBuilder(DefaultImplementor.java:137) ~[core-2.7.0.0.jar:?]
at org.opensearch.sql.planner.DefaultImplementor.visitTableScanBuilder(DefaultImplementor.java:52) ~[core-2.7.0.0.jar:?]
at org.opensearch.sql.storage.read.TableScanBuilder.accept(TableScanBuilder.java:121) ~[core-2.7.0.0.jar:?]
at org.opensearch.sql.planner.DefaultImplementor.visitChild(DefaultImplementor.java:153) ~[core-2.7.0.0.jar:?]
at org.opensearch.sql.planner.DefaultImplementor.visitProject(DefaultImplementor.java:77) ~[core-2.7.0.0.jar:?]
at org.opensearch.sql.planner.DefaultImplementor.visitProject(DefaultImplementor.java:52) ~[core-2.7.0.0.jar:?]
at org.opensearch.sql.planner.logical.LogicalProject.accept(LogicalProject.java:42) ~[core-2.7.0.0.jar:?]
at org.opensearch.sql.opensearch.storage.OpenSearchIndex.implement(OpenSearchIndex.java:163) ~[opensearch-2.7.0.0.jar:?]
at org.opensearch.sql.planner.Planner.plan(Planner.java:40) ~[core-2.7.0.0.jar:?]
at org.opensearch.sql.executor.QueryService.plan(QueryService.java:100) ~[core-2.7.0.0.jar:?]
at org.opensearch.sql.executor.QueryService.lambda$executePlan$1(QueryService.java:67) ~[core-2.7.0.0.jar:?]
at java.util.Optional.ifPresentOrElse(Optional.java:198) ~[?:?]
at org.opensearch.sql.executor.QueryService.executePlan(QueryService.java:64) [core-2.7.0.0.jar:?]
at org.opensearch.sql.executor.QueryService.execute(QueryService.java:43) [core-2.7.0.0.jar:?]
at org.opensearch.sql.executor.execution.QueryPlan.execute(QueryPlan.java:50) [core-2.7.0.0.jar:?]
at org.opensearch.sql.opensearch.executor.OpenSearchQueryManager.lambda$submit$0(OpenSearchQueryManager.java:33) [opensear ch-2.7.0.0.jar:?]
at org.opensearch.sql.opensearch.executor.OpenSearchQueryManager.lambda$withCurrentContext$1(OpenSearchQueryManager.java:4 7) [opensearch-2.7.0.0.jar:?] at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747) [opensearch-2 .7.0.jar:2.7.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
Opensearch-Dashboard logs
{"type":"response","@timestamp":"2023-05-25T01:20:40Z","tags":[],"pid":650659,"method":"post","statusCode":400,"req":{"url":"/api/saved_objects/observability-visualization","method":"post","headers":{"host":"ansible.domain.com:5601","connection":"keep-alive","content-length":"769","sec-ch-ua":"\"Chromium\";v=\"112\", \"Google Chrome\";v=\"112\", \"Not:A-Brand\";v=\"99\"","content-type":"application/json","osd-xsrf":"osd-fetch","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36","osd-version":"2.7.0","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","origin":"https://ansible.domain.com:5601","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://ansible.domain.com:5601/app/observability-logs","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9","securitytenant":"admin_tenant"},"remoteAddress":"192.168.1.100","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36","referer":"https://ansible.domain.com:5601/app/observability-logs"},"res":{"statusCode":400,"responseTime":24,"contentLength":9},"message":"POST /api/saved_objects/observability-visualization 400 24ms - 9.0B"}
{"type":"response","@timestamp":"2023-05-25T01:20:42Z","tags":[],"pid":650659,"method":"get","statusCode":200,"req":{"url":"/api/saved_objects/_find?type=observability-panel","method":"get","headers":{"host":"ansible.domain.com:5601","connection":"keep-alive","sec-ch-ua":"\"Chromium\";v=\"112\", \"Google Chrome\";v=\"112\", \"Not:A-Brand\";v=\"99\"","content-type":"application/json","osd-xsrf":"osd-fetch","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36","osd-version":"2.7.0","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://ansible.domain.com:5601/app/observability-logs","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9","securitytenant":"admin_tenant"},"remoteAddress":"192.168.1.100","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36","referer":"https://ansible.domain.com:5601/app/observability-logs"},"res":{"statusCode":200,"responseTime":16,"contentLength":9},"message":"GET /api/saved_objects/_find?type=observability-panel 200 16ms - 9.0B"}
{"type":"response","@timestamp":"2023-05-25T01:20:42Z","tags":[],"pid":650659,"method":"get","statusCode":200,"req":{"url":"/api/observability/operational_panels/panels","method":"get","headers":{"host":"ansible.domain.com:5601","connection":"keep-alive","sec-ch-ua":"\"Chromium\";v=\"112\", \"Google Chrome\";v=\"112\", \"Not:A-Brand\";v=\"99\"","content-type":"application/json","osd-xsrf":"osd-fetch","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36","osd-version":"2.7.0","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://ansible.domain.com:5601/app/observability-logs","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9","securitytenant":"admin_tenant"},"remoteAddress":"192.168.1.100","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36","referer":"https://ansible.domain.com:5601/app/observability-logs"},"res":{"statusCode":200,"responseTime":25,"contentLength":9},"message":"GET /api/observability/operational_panels/panels 200 25ms - 9.0B"}
{"type":"log","@timestamp":"2023-05-25T01:20:43Z","tags":["error","opensearch","data"],"pid":650659,"message":"[strict_dynamic_mapping_exception]: mapping set to strict, dynamic introduction of [observability-visualization] within [_doc] is not allowed"}
{"type":"response","@timestamp":"2023-05-25T01:20:43Z","tags":[],"pid":650659,"method":"post","statusCode":400,"req":{"url":"/api/saved_objects/observability-visualization","method":"post","headers":{"host":"ansible.domain.com:5601","connection":"keep-alive","content-length":"769","sec-ch-ua":"\"Chromium\";v=\"112\", \"Google Chrome\";v=\"112\", \"Not:A-Brand\";v=\"99\"","content-type":"application/json","osd-xsrf":"osd-fetch","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36","osd-version":"2.7.0","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","origin":"https://ansible.domain.com:5601","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://ansible.domain.com:5601/app/observability-logs","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9","securitytenant":"admin_tenant"},"remoteAddress":"192.168.1.100","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36","referer":"https://ansible.domain.com:5601/app/observability-logs"},"res":{"statusCode":400,"responseTime":20,"contentLength":9},"message":"POST /api/saved_objects/observability-visualization 400 20ms - 9.0B"}
{"type":"response","@timestamp":"2023-05-25T01:32:01Z","tags":[],"pid":650659,"method":"get","statusCode":200,"req":{"url":"/api/saved_objects/_find?type=observability-panel","method":"get","headers":{"host":"ansible.domain.com:5601","connection":"keep-alive","sec-ch-ua":"\"Chromium\";v=\"112\", \"Google Chrome\";v=\"112\", \"Not:A-Brand\";v=\"99\"","content-type":"application/json","osd-xsrf":"osd-fetch","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36","osd-version":"2.7.0","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://ansible.domain.com:5601/app/observability-logs","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9","securitytenant":"admin_tenant"},"remoteAddress":"192.1688.1.100","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36","referer":"https://ansible.domain.com:5601/app/observability-logs"},"res":{"statusCode":200,"responseTime":21,"contentLength":9},"message":"GET /api/saved_objects/_find?type=observability-panel 200 21ms - 9.0B"}
{"type":"response","@timestamp":"2023-05-25T01:32:01Z","tags":[],"pid":650659,"method":"get","statusCode":200,"req":{"url":"/api/observability/operational_panels/panels","method":"get","headers":{"host":"ansible.domain.com:5601","connection":"keep-alive","sec-ch-ua":"\"Chromium\";v=\"112\", \"Google Chrome\";v=\"112\", \"Not:A-Brand\";v=\"99\"","content-type":"application/json","osd-xsrf":"osd-fetch","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36","osd-version":"2.7.0","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://ansible.domain.com:5601/app/observability-logs","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9","securitytenant":"admin_tenant"},"remoteAddress":"192.168.1.100","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36","referer":"https://ansible.domain.com:5601/app/observability-logs"},"res":{"statusCode":200,"responseTime":19,"contentLength":9},"message":"GET /api/observability/operational_panels/panels 200 19ms - 9.0B"}
Not sure whats going on. Any help would be apperciated.
Thanks in advance.
, I deleted and recreated the .opensearch-observability && .opendistro_security index set and all is good.