Metrics and Traces ingestion to opensearch

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser): 3.1.0

Describe the issue: Metrics and Traces Ingestion to Opensearch

Configuration: I have deployed Fluentbit, opensearch and Opensearch Dashboards in my AKS Azure environment. Currently getting the logs from kubernetes pods.

I am lookin for options to achieve end to end monitoring using opensearch apart of logging which already achived. Like the Metrics, Traces, Alerts, Dashboards for metrics & logs..etc

I know people suggest to use the Prmoetheus + grafane for metrics, traces and alerting… But I want to stay at one place not to bring too many components for monitoring which is difficult to main.

So I need suggestions here on achieving this.

Thanks
Seshadri

2

@seshadri How are you going to collect the metrics? What will be your metrics source?

3

Hi @pablo Basically some of our projects ingesting metrics from opentelemetry using an apm collector.. so the idea is to expose them a opensource endpoint where they can ingest metrics to collect and show them in opensearch dashboards.

Also I am trying to achieve the default metrics for our workloads within the the kubernetes cluster and VMs… I remember when we were using elastic earlier we get some default dashboards and metrics for the workloads.. so somehow trying to achive the same here…

Please some suggestions if you have any…

I am currently lookin at this prometheus-exporter-3.2.0.0.zip plugin activated on the nodes, trying to get it work and see what kind of metrics it brings in.

THanks
Seshadri

1 Like
4

@seshadri I’ve recently built a scenario with Prometheus exporter 3.2 and Prometheus Operator.
Grafana is optional with Prometheus as you can connect OpenSearch Dashboard directly to Prometheus with Data Source Prometheus plugin.
I didn’t try OpenTelemetry but if you’re looking for metrics from OpenSearch cluster then Prometheus is more mature than OpenTelemetry at the moment.

1 Like
5

@pablo that’s great to hear.

Could you throw some insights on it? Basically what I am trying to achieve and couldn’t it work so far.

below is my opensearch-values.yaml:

imagePullSecrets:
  - name: logging-reg-jfrog

image:
  repository: xxxct.jfrog.io/ops-docker/opensearchproject/opensearch
  tag: 3.2.0
  pullPolicy: Always

persistence:
  enabled: true
  storageClass: ""
  size: 30Gi
  image: xxxct.jfrog.io/ops-docker/busybox
  imageTag: latest

sysctlInit:
  enabled: true
  image: xxxct.jfrog.io/ops-docker/busybox
  imageTag: latest
  command: ["sysctl", "-w", "vm.max_map_count=262144"]
  securityContext:
    privileged: true

extraEnvs:
  - name: DISABLE_INSTALL_DEMO_CONFIG
    value: "true"
  - name: DISABLE_SECURITY_PLUGIN
    value: "false"
  - name: OPENSEARCH_USERNAME
    valueFrom:
      secretKeyRef:
        name: opensearch-auth-secret
        key: username
  - name: OPENSEARCH_PASSWORD
    valueFrom:
      secretKeyRef:
        name: opensearch-auth-secret
        key: password
  - name: CLIENT_ID
    valueFrom:
      secretKeyRef:
        name: os-client-id-secret
        key: client-id
  - name: CLIENT_SECRET
    valueFrom:
      secretKeyRef:
        name: os-client-id-secret
        key: client-secret

opensearchJavaOpts: "-Xms1g -Xmx1g"

securityConfig:
  enabled: true
  admin:
    secretName: opensearch-auth-secret
    usernameKey: username
    passwordKey: password
  path: /usr/share/opensearch/config/opensearch-security
  configSecret: os-config-secret
  internalUsersSecret: os-internal-users-secret
  # rolesSecret: os-roles-secret
  rolesMappingSecret: os-roles-mapping-secret
  tenantsSecret: os-tenants-secret
  nodesDnSecret: os-nodes-dn-secret
  config:
    securityConfigSecret: ""
    dataComplete: true
	
extraVolumes:
  - name: opensearch-tls
    secret:
      secretName: opensearch-tls
  
  - name: saml-metadata
    secret:
      secretName: saml-metadata

extraVolumeMounts:
  - name: opensearch-tls
    mountPath: /usr/share/opensearch/config/certs
    readOnly: true

  - name: saml-metadata
    mountPath: /usr/share/opensearch/config/metadata_file.xml
    subPath: metadata_file.xml
    readOnly: true

config:
  
  opensearch.yml: |
    network.host: 0.0.0.0
    plugins.security.ssl.transport.enabled: true
    plugins.security.ssl.transport.pemcert_filepath: /usr/share/opensearch/config/certs/tls.crt
    plugins.security.ssl.transport.pemkey_filepath: /usr/share/opensearch/config/certs/tls.key
    plugins.security.ssl.transport.pemtrustedcas_filepath: /usr/share/opensearch/config/certs/tls.crt
    plugins.security.ssl.transport.enforce_hostname_verification: false

    plugins.security.ssl.http.enabled: true
    plugins.security.ssl.http.pemcert_filepath: /usr/share/opensearch/config/certs/tls.crt
    plugins.security.ssl.http.pemkey_filepath: /usr/share/opensearch/config/certs/tls.key
    plugins.security.ssl.http.pemtrustedcas_filepath: /usr/share/opensearch/config/certs/tls.crt
    plugins.security.allow_unsafe_democertificates: false
    plugins.security.allow_default_init_securityindex: true

    plugins.security.authcz.admin_dn:
      - "CN=opensearch-cluster-master.logging.svc.cluster.local"
      - "CN=opensearch-cluster-master"
      - "CN=opensearch-cluster-master.logging"
      - "CN=opensearch-cluster-master.logging.svc"

    plugins.security.nodes_dn:
      - "CN=opensearch-cluster-master.logging.svc.cluster.local"
      - "CN=opensearch-cluster-master"
      - "CN=opensearch-cluster-master.logging"
      - "CN=opensearch-cluster-master.logging.svc"
    plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
    plugins.security.system_indices.enabled: true
   
resources:
  requests:
    cpu: "500m"
    memory: "2Gi"
  limits:
    cpu: "1"
    memory: "4Gi"

plugins:
  enabled: true
  installList:
    - "https://xxxct.jfrog.io/artifactory/shd-helm/prometheus-exporter-3.2.0.0.zip"
  removeList: [ ]
  pluginCredentialsFromSecret:
    secretName: logging-reg-jfrog
    usernameKey: username
    passwordKey: password

metrics:
  enabled: true
  serviceMonitor:
    enabled: true
    interval: 30s
    scrapeTimeout: 10s
    labels:
      release: prometheus

opensearch-dashboards-values.yaml:

imagePullSecrets:
  - name: logging-reg-jfrog

image:
  repository: xxxct.jfrog.io/ops-docker/opensearchproject/opensearch-dashboards
  tag: 3.2.0
  pullPolicy: Always

replicaCount: 1

opensearchHosts: "https://opensearch-cluster-master.logging.svc.cluster.local:9200"

resources:
  requests:
    cpu: "100m"
    memory: "512Mi"
  limits:
    cpu: "100m"
    memory: "512Mi"

service:
  type: ClusterIP
  port: 5601

ingress:
  enabled: true
  ingressClassName: nginx
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    cert-manager.io/cluster-issuer: selfsigned-issuer
    nginx.ingress.kubernetes.io/proxy-buffer-size: 256k
  hosts:
    - host: opensearch.int.app.xxx.com
      paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: opensearch-dashboards
              port:
                number: 5601
  tls:
    - secretName: opensearch-dashboards-cert
      hosts:
        - opensearch.int.app.xxx.com

extraEnvs:
  - name: OPENSEARCH_USERNAME
    valueFrom:
      secretKeyRef:
        name: opensearch-auth-secret
        key: username
  - name: OPENSEARCH_PASSWORD
    valueFrom:
      secretKeyRef:
        name: opensearch-auth-secret
        key: password
  - name: CLIENT_ID
    valueFrom:
      secretKeyRef:
        name: os-client-id-secret
        key: client-id
  - name: CLIENT_SECRET
    valueFrom:
      secretKeyRef:
        name: os-client-id-secret
        key: client-secret
  
  - name: OPENSEARCH_SSL_CERTIFICATE_AUTHORITIES
    value: "/usr/share/opensearch-dashboards/config/certs/ca.crt"
  - name: OPENSEARCH_SSL_CERTIFICATE
    value: "/usr/share/opensearch-dashboards/config/certs/tls.crt"
  - name: OPENSEARCH_SSL_KEY
    value: "/usr/share/opensearch-dashboards/config/certs/tls.key"


config:
  opensearch_dashboards.yml: |
    server.host: "0.0.0.0"
    server.rewriteBasePath: false
    opensearch.hosts: ["https://opensearch-cluster-master.logging.svc.cluster.local:9200"]
    opensearch.ssl.verificationMode: none
    opensearch.ssl.certificateAuthorities: ${OPENSEARCH_SSL_CERTIFICATE_AUTHORITIES}
    opensearch.ssl.certificate: ${OPENSEARCH_SSL_CERTIFICATE}
    opensearch.ssl.key: ${OPENSEARCH_SSL_KEY}
    opensearch.ssl.keyPassphrase: ""
    opensearch_security.allow_client_certificates: true
    opensearch_security.cookie.secure: false
    opensearch.requestTimeout: 120000
    opensearch.pingTimeout: 120000
    opensearch.shardTimeout: 120000
    opensearch.username: "${OPENSEARCH_USERNAME}"
    opensearch.password: "${OPENSEARCH_PASSWORD}"
    opensearch_security.auth.multiple_auth_enabled: true
    opensearch_security.auth.type: ["saml","basicauth"]
    opensearch.requestHeadersWhitelist: [authorization, securitytenant]
    server.xsrf.whitelist:
      - /_opendistro/_security/saml/acs
      - /_opendistro/_security/saml/acs/idpinitiated
      - /_opendistro/_security/saml/logout
      - /_plugins/_security/saml/acs
      - /_plugins/_security/saml/acs/idpinitiated
      - /_plugins/_security/saml/logout
    opensearch_security.multitenancy.enabled: true
    opensearch_security.multitenancy.tenants.enable_global: true
    opensearch_security.multitenancy.tenants.enable_private: true
    opensearch_security.multitenancy.tenants.preferred: ["Private", "Global"]
    logging.verbose: true
    logging.events: ["authentication", "security", "http", "error"]


extraVolumes:
  - name: opensearch-ca-cert
    secret:
      secretName: opensearch-tls

extraVolumeMounts:
  - name: opensearch-ca-cert
    mountPath: /usr/share/opensearch-dashboards/config/certs
    readOnly: true

Where I am trying add the prometheus plugin and trying to access it with https://opensearch.int.app.xxx.com/_prometheus/metrics but couldn’t get this worked…

I am surely missing something here… Please suggest.

Thanks
Seshadri