User / Role for Fluentd

Security
,
1

Dear all,

I’ve managed to get OpenSearch and the Dashboard up and running with the internal user database. Using self-signed TLS certificates for OpenSearch and a reverse proxy for the dashboard.

My setup is essentially as follows

Multiple Docker Hosts, which having fluent-bit installed. forwarding traffic to one centralised fluentd setup, which should send the traffic top OpenSearch. I’ve been struggling with the security plugin even during the previous two days I needed to setup a basic environment and now need some help with getting fluentd to talk to OpenSearch.

Running OS in a docker container with these config files (removed all comments):

opensearch.yml

network.host: 0.0.0.0
plugins.security.nodes_dn:
- "CN=opensearch-node1.my-domain.de,OU=IT,O=MyOrg,L=MyCity,ST=MyState,C=DE"

plugins.security.nodes_dn_dynamic_config_enabled: false

plugins.security.authcz.admin_dn:
- "emailAddress=admin@domain,CN=admin@domain,OU=IT,O= MyOrg,L= MyCity,ST= MyState,C=DE"

plugins.security.roles_mapping_resolution: MAPPING_ONLY
plugins.security.ssl.transport.pemcert_filepath: opensearch-node1.crt.pem
plugins.security.ssl.transport.pemkey_filepath: opensearch-node1.key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: ca.crt.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: opensearch-node1.crt.pem
plugins.security.ssl.http.pemkey_filepath: opensearch-node1.key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: ca.crt.pem
plugins.security.audit.type: internal_opensearch
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]

plugins.security.allow_default_init_securityindex: true
plugins.security.restapi.password_validation_regex: '(?=.*[A-Z])(?=.*[^a-zA-Z\d])(?=.*[0-9])(?=.*[a-z]).{8,}'
plugins.security.restapi.password_validation_error_message: "Password must be minimum 8 characters long and must contain at least one uppercase letter, one lowercase letter, one digit, and one special character."

config.yml

_meta:
  type: "config"
  config_version: 2

config:
  dynamic:
    authc:
      basic_internal_auth_domain:
        description: "Authenticate via HTTP Basic against internal users database"
        http_enabled: true
        transport_enabled: true
        order: 1
        http_authenticator:
          type: basic
          challenge: true
        authentication_backend:
          type: intern

internal_users.yml

---
# This is the internal user database
# The hash value is a bcrypt hash and can be generated with plugin/tools/hash.sh

_meta:
  type: "internalusers"
  config_version: 2

# Define your internal users here

admin:
  hash: "somehash"
  reserved: true
  backend_roles:
    - "admin"
  description: "Built-in admin user"

kibanaserver:
  hash: "somehash"
  reserved: true
  description: "User for the OpenSearch Dashboards server"

I can login with the admin user and now I can also see the Security options in the dashboard, so I assume I would be doing the remaining configuration from there.

The (for now very basic) fluentd config looks as follows:

<source>
  type forward
  bind 0.0.0.0
  port 24224
  <security>
    self_hostname fluentd.my-domain.de
    shared_key somekey
  </security>
  <transport tls>
    version                TLSv1_2
    ca_path                /fluentd/etc/ssl/ca.crt.pem
    cert_path              /fluentd/etc/ssl/fluentd.crt.pem
    private_key_path       /fluentd/etc/ssl/fluentd.key.pem
    private_key_passphrase somepassphrase
    client_cert_auth       true
  </transport>
</source>

<match source.docker.*>
  @type opensearch
  host opensearch-node1
  port 9200
  scheme https
  ssl_verify false
  index_name vpscontabo
  ca_file /fluentd/etc/ssl/ca.crt.pem
</match>

This yields a 401 unauthorised error though on the fluentd side. I know its a very open question, but which steps do I need to take on the OpenSearch side to ingest the logs from fluentd?

Thanks in advance
Daniel

2

@Wasabi - thank you so much for providing all of your info. It helps.

Forgive the silly question, but I’m not seeing a username and password configured in your source.docker match. Consider fluentd’s documentation on configuring the opensearch output plugin:

I would let it use the default admin username and password to narrow out any other issues. If it works like that, at least we know the issue is somewhere else. :slight_smile:

3

Hi @nateynate, thank you so much for taking the time to respond.

I somehow didn’t want to use the admin credentials in a static configuration file, so I tried to figure out which permissions would be needed (wanted to create a role for fluentd-ingress or something), but couldn’t find this in the documentation (neither on the OpenSearch nor on the fluentd plugin documentation).

With the admin credentials it looks fine, the logs are being forwarded and are visible in the Discovery section in the dashboard. So I guess the question boils down to this: what are the minimal permissions I need for a role that can handle fluentd ingress?

Thanks!

1 Like
4

I don’t want to mislead you here - the various types of permissions that are available aren’t something that I have studied extensively, so I don’t quite feel like an expert.

I imagine the only thing you’d need is the ability to write data to an index, and perhaps create a new index if you provide a new one in your config file that doesn’t exist.

I’ll make sure that I get myself familiar with those topics, as this is a perfectly valid question.

Nate

5

Hi Nate,

I managed to get it working with the following set of permissions:

Cluster Permissions: cluster:monitor
Index Permissions: crud,create_index on whatever index pattern you plan to use

Not sure if that is the minimal set though (the plugin did complain about the cluster:monitor permission missing), so maybe there’s some more research that can be done and it might be worth to add this to the plugin documentation.

1 Like
6

This would be a perfect thing to contribute to our documentation-website in the form of an issue. If you ever have trouble finding documentation or find existing documentation lacking, here and github would be the place to bring it up.

I’m glad you got it going! And, a special thanks for coming back to share your solution with everyone. It’s how you build a community!

Nate