Permissions for fluent-bit writes

Security
#1

We have a set-up where we use AWS Elasticsearch service (with ES 7.7, i.e. OpenDistro 1.8) and write log data from fluent-bit running in EKS Kubernetes clusters, using the aws-for-fluent-bit Docker image (v2.8.0)

This works fine - if we set the access controls to full access for the fluent-bit IAM role. However, if we try to restrict permissions to only the operations that fluent-bit perform (essentially bulk writes), it stops working. I have tried various combinations of permissions, but not been successful unless I set full permissions (i.e. * ), which seems wrong.

Does anyone have a working setup for fluent-bit and using AWS Elasticsearch service, which is not full access?

#2

@eriklz I am not using AWS Elasticsearch but I have Fluent Bit working with ODFE 1.7.0. In my case, the security setting for my logcollector role are:

{
  "index_permissions": [
    {
      "index_patterns": [
        "myindexes_*"
      ],
      "fls": [],
      "masked_fields": [],
      "allowed_actions": [
        "crud",
        "create_index"
      ]
    }
  ],
  "tenant_permissions": [],
  "cluster_permissions": []
}

I don’t know if that helps at all; things may be different with AWS Elasticsearch.

#3

That is one combination I have tried and that did not work, even setting “index:" and "cluster:” does not help. Only thing that worked was to map the AWS IAM role as backend_role to the pre-defined “all_access” role.
For my custom role settings I did create a separate log_writer role and had a mapping which associated the backend_role with that role.

#4

@eriklz

Did you find the solution to your issue? Are you still running ODFE 1.8?

#5

@pablo No, not while I was working on that project. I am not working on that project anymore though since December, so not sure what the current status is.

#6

Hello.
I don’t know if it’s offtopic…if it is I apologize.

I’m having problems trying to send data from Fluent-bit to a self-hosted OpenSearch server.
Basically I’m running an docker container from the official image which is a single-node cluster with default credentials and demo snake-oil TLS certificate but a command like fluent-bit -v -i cpu -t cpu -o es -p Host=192.168.1.145 -p Port=9200 -p Index=unIndex -p HTTP_User=admin -p HTTP_Passwd=admin -p Tls=on -p Tls.verify=off -m "*" doesn’t send anything.

I’ve even tried to removing security plugin and trying to connect through plain HTTP to see if it was a problem about encryption/authentication but it’s the same: transfer between Fluent-bit and OpenSearch doesn’t occur

I’ve open this issue in Fluent-bit github (Add compatibility for self-hosted OpenSearch servers in "es" output plugin, please! · Issue #4550 · fluent/fluent-bit · GitHub) but for completeness Iḿ writting this problem here, because I’n really stuck.

Thanks a lot