We have a set-up where we use AWS Elasticsearch service (with ES 7.7, i.e. OpenDistro 1.8) and write log data from fluent-bit running in EKS Kubernetes clusters, using the aws-for-fluent-bit Docker image (v2.8.0)
This works fine - if we set the access controls to full access for the fluent-bit IAM role. However, if we try to restrict permissions to only the operations that fluent-bit perform (essentially bulk writes), it stops working. I have tried various combinations of permissions, but not been successful unless I set full permissions (i.e. * ), which seems wrong.
Does anyone have a working setup for fluent-bit and using AWS Elasticsearch service, which is not full access?
@eriklz I am not using AWS Elasticsearch but I have Fluent Bit working with ODFE 1.7.0. In my case, the security setting for my logcollector role are:
I don’t know if that helps at all; things may be different with AWS Elasticsearch.
That is one combination I have tried and that did not work, even setting “index:" and "cluster:” does not help. Only thing that worked was to map the AWS IAM role as backend_role to the pre-defined “all_access” role.
For my custom role settings I did create a separate log_writer role and had a mapping which associated the backend_role with that role.
Did you find the solution to your issue? Are you still running ODFE 1.8?
@pablo No, not while I was working on that project. I am not working on that project anymore though since December, so not sure what the current status is.
I don’t know if it’s offtopic…if it is I apologize.
I’m having problems trying to send data from Fluent-bit to a self-hosted OpenSearch server.
Basically I’m running an docker container from the official image which is a single-node cluster with default credentials and demo snake-oil TLS certificate but a command like
fluent-bit -v -i cpu -t cpu -o es -p Host=192.168.1.145 -p Port=9200 -p Index=unIndex -p HTTP_User=admin -p HTTP_Passwd=admin -p Tls=on -p Tls.verify=off -m "*" doesn’t send anything.
I’ve even tried to removing security plugin and trying to connect through plain HTTP to see if it was a problem about encryption/authentication but it’s the same: transfer between Fluent-bit and OpenSearch doesn’t occur
I’ve open this issue in Fluent-bit github (Add compatibility for self-hosted OpenSearch servers in "es" output plugin, please! · Issue #4550 · fluent/fluent-bit · GitHub) but for completeness Iḿ writting this problem here, because I’n really stuck.
Thanks a lot