Hello everyone,
I have to say that I have been working with Kibana, OpenSearch for a very long time. Over the past two weeks, I had to configure OpenSearch in conjunction with Fluent-Bit on several servers.
I have to admit that this has been by far the most frustrating experience I’ve had in the last 15 years. And that’s despite the fact that we spent several weeks struggling with Jboss4 RMI and encrypting a legacy system while hundreds of thousands of clients were pushing the system to its limits…
The main reason for this frustrating experience is the highly disappointing “security plugin” and its configuration. Jesus Christ…
Let me start from the beginning.
Installation
You can download docker-compose.yml from the Opensearch website. It says there:
“The best way to try out OpenSearch is to use Docker Compose.”
Why try it out? I don’t want to try it out, I want to use it productively. If you want to try it out, there’s a playground where you can really try out everything. It seems to be a trend right now to offer complex systems for people to try out—without security, without overhead, completely disconnected from reality…
Anyway. I created certificates, customized docker-compose, and after solving a bunch of errors through trial and error, Google, AI, and “securityadmin.sh,” Opensearch finally started.
Let’s go! But wait, this looks completely different from the “try-out” version. The UI looks completely different, workspaces are suddenly there…
Never mind. Workspace created. Then I thought, okay, very good, seems to be working—now I’ll quickly create the roles and users via the UI so that a few people can start creating dashboards. Well… I’m logged in as “admin,” but I can’t create any users. I can’t create any roles either. But maybe permissions? Nope, not that either.
“You do not have permissions to view this data for Local cluster.”
Why should I, I’m just an admin… At the same time, I can’t also change the password for this admin! The error message:
Resource ‘admin’ is reserved.
What the ??? I mean now I know what it is, but
I’m sure the genius who came up with all this had something in mind, but it’s not intuitive at all, it’s not logical, and it’s completely unclear what to do. You can’t expect a newcomer to grasp, understand, and find all of that.
OK, after a lot of back and forth, I mounted the yml files: role.yml, roles_mapping.yml, etc. This is hell. This unnecessary complexity is overwhelming. It’s absolutely unclear what you’re supposed to write in these files. Yes, there’s a documentation page for it, but it only has examples. How am I supposed to know which allowed_actions my admin user needs?
How can I be sure that I am not granting a user too many permissions, or in my case, too few:
no permissions for [indices:admin/get] and User [name=admin, backend_roles=[admin], requestedTenant=null]
It’s unbelievable that you have to read and understand tons of documentation to accomplish a simple task: creating users. roles, mappings, action groups, internal users.
And I’m not alone, this question from 2021 where user has the same problem: Best practices for managing the admin user
Just when I thought nothing could surprise me, Opensearch went one better. If you want to create a initial user, you have to use the script “…/tools/hash.sh” to create a password hash, enter it in internal_user.yml, and then call securityadmin.sh to re-read the yml files and update the corresponding index.
A complete disaster in terms of security. Sending plain text passwords to a sh script via terminal. If you search Google for /opendistro_security/tools/hash.sh, you will find many results suggesting this as a genuine, fully-fledged solution for creating a user. Even Copilot suggested this to me as THE solution. Did many of the users who did this also clean up their shell history? Or, in other words, how many plaintext passwords are still lying around in the shell history?
This is the result of the complexity of the security plugin.
But hey, you can also create a user via the Rest API, and I like it really. A quick PUT request and it should work. But what do I write in the body? opendistro_security_roles – what’s that? Which backend_roles? No idea, I just want the user to be able to search all indexes. So, next round. Reading, asking questions, Googling, AI…
So please forgive my grumbling, but I need to vent my frustration and I don’t have any more computer mice to throw against the wall.
Security is important, but not like this. Perhaps you can create or share a fully functional, REAL production configuration with REAL roles and users on (github or in download section) with full configured docker-compose.
Simple real-life cases:
There is an admin who is allowed to write and there are users who are allowed to read. The user is given an initial password and changes it when they log in for the first time.
P.S.
And please clearly state somewhere that the latest version of filebeat is not compatible with opensearch 3.4.
It took me many hours and a lot of frustration to figure out why the logs couldn’t be pushed.
Your website, download section says: Opensearch is compatible with filebeat. So you install the latest version without a second thought and then despair during setup.