Reporting is not working for me -- complains about invalid index name "_opendistro"

OpenSearch Dashboards Reporting Plugin
1

I feel like I’m missing something obvious here.
I have a user called “joe_user” in internal_users.yml

joe_user:
  hash: <redacted>
  reserved: false
  hidden: false
  backend_roles:
  - "admin"
  description: "Joe User"

who is mapped to a backend_role called “admin”
which in roles_mapping.yml should map to the predefined security role of “all_access”.
I’m looking at the list of predefined security roles which includes “all_access”.

all_access:
  reserved: true
  hidden: false
  backend_roles:
  - "admin"

And that user seems to work just fine – except for generating reports.
I’m not using the report scheduler plugin.
I’m just trying to generate a PDF report.
I went to my dashboard, I clicked “Reporting”, I selected generate PDF report and I get an error

Screen Shot 2022-05-03 at 5.35.17 PM
Screen Shot 2022-05-03 at 5.35.17 PM896×356 9.61 KB

There is nothing relevant in the elasticsearch log, but I did find this success in the kibana.log

May 03 17:35:27 es-master kibana[15546]: {"type":"response","@timestamp":"2022-05-03T21:35:26Z","tags":[],"pid":15546,"method":"post","statusCode":200,"req":{"url":"/api/ui_metric/report","method":"post","headers":{"host":"10.43.117.10:5601","connection":"keep-alive","content-length":"123","sec-ch-ua":"\" Not A;Brand\";v=\
\"99\", \"Chromium\";v=\"99\", \"Google Chrome\";v=\"99\"","content-type":"application/json","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.83 Safari/537.36","kbn-version":"7.10.2","sec-ch-ua-platform":"\"macOS\"","accept":"*/*"\
,"origin":"https://10.43.117.10:5601","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://10.43.117.10:5601/app/dashboards","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"10.10.100.156","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac\
 OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.83 Safari/537.36","referer":"https://10.43.117.10:5601/app/dashboards"},"res":{"statusCode":200,"responseTime":302,"contentLength":9},"message":"POST /api/ui_metric/report 200 302ms - 9.0B"}

If I open up developer tools, I see this issue:

{"statusCode":400,"error":"Bad Request","message":"Invalid index name [_opendistro], must not start with '_'."}

The offending request url looks something like this:



/app/dashboards#/view/cf8654b0-a1f8-11ea-8847-eb065f5fdfa5?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:'2022-05-03T21:25:36.872Z',to:'2022-05-03T21:40:36.872Z'))&_a=(description:'A%20dashboard%20of%20all%20of%20the%20EFS%20Audit%20visualizations',filters:!(),fullScreenMode:!f,options:(hidePanelTitles:!f,\
useMargins:!t),query:(language:kuery,query:''),timeRestore:!f,title:'My%20EFS%20Audit%20Events%20dashboard',viewMode:view)

and here is an image of the request body:

Screen Shot 2022-05-03 at 5.48.54 PM
Screen Shot 2022-05-03 at 5.48.54 PM2572×988 302 KB

2

What version of Elasticsearch are you using? When digging around I see that a lot of people have this issue when using a pre- 7.x version of ES.

3

I’m using elasticsearch 7.10.2

4

and for kibana I’m using
opendistroforelasticsearch-kibana/stable,now 1.13.2 amd64 [installed,automatic]

5

And now I do see some errors in the elasticsearch log:

[2022-05-04T09:47:11,935][INFO ][c.a.o.s.p.PrivilegesEvaluator] [es-master] No index-level perm match for User [name=kibanaserver, backend_roles=[], requestedTenant=null] Resolved [aliases=[*], allIndices=[*], types=[*], originalRequested=[*], remoteIndices=[]] [Action [indices:monitor/settings/get]] [RolesChecked [kibana\
_server]]
[2022-05-04T09:47:11,937][INFO ][c.a.o.s.p.PrivilegesEvaluator] [es-master] No permissions for [indices:monitor/settings/get]

I am using kibanaserver as the user that kibana uses to connect to elasticsearch.
My kibana.yml has this relevant line:

elasticsearch.username: kibanaserver
6

I upgraded the kibana_server user to have all_access, and now I don’t get any errors, but I don’t get a report either. :frowning:

7

I tried going to the reports tab and I see different errors:

{"statusCode":400,"error":"Bad Request","message":"request [/_opendistro/_reports/instances] contains unrecognized parameter: [maxItems]"}
8

Do I need to install some more plugins?

I’m looking at Reporting plugin not working - #4 by comijac.

I wasn’t trying to schedule anything so I did not install the reports-scheduler plugin.
It seem that I may need that plugin anyway as well the kibana-reports plugin (I’ll see if I can figure out how to install those two).

9

@iamthealex-es you are making quick progress on this which is great! I think you are right. I believe what happens is when you want to generate a report it makes a call to the “reports-scheduler” plugin and simply schedules a report to be generated at that moment. This is because the reports are generated async.

Try that out and hopefully that should resolve it!

10

I installed the reports-scheduler plugin.
And apparently I already had the kibana-reports plugins
I am now able to generate a report.
Now, I can go back and start removing permissions to go back to minimal privileges needed.
I don’t have a deep understanding of the security model.
Is there any reason not to grant the kibana_server all_access?
I’m a bit confused why there is a kibana_server role account at all.
I would’ve expected that the user I login as would just be the only user involved in the transactions between kibana and elasticsearch.

bin/kibana plugin --install
opendistroAlertingKibana@1.13.0.0
opendistroAnomalyDetectionKibana@1.13.0.0
opendistroGanttChartKibana@1.13.0.0
opendistroIndexManagementKibana@1.13.0.1
opendistroNotebooksKibana@1.13.2.0
opendistroQueryWorkbenchKibana@1.13.0.0
opendistroReportsKibana@1.13.2.0
opendistroSecurityKibana@1.13.0.1
opendistroTraceAnalyticsKibana@1.13.2.0
1 Like
11

Kibana uses a kibanaserver user internally to talk to Elasticsearch when performing management calls. The only role that the kibanaserver user has is kibana_server role that only has permissions to .kibana* indices. But all_access role has all permissions to all indices, which is not what kibanaserver user is supposed to have.

1 Like
12

I will try to go back and reduce the privileges for kibanaserver to see if I can still generate a report. It will take a day before I can run the experiment.

Note that I did see errors in the elasticsearch.log so I predict I will have to add the ‘reports_full_access’ to the kibanaserver user. See this post above: Reporting is not working for me -- complains about invalid index name "_opendistro" - #5 by iamthealex-es

13

I have reports working for my admin user. Yay!

I was able to put back the kibanaserver user to the kibana_server role which is the original and principled set of permissions. Yay!

I am trying to get report generation working for my non-admin user.
I have my non-admin user mapped to 2 security roles.
The first security role is the role the non-admin user always had:

read_only_index:
  reserved: true
  hidden: false
  backend_roles:
  - "reader"

I tried adding a second security role to that backend group:

reports_full_access:
  reserved: true
  hidden: false
  backend_roles:
  - "reader"

But apparently the opendistro_security only looks at the access for the first security role (not the second).

[2022-05-06T12:56:30,496][INFO ][c.a.o.s.p.PrivilegesEvaluator] [es-master] No cluster-level perm match for User [name=joe_user, backend_roles=[reader], requestedTenant=null] Resolved [aliases=[*], allIndices=[*], types=[*], originalRequested=[*], remoteIndices=[]] [Action [cluster:admin/opendistro/reports/definition/create]] [RolesChecked [read_only_index]]. No permissions for [cluster:admin/opendistro/reports/definition/create]

It says the (security) roles checked were “read_only_index”, but I tried to give a second predefined security role of “reports_full_access” which is defined in Users and Roles - Open Distro Documentation. Boo!

1 Like
14

Do I really have to give up on predefined security roles and combine the access privileges from the 2 security roles into a single list of cluster and index access privileges?

15

Apparently mapping a user to multiple security groups is not fully supported (at least in my version of opendistro), so I changed the existing security role for non-admin users to include

    - cluster:admin/opendistro/reports/*

It all seems to be working now.