Weird error when trying to update index pattern

Security
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

OpenSearch 2.9.0 on Linux

Describe the issue:

Few days ago I noticed that users (including admins) are unable to update index pattern.
All went as usual, sometimes number of fields changes, however, when you reload index pattern, number of fields remains the same.

Exploring logs, I noticed what seems quite weird to me:

{“type”:“error”,“@timestamp”:“2024-02-01T13:09:41Z”,“tags”:,“pid”:1,“level”:“error”,“error”:{“message”:“Internal Server Error”,“name”:“Error”,“stack”:“Error: Internal Server Error\n at HapiResponseAdapter.toError (/usr/share/opensearch-dashboards/src/core/server/http/router/response_adapter.js:143:19)\n at HapiResponseAdapter.toHapiResponse (/usr/share/opensearch-dashboards/src/core/server/http/router/response_adapter.js:97:19)\n at HapiResponseAdapter.handle (/usr/share/opensearch-dashboards/src/core/server/http/router/response_adapter.js:92:17)\n at Router.handle (/usr/share/opensearch-dashboards/src/core/server/http/router/router.js:164:34)\n at runMicrotasks ()\n at processTicksAndRejections (node:internal/process/task_queues:96:5)\n at handler (/usr/share/opensearch-dashboards/src/core/server/http/router/router.js:124:50)\n at exports.Manager.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)\n at Object.internals.handler (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/handler.js:46:20)\n at exports.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/handler.js:31:20)\n at Request._lifecycle (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:371:32)\n at Request._execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:281:9)”},“url”:“http://logs-staging.nl2.exadel.com/api/saved_objects/index-pattern/1da8d2d0-c103-11ee-90f5-db300e02a36f",“message”:"Internal Server Error”}

Configuration:

Doesn’t seem to had changed last few months

Relevant Logs or Screenshots:

{“type”:“response”,“@timestamp”:“2024-02-01T13:09:40Z”,“tags”:,“pid”:1,“method”:“get”,“statusCode”:200,“req”:{“url”:“/api/index_patterns/_fields_for_wildcard?pattern=minio-%2A&meta_fields=_source&meta_fields=_id&meta_fields=_type&meta_fields=_index&meta_fields=_score”,“method”:“get”,“headers”:{“connection”:“upgrade”,“host”:“logs-staging.nl2.exadel.com”,“sec-ch-ua”:“"Not A(Brand";v="99", "Google Chrome";v="121", "Chromium";v="121"”,“content-type”:“application/json”,“osd-xsrf”:“osd-fetch”,“sec-ch-ua-mobile”:“?0”,“user-agent”:“Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36”,“osd-version”:“2.9.0”,“sec-ch-ua-platform”:“"Linux"”,“accept”:“/”,“sec-fetch-site”:“same-origin”,“sec-fetch-mode”:“cors”,“sec-fetch-dest”:“empty”,“referer”:“https://logs-staging.nl2.exadel.com/app/management/opensearch-dashboards/indexPatterns/patterns/1da8d2d0-c103-11ee-90f5-db300e02a36f",“accept-encoding”:"gzip, deflate, br”,“accept-language”:“en-US,en;q=0.9”,“securitytenant”:“EISD_DEV”},“remoteAddress”:“127.0.0.1”,“userAgent”:“Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36”,“referer”:“https://logs-staging.nl2.exadel.com/app/management/opensearch-dashboards/indexPatterns/patterns/1da8d2d0-c103-11ee-90f5-db300e02a36f"},“res”:{“statusCode”:200,“responseTime”:16,“contentLength”:9},“message”:"GET /api/index_patterns/_fields_for_wildcard?pattern=minio-%2A&meta_fields=_source&meta_fields=_id&meta_fields=_type&meta_fields=_index&meta_fields=_score 200 16ms - 9.0B”}
{“type”:“log”,“@timestamp”:“2024-02-01T13:09:41Z”,“tags”:[“error”,“opensearch”,“data”],“pid”:1,“message”:“[security_exception]: Update is not supported when FLS or DLS or Fieldmasking is activated”}
{“type”:“error”,“@timestamp”:“2024-02-01T13:09:41Z”,“tags”:,“pid”:1,“level”:“error”,“error”:{“message”:“Internal Server Error”,“name”:“Error”,“stack”:“Error: Internal Server Error\n at HapiResponseAdapter.toError (/usr/share/opensearch-dashboards/src/core/server/http/router/response_adapter.js:143:19)\n at HapiResponseAdapter.toHapiResponse (/usr/share/opensearch-dashboards/src/core/server/http/router/response_adapter.js:97:19)\n at HapiResponseAdapter.handle (/usr/share/opensearch-dashboards/src/core/server/http/router/response_adapter.js:92:17)\n at Router.handle (/usr/share/opensearch-dashboards/src/core/server/http/router/router.js:164:34)\n at runMicrotasks ()\n at processTicksAndRejections (node:internal/process/task_queues:96:5)\n at handler (/usr/share/opensearch-dashboards/src/core/server/http/router/router.js:124:50)\n at exports.Manager.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)\n at Object.internals.handler (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/handler.js:46:20)\n at exports.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/handler.js:31:20)\n at Request._lifecycle (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:371:32)\n at Request._execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:281:9)”},“url”:“http://logs-staging.nl2.exadel.com/api/saved_objects/index-pattern/1da8d2d0-c103-11ee-90f5-db300e02a36f",“message”:"Internal Server Error”}
{“type”:“response”,“@timestamp”:“2024-02-01T13:09:41Z”,“tags”:,“pid”:1,“method”:“put”,“statusCode”:500,“req”:{“url”:“/api/saved_objects/index-pattern/1da8d2d0-c103-11ee-90f5-db300e02a36f”,“method”:“put”,“headers”:{“connection”:“upgrade”,“host”:“logs-staging.nl2.exadel.com”,“content-length”:“47226”,“sec-ch-ua”:“"Not A(Brand";v="99", "Google Chrome";v="121", "Chromium";v="121"”,“content-type”:“application/json”,“osd-xsrf”:“osd-fetch”,“sec-ch-ua-mobile”:“?0”,“user-agent”:“Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36”,“osd-version”:“2.9.0”,“sec-ch-ua-platform”:“"Linux"”,“accept”:“/”,“origin”:“https://logs-staging.nl2.exadel.com”,“sec-fetch-site”:“same-origin”,“sec-fetch-mode”:“cors”,“sec-fetch-dest”:“empty”,“referer”:“https://logs-staging.nl2.exadel.com/app/management/opensearch-dashboards/indexPatterns/patterns/1da8d2d0-c103-11ee-90f5-db300e02a36f",“accept-encoding”:"gzip, deflate, br”,“accept-language”:“en-US,en;q=0.9”,“securitytenant”:“EISD_DEV”},“remoteAddress”:“127.0.0.1”,“userAgent”:“Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36”,“referer”:“https://logs-staging.nl2.exadel.com/app/management/opensearch-dashboards/indexPatterns/patterns/1da8d2d0-c103-11ee-90f5-db300e02a36f"},“res”:{“statusCode”:500,“responseTime”:14,“contentLength”:9},“message”:"PUT /api/saved_objects/index-pattern/1da8d2d0-c103-11ee-90f5-db300e02a36f 500 14ms - 9.0B”}

Full log:

2

Just to make sure, this is how my request looks like at security-auditlog:

image
image855×776 80.7 KB

3

Have you made any changes to the user’s roles in terms of FLS, DLS or field masking (Anonymization )?

Could you share roles_mapping.yml and the name of the test user?

4

Thank you @pablo for your reply!

I just realized that adding fake DLS to ‘sa’ (our sysadmins’ group) How is DLS applied when user has multiple roles - #6 by rlevitsky
have broken this.

So either some our sysadmins (those who are in any devs group) are unable to see all their indices or nobody (including cluster_admins) is able to update any index pattern (you can delete and recreate, though, but that’s not convenient).

Is there any chance we can have both?