Cannot Generate PDF / PNG Reports in ODFE 1.13.2 (CSV Works Fine)

OpenSearch Dashboards Reporting Plugin
1

I am attempting to generate PDF and PNG reports from the Kibana UI. When I click Download PDF/PNG in the UI, I immediately see a 401 response in the Kibana logs for the endpoint /app/visualize?security_tenant=global (pasted below), and 1 minute later I get a timeout and 500 error. I see the exact same behavior when I use the private tenant, too.

However, I have no problem generating and downloading CSV reports with both the global and private tenants, so I am confused by the 401 error and am not sure where it is coming from.

I am using OpenDistro Elastic & Kibana versions 1.13.2 in docker containers behind an Apache2 reverse proxy. I also have Proxy Auth enabled.
Please see below for the log output (starts at initial download request and ends after timeout; includes 401 error).

od-kibana           | {"type":"response","@timestamp":"2021-07-27T14:04:30Z","tags":[],"pid":1,"method":"get","statusCode":200,"req":{"url":"/api/v1/multitenancy/tenant","method":"get","headers":{"host":"xx.xxx.xxx.xxx","sec-ch-ua":"\" Not;A Brand\";v=\"99\", \"Google Chrome\";v=\"91\", \"Chromium\";v=\"91\"","pragma":"no-cache","accept-language":"en-US,en;q=0.9,zh-CN;q=0.8,zh;q=0.7,zh-TW;q=0.6","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36","content-type":"application/json","accept":"*/*","kbn-version":"7.10.2","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://xx.xxx.xxx.xxx/kibana/app/visualize","accept-encoding":"gzip, deflate, br","oidc_claim_sub":"8adjkf-wip98-1242-90aa-dfjai082","oidc_claim_email_verified":"1","oidc_claim_roles":"default-roles-myrealm,offline_access,test_role,uma_authorization","oidc_claim_name":"Fname Lname","oidc_claim_preferred_username":"flname","oidc_claim_given_name":"Fname","oidc_claim_family_name":"Lname","oidc_claim_email":"Fname.Lname@group.domain.com","oidc_claim_exp":"1627393756","oidc_claim_iat":"1627393456","oidc_claim_auth_time":"1627393257","oidc_claim_jti":"b8908a33-ab9q-183a-a72b-j8quyew88123p12","oidc_claim_iss":"https://xx.xxx.xxx.xxx/auth/realms/myrealm","oidc_claim_aud":"apache-proxy","oidc_claim_typ":"ID","oidc_claim_azp":"apache-proxy","oidc_claim_nonce":"bALqrSKbe2QcU_9yyTUDVKet77cqCBQmXSr-LwnvofQ","oidc_claim_session_state":"q583nf54-ad64-10q9-ac36-b28a90402780","oidc_claim_at_hash":"q5qikAOIJ1281alkKIQPlQAui","oidc_claim_acr":"0","oidc_access_token":"eyJ................","oidc_access_token_expires":"1627393756","x-proxy-user":"flname","x-proxy-roles":"admin","x-forwarded-for":"xxx.xx.xxx.xxx","x-forwarded-host":"xx.xxx.xxx.xxx","x-forwarded-server":"10.0.1.45","connection":"Keep-Alive","securitytenant":""},"remoteAddress":"xxx.xx.xxx.xxx","userAgent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36","referer":"https://xx.xxx.xxx.xxx/kibana/app/visualize"},"res":{"statusCode":200,"responseTime":10,"contentLength":9},"message":"GET /api/v1/multitenancy/tenant 200 10ms - 9.0B"}

od-kibana           | {"type":"log","@timestamp":"2021-07-27T14:04:30Z","tags":["info","plugins","opendistroReportsKibana"],"pid":1,"message":"domain enables security, use session cookie to access"}

od-kibana           | {"type":"log","@timestamp":"2021-07-27T14:04:30Z","tags":["info","plugins","opendistroReportsKibana"],"pid":1,"message":"original queryUrl http://0:5601/kibana/app/visualize?security_tenant=global#/edit/4d739760-eee3-11eb-8f1e-1991bfa88630?type=markdown&_g=(filters:!(),refreshInterval:(pause:!f,value:900000),time:(from:'2021-07-26T14:04:29.203Z',to:'2021-07-27T14:04:29.203Z'))&_a=(filters:!(),linked:!f,query:(language:kuery,query:''),uiState:(),vis:(aggs:!(),params:(fontSize:12,markdown:'Test%20Markdown',openLinksInNewTab:!f),title:Test,type:markdown))"}

od-kibana           | {"type":"response","@timestamp":"2021-07-27T14:04:30Z","tags":[],"pid":1,"method":"get","statusCode":401,"req":{"url":"/app/visualize?security_tenant=global","method":"get","headers":{"host":"0.0.0.0:5601","connection":"keep-alive","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/77.0.3844.0 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3","accept-encoding":"gzip, deflate","accept-language":"en-US"},"remoteAddress":"127.0.0.1","userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/77.0.3844.0 Safari/537.36"},"res":{"statusCode":401,"responseTime":2,"contentLength":9},"message":"GET /app/visualize?security_tenant=global 401 2ms - 9.0B"}

od-kibana           | {"type":"log","@timestamp":"2021-07-27T14:04:31Z","tags":["info","plugins","opendistroReportsKibana"],"pid":1,"message":"page url http://0.0.0.0:5601/kibana/app/visualize?security_tenant=global#/edit/4d739760-eee3-11eb-8f1e-1991bfa88630?type=markdown&_g=(filters:!(),refreshInterval:(pause:!f,value:900000),time:(from:'2021-07-26T14:04:29.203Z',to:'2021-07-27T14:04:29.203Z'))&_a=(filters:!(),linked:!f,query:(language:kuery,query:''),uiState:(),vis:(aggs:!(),params:(fontSize:12,markdown:'Test%20Markdown',openLinksInNewTab:!f),title:Test,type:markdown))"}

od-kibana           | {"type":"log","@timestamp":"2021-07-27T14:05:32Z","tags":["error","plugins","opendistroReportsKibana"],"pid":1,"message":"Failed to generate report: TimeoutError: waiting for selector \".visEditor__content\" failed: timeout 60000ms exceeded"}

od-kibana           | {"type":"log","@timestamp":"2021-07-27T14:05:32Z","tags":["error","plugins","opendistroReportsKibana"],"pid":1,"message":"{ TimeoutError: waiting for selector \".visEditor__content\" failed: timeout 60000ms exceeded\n    at new WaitTask (/usr/share/kibana/plugins/opendistroReportsKibana/node_modules/puppeteer-core/lib/DOMWorld.js:549:28)\n    at DOMWorld._waitForSelectorOrXPath (/usr/share/kibana/plugins/opendistroReportsKibana/node_modules/puppeteer-core/lib/DOMWorld.js:478:22)\n    at DOMWorld.waitForSelector (/usr/share/kibana/plugins/opendistroReportsKibana/node_modules/puppeteer-core/lib/DOMWorld.js:432:17)\n    at Frame.waitForSelector (/usr/share/kibana/plugins/opendistroReportsKibana/node_modules/puppeteer-core/lib/FrameManager.js:627:47)\n    at Frame.<anonymous> (/usr/share/kibana/plugins/opendistroReportsKibana/node_modules/puppeteer-core/lib/helper.js:112:23)\n    at Page.waitForSelector (/usr/share/kibana/plugins/opendistroReportsKibana/node_modules/puppeteer-core/lib/Page.js:1095:29)\n    at createVisualReport (/usr/share/kibana/plugins/opendistroReportsKibana/server/routes/utils/visual_report/visualReportHelper.ts:134:18) name: 'TimeoutError' }"}

od-kibana           | {"type":"error","@timestamp":"2021-07-27T14:04:30Z","tags":[],"pid":1,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Internal Server Error\n    at HapiResponseAdapter.toError (/usr/share/kibana/src/core/server/http/router/response_adapter.js:132:19)\n    at HapiResponseAdapter.toHapiResponse (/usr/share/kibana/src/core/server/http/router/response_adapter.js:86:19)\n    at HapiResponseAdapter.handle (/usr/share/kibana/src/core/server/http/router/response_adapter.js:81:17)\n    at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:164:34)"},"url":{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":null,"search":"?timezone=America/New_York","query":{"timezone":"America/New_York"},"pathname":"/api/reporting/generateReport","path":"/api/reporting/generateReport?timezone=America/New_York","href":"/api/reporting/generateReport?timezone=America/New_York"},"message":"Internal Server Error"}

od-kibana           | {"type":"response","@timestamp":"2021-07-27T14:04:30Z","tags":[],"pid":1,"method":"post","statusCode":500,"req":{"url":"/api/reporting/generateReport?timezone=America%2FNew_York","method":"post","headers":{"host":"xx.xxx.xxx.xxx","sec-ch-ua":"\" Not;A Brand\";v=\"99\", \"Google Chrome\";v=\"91\", \"Chromium\";v=\"91\"","pragma":"no-cache","accept-language":"en-US,en;q=0.9,zh-CN;q=0.8,zh;q=0.7,zh-TW;q=0.6","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36","content-type":"application/json","accept":"*/*","kbn-version":"7.10.2","origin":"https://xx.xxx.xxx.xxx","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://xx.xxx.xxx.xxx/kibana/app/visualize","accept-encoding":"gzip, deflate, br","oidc_claim_sub":"8adjkf-wip98-1242-90aa-dfjai082","oidc_claim_email_verified":"1","oidc_claim_roles":"default-roles-myrealm,offline_access,test_role,uma_authorization","oidc_claim_name":"Fname Lname","oidc_claim_preferred_username":"flname","oidc_claim_given_name":"Fname","oidc_claim_family_name":"Lname","oidc_claim_email":"Fname.Lname@group.domain.com","oidc_claim_exp":"1627393756","oidc_claim_iat":"1627393456","oidc_claim_auth_time":"1627393257","oidc_claim_jti":"b8908a33-ab9q-183a-a72b-j8quyew88123p12","oidc_claim_iss":"https://xx.xxx.xxx.xxx/auth/realms/myrealm","oidc_claim_aud":"apache-proxy","oidc_claim_typ":"ID","oidc_claim_azp":"apache-proxy","oidc_claim_nonce":"bALqrSKbe2QcU_9yyTUDVKet77cqCBQmXSr-LwnvofQ","oidc_claim_session_state":"q583nf54-ad64-10q9-ac36-b28a90402780","oidc_claim_at_hash":"q5qikAOIJ1281alkKIQPlQAui","oidc_claim_acr":"0","oidc_access_token":"eyJ................","oidc_access_token_expires":"1627393756","x-proxy-user":"flname","x-proxy-roles":"admin","x-forwarded-for":"xxx.xx.xxx.xxx","x-forwarded-host":"xx.xxx.xxx.xxx","x-forwarded-server":"10.0.1.45","content-length":"895","connection":"Keep-Alive","securitytenant":""},"remoteAddress":"xxx.xx.xxx.xxx","userAgent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36","referer":"https://xx.xxx.xxx.xxx/kibana/app/visualize"},"res":{"statusCode":500,"responseTime":62131,"contentLength":9},"message":"POST /api/reporting/generateReport?timezone=America%2FNew_York 500 62131ms - 9.0B"}
2

This is definitely related, can you send more information eg. Any errors from Kibanaog with increased verbosity. Thanks

3

@anirudha I added “logging.verbose: true” to my kibana.yml, but none of the additional output (pasted at the bottom of this comment) appears to be related to the 401 error.

I also tried adding the following to my elasticsearch log4j2.properties

logger.opendistro_security.name = com.amazon.opendistroforelasticsearch.security
logger.opendistro_security.level = trace
logger.opendistro_security.appenderRef.rolling.ref = rolling
logger.opendistro_security.appenderRef.rolling_old.ref = rolling_old
logger.opendistro_security.additivity = false

Is there another way to increase logging output that may provide additional detail on where the 401 is coming from?

od-kibana | {"type":"log","@timestamp":"2021-07-29T14:04:42Z","tags":["debug","metrics"],"pid":1,"message":"Refreshing metrics"}

od-kibana | {"type":"response","@timestamp":"2021-07-29T14:04:43Z","tags":[],"pid":1,"method":"get","statusCode":200,"req":{"url":"/api/v1/multitenancy/tenant","method":"get","headers":{"host":"xx.xxx.xxx.xxx","sec-ch-ua":"\" Not;A Brand\";v=\"99\", \"Google Chrome\";v=\"91\", \"Chromium\";v=\"91\"","pragma":"no-cache","accept-language":"en-US,en;q=0.9,zh-CN;q=0.8,zh;q=0.7,zh-TW;q=0.6","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36","content-type":"application/json","accept":"*/*","kbn-version":"7.10.2","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://xx.xxx.xxx.xxx/kibana/app/visualize","accept-encoding":"gzip, deflate, br","oidc_claim_sub":"8n91kaou-my67-9018-18aj-aosu981773baj","oidc_claim_email_verified":"1","oidc_claim_roles":"default-roles-myrealm,offline_access,test_role,uma_authorization","oidc_claim_name":"Fname Lname","oidc_claim_preferred_username":"rlname","oidc_claim_given_name":"Fname","oidc_claim_family_name":"Lname","oidc_claim_email":"fname.lname@group.org.com","oidc_claim_exp":"1627566733","oidc_claim_iat":"1627566433","oidc_claim_auth_time":"1627566433","oidc_claim_jti":"59cc1784-2b29-4d6b-bdb2-f3f94b9f43a3","oidc_claim_iss":"https://xx.xxx.xxx.xxx/auth/realms/myrealm","oidc_claim_aud":"apache-proxy","oidc_claim_typ":"ID","oidc_claim_azp":"apache-proxy","oidc_claim_nonce":"kJIez6KZd08lurC7FjmQoNsusFim2Ull8GvDyBaYU5E","oidc_claim_session_state":"199iurnla-9987-35c9-7n1j-0d7662k13oq444","oidc_claim_at_hash":"4OzdiYhLkFKrg2gwtF7wcA","oidc_claim_acr":"1","oidc_access_token":"eyJ........","oidc_access_token_expires":"1627566733","x-proxy-user":"rlname","x-proxy-roles":"admin","x-forwarded-for":"xxx.xx.xxx.xx","x-forwarded-host":"xx.xxx.xxx.xxx","x-forwarded-server":"xx.x.x.xx","connection":"Keep-Alive","securitytenant":""},"remoteAddress":"xxx.xx.x.x","userAgent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36","referer":"https://xx.xxx.xxx.xxx/kibana/app/visualize"},"res":{"statusCode":200,"responseTime":5,"contentLength":9},"message":"GET /api/v1/multitenancy/tenant 200 5ms - 9.0B"}

od-kibana | {"type":"log","@timestamp":"2021-07-29T14:04:44Z","tags":["info","plugins","opendistroReportsKibana"],"pid":1,"message":"domain enables security, use session cookie to access"}

od-kibana | {"type":"log","@timestamp":"2021-07-29T14:04:44Z","tags":["info","plugins","opendistroReportsKibana"],"pid":1,"message":"original queryUrl http://0:5601/kibana/app/visualize?security_tenant=global#/edit/4d739760-eee3-11eb-8f1e-1991bfa88630?_g=(filters:!(),refreshInterval:(pause:!f,value:900000),time:(from:'2021-07-28T14:04:42.671Z',to:'2021-07-29T14:04:42.671Z'))&_a=(filters:!(),linked:!f,query:(language:kuery,query:''),uiState:(),vis:(aggs:!(),params:(fontSize:12,markdown:'Test%20Markdown',openLinksInNewTab:!f),title:Test,type:markdown))"}

od-kibana | {"type":"response","@timestamp":"2021-07-29T14:04:44Z","tags":[],"pid":1,"method":"get","statusCode":401,"req":{"url":"/app/visualize?security_tenant=global","method":"get","headers":{"host":"0.0.0.0:5601","connection":"keep-alive","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/77.0.3844.0 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3","accept-encoding":"gzip, deflate","accept-language":"en-US"},"remoteAddress":"127.0.0.1","userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/77.0.3844.0 Safari/537.36"},"res":{"statusCode":401,"responseTime":3,"contentLength":9},"message":"GET /app/visualize?security_tenant=global 401 3ms - 9.0B"}

od-kibana | {"type":"ops","@timestamp":"2021-07-29T14:04:44Z","tags":[],"pid":1,"os":{"load":[0.8193359375,0.80322265625,0.7607421875],"mem":{"total":16304513024,"free":7904096256},"uptime":7433},"proc":{"uptime":397.984,"mem":{"rss":166649856,"heapTotal":126693376,"heapUsed":116199896,"external":3971302},"delay":0.12289100047200918},"load":{"requests":{"5601":{"total":8,"disconnects":0,"statusCodes":{"200":6,"401":1}}},"responseTimes":{"5601":{"avg":5.857142857142857,"max":11}},"sockets":{"http":{"total":0},"https":{"total":0}}},"message":"memory: 110.8MB uptime: 0:06:38 load: [0.82 0.80 0.76] delay: 0.123"}

od-kibana | {"type":"log","@timestamp":"2021-07-29T14:04:45Z","tags":["info","plugins","opendistroReportsKibana"],"pid":1,"message":"page url http://0.0.0.0:5601/kibana/app/visualize?security_tenant=global#/edit/4d739760-eee3-11eb-8f1e-1991bfa88630?_g=(filters:!(),refreshInterval:(pause:!f,value:900000),time:(from:'2021-07-28T14:04:42.671Z',to:'2021-07-29T14:04:42.671Z'))&_a=(filters:!(),linked:!f,query:(language:kuery,query:''),uiState:(),vis:(aggs:!(),params:(fontSize:12,markdown:'Test%20Markdown',openLinksInNewTab:!f),title:Test,type:markdown))"}

od-kibana | {"type":"response","@timestamp":"2021-07-29T14:04:44Z","tags":[],"pid":1,"method":"post","statusCode":200,"req":{"url":"/api/ui_metric/report","method":"post","headers":{"host":"xx.xxx.xxx.xxx","sec-ch-ua":"\" Not;A Brand\";v=\"99\", \"Google Chrome\";v=\"91\", \"Chromium\";v=\"91\"","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36","kbn-version":"7.10.2","content-type":"application/json","accept":"*/*","origin":"https://xx.xxx.xxx.xxx","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://xx.xxx.xxx.xxx/kibana/app/visualize","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9","oidc_claim_sub":"8n91kaou-my67-9018-18aj-aosu981773baj","oidc_claim_email_verified":"1","oidc_claim_roles":"default-roles-myrealm,offline_access,test_role,uma_authorization","oidc_claim_name":"Fname Lname","oidc_claim_preferred_username":"rlname","oidc_claim_given_name":"Fname","oidc_claim_family_name":"Lname","oidc_claim_email":"fname.lname@group.org.com","oidc_claim_exp":"1627566733","oidc_claim_iat":"1627566433","oidc_claim_auth_time":"1627566433","oidc_claim_jti":"59cc1784-2b29-4d6b-bdb2-f3f94b9f43a3","oidc_claim_iss":"https://xx.xxx.xxx.xxx/auth/realms/myrealm","oidc_claim_aud":"apache-proxy","oidc_claim_typ":"ID","oidc_claim_azp":"apache-proxy","oidc_claim_nonce":"kJIez6KZd08lurC7FjmQoNsusFim2Ull8GvDyBaYU5E","oidc_claim_session_state":"199iurnla-9987-35c9-7n1j-0d7662k13oq444","oidc_claim_at_hash":"4OzdiYhLkFKrg2gwtF7wcA","oidc_claim_acr":"1","oidc_access_token":"eyJ........","oidc_access_token_expires":"1627566733","x-proxy-user":"rlname","x-proxy-roles":"admin","x-forwarded-for":"xxx.xx.xxx.xx","x-forwarded-host":"xx.xxx.xxx.xxx","x-forwarded-server":"xx.x.x.xx","content-length":"169","connection":"Keep-Alive","securitytenant":""},"remoteAddress":"xxx.xx.x.x","userAgent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36","referer":"https://xx.xxx.xxx.xxx/kibana/app/visualize"},"res":{"statusCode":200,"responseTime":152,"contentLength":9},"message":"POST /api/ui_metric/report 200 152ms - 9.0B"}

od-kibana | {"type":"log","@timestamp":"2021-07-29T14:04:47Z","tags":["debug","metrics"],"pid":1,"message":"Refreshing metrics"}

od-kibana | {"type":"ops","@timestamp":"2021-07-29T14:04:49Z","tags":[],"pid":1,"os":{"load":[0.75341796875,0.78955078125,0.75634765625],"mem":{"total":16304513024,"free":8088416256},"uptime":7438},"proc":{"uptime":402.986,"mem":{"rss":166371328,"heapTotal":126693376,"heapUsed":115955488,"external":4101906},"delay":0.0932820001617074},"load":{"requests":{"5601":{"total":0,"disconnects":0,"statusCodes":{"200":1}}},"responseTimes":{"5601":{"avg":152,"max":152}},"sockets":{"http":{"total":0},"https":{"total":0}}},"message":"memory: 110.6MB uptime: 0:06:43 load: [0.75 0.79 0.76] delay: 0.093"}

od-kibana | {"type":"log","@timestamp":"2021-07-29T14:04:52Z","tags":["debug","metrics"],"pid":1,"message":"Refreshing metrics"}

od-kibana | {"type":"ops","@timestamp":"2021-07-29T14:04:54Z","tags":[],"pid":1,"os":{"load":[0.69287109375,0.7763671875,0.751953125],"mem":{"total":16304513024,"free":8089174016},"uptime":7443},"proc":{"uptime":407.987,"mem":{"rss":166612992,"heapTotal":126693376,"heapUsed":116202512,"external":4241598},"delay":0.09310999978333712},"load":{"requests":{"5601":{"total":0,"disconnects":0,"statusCodes":{}}},"responseTimes":{"5601":{"avg":null,"max":0}},"sockets":{"http":{"total":0},"https":{"total":0}}},"message":"memory: 110.8MB uptime: 0:06:48 load: [0.69 0.78 0.75] delay: 0.093"}

od-kibana | {"type":"log","@timestamp":"2021-07-29T14:04:57Z","tags":["debug","metrics"],"pid":1,"message":"Refreshing metrics"}

od-kibana | {"type":"ops","@timestamp":"2021-07-29T14:04:59Z","tags":[],"pid":1,"os":{"load":[0.7978515625,0.796875,0.7587890625],"mem":{"total":16304513024,"free":8088657920},"uptime":7448},"proc":{"uptime":412.989,"mem":{"rss":166850560,"heapTotal":126693376,"heapUsed":116436416,"external":4373098},"delay":0.09290800057351589},"load":{"requests":{"5601":{"total":0,"disconnects":0,"statusCodes":{}}},"responseTimes":{"5601":{"avg":null,"max":0}},"sockets":{"http":{"total":0},"https":{"total":0}}},"message":"memory: 111.0MB uptime: 0:06:53 load: [0.80 0.80 0.76] delay: 0.093"}

od-kibana | {"type":"log","@timestamp":"2021-07-29T14:05:02Z","tags":["debug","metrics"],"pid":1,"message":"Refreshing metrics"}

od-kibana | {"type":"ops","@timestamp":"2021-07-29T14:05:04Z","tags":[],"pid":1,"os":{"load":[0.814453125,0.80029296875,0.76025390625],"mem":{"total":16304513024,"free":8089948160},"uptime":7453},"proc":{"uptime":417.991,"mem":{"rss":167133184,"heapTotal":126693376,"heapUsed":116670168,"external":4504598},"delay":0.09504599962383509},"load":{"requests":{"5601":{"total":0,"disconnects":0,"statusCodes":{}}},"responseTimes":{"5601":{"avg":null,"max":0}},"sockets":{"http":{"total":0},"https":{"total":0}}},"message":"memory: 111.3MB uptime: 0:06:58 load: [0.81 0.80 0.76] delay: 0.095"}

od-kibana | {"type":"log","@timestamp":"2021-07-29T14:05:07Z","tags":["debug","metrics"],"pid":1,"message":"Refreshing metrics"}

od-kibana | {"type":"ops","@timestamp":"2021-07-29T14:05:09Z","tags":[],"pid":1,"os":{"load":[0.7490234375,0.78662109375,0.755859375],"mem":{"total":16304513024,"free":8039424000},"uptime":7458},"proc":{"uptime":422.991,"mem":{"rss":166412288,"heapTotal":127217664,"heapUsed":116026952,"external":4101962},"delay":0.09433500003069639},"load":{"requests":{"5601":{"total":0,"disconnects":0,"statusCodes":{}}},"responseTimes":{"5601":{"avg":null,"max":0}},"sockets":{"http":{"total":0},"https":{"total":0}}},"message":"memory: 110.7MB uptime: 0:07:03 load: [0.75 0.79 0.76] delay: 0.094"}

od-kibana | {"type":"log","@timestamp":"2021-07-29T14:05:12Z","tags":["debug","metrics"],"pid":1,"message":"Refreshing metrics"}

od-kibana | {"type":"ops","@timestamp":"2021-07-29T14:05:15Z","tags":[],"pid":1,"os":{"load":[0.68896484375,0.7734375,0.75146484375],"mem":{"total":16304513024,"free":8089157632},"uptime":7463},"proc":{"uptime":427.993,"mem":{"rss":166727680,"heapTotal":127217664,"heapUsed":116272248,"external":4233462},"delay":0.08964299969375134},"load":{"requests":{"5601":{"total":0,"disconnects":0,"statusCodes":{}}},"responseTimes":{"5601":{"avg":null,"max":0}},"sockets":{"http":{"total":0},"https":{"total":0}}},"message":"memory: 110.9MB uptime: 0:07:08 load: [0.69 0.77 0.75] delay: 0.090"}

od-kibana | {"type":"log","@timestamp":"2021-07-29T14:05:17Z","tags":["debug","metrics"],"pid":1,"message":"Refreshing metrics"}

od-kibana | {"type":"ops","@timestamp":"2021-07-29T14:05:20Z","tags":[],"pid":1,"os":{"load":[0.6337890625,0.76025390625,0.7470703125],"mem":{"total":16304513024,"free":8018616320},"uptime":7468},"proc":{"uptime":432.995,"mem":{"rss":166916096,"heapTotal":127217664,"heapUsed":116505328,"external":4364962},"delay":0.16557299997657537},"load":{"requests":{"5601":{"total":0,"disconnects":0,"statusCodes":{}}},"responseTimes":{"5601":{"avg":null,"max":0}},"sockets":{"http":{"total":0},"https":{"total":0}}},"message":"memory: 111.1MB uptime: 0:07:13 load: [0.63 0.76 0.75] delay: 0.166"}

od-kibana | {"type":"log","@timestamp":"2021-07-29T14:05:22Z","tags":["debug","metrics"],"pid":1,"message":"Refreshing metrics"}

od-kibana | {"type":"ops","@timestamp":"2021-07-29T14:05:25Z","tags":[],"pid":1,"os":{"load":[0.66357421875,0.7646484375,0.74853515625],"mem":{"total":16304513024,"free":8088109056},"uptime":7473},"proc":{"uptime":437.996,"mem":{"rss":167096320,"heapTotal":127217664,"heapUsed":116738416,"external":4496462},"delay":0.10125199984759092},"load":{"requests":{"5601":{"total":0,"disconnects":0,"statusCodes":{}}},"responseTimes":{"5601":{"avg":null,"max":0}},"sockets":{"http":{"total":0},"https":{"total":0}}},"message":"memory: 111.3MB uptime: 0:07:18 load: [0.66 0.76 0.75] delay: 0.101"}

od-kibana | {"type":"log","@timestamp":"2021-07-29T14:05:27Z","tags":["debug","metrics"],"pid":1,"message":"Refreshing metrics"}

od-kibana | {"type":"ops","@timestamp":"2021-07-29T14:05:30Z","tags":[],"pid":1,"os":{"load":[0.6103515625,0.751953125,0.744140625],"mem":{"total":16304513024,"free":8090431488},"uptime":7478},"proc":{"uptime":442.998,"mem":{"rss":166510592,"heapTotal":127217664,"heapUsed":116097144,"external":4102018},"delay":0.09809799958020449},"load":{"requests":{"5601":{"total":0,"disconnects":0,"statusCodes":{}}},"responseTimes":{"5601":{"avg":null,"max":0}},"sockets":{"http":{"total":0},"https":{"total":0}}},"message":"memory: 110.7MB uptime: 0:07:23 load: [0.61 0.75 0.74] delay: 0.098"}

od-kibana | {"type":"log","@timestamp":"2021-07-29T14:05:32Z","tags":["debug","metrics"],"pid":1,"message":"Refreshing metrics"}

od-kibana | {"type":"ops","@timestamp":"2021-07-29T14:05:35Z","tags":[],"pid":1,"os":{"load":[0.9619140625,0.82275390625,0.76708984375],"mem":{"total":16304513024,"free":8045264896},"uptime":7483},"proc":{"uptime":448,"mem":{"rss":166768640,"heapTotal":127217664,"heapUsed":116330248,"external":4233518},"delay":0.09495500009506941},"load":{"requests":{"5601":{"total":0,"disconnects":0,"statusCodes":{}}},"responseTimes":{"5601":{"avg":null,"max":0}},"sockets":{"http":{"total":0},"https":{"total":0}}},"message":"memory: 110.9MB uptime: 0:07:28 load: [0.96 0.82 0.77] delay: 0.095"}

od-kibana | {"type":"log","@timestamp":"2021-07-29T14:05:37Z","tags":["debug","metrics"],"pid":1,"message":"Refreshing metrics"}

od-kibana | {"type":"ops","@timestamp":"2021-07-29T14:05:40Z","tags":[],"pid":1,"os":{"load":[0.884765625,0.80908203125,0.7626953125],"mem":{"total":16304513024,"free":8044232704},"uptime":7488},"proc":{"uptime":453.002,"mem":{"rss":166768640,"heapTotal":127217664,"heapUsed":116563376,"external":4365018},"delay":0.09859300032258034},"load":{"requests":{"5601":{"total":0,"disconnects":0,"statusCodes":{}}},"responseTimes":{"5601":{"avg":null,"max":0}},"sockets":{"http":{"total":0},"https":{"total":0}}},"message":"memory: 111.2MB uptime: 0:07:33 load: [0.88 0.81 0.76] delay: 0.099"}

od-kibana | {"type":"log","@timestamp":"2021-07-29T14:05:42Z","tags":["debug","metrics"],"pid":1,"message":"Refreshing metrics"}

od-kibana | {"type":"ops","@timestamp":"2021-07-29T14:05:45Z","tags":[],"pid":1,"os":{"load":[0.8134765625,0.79541015625,0.75830078125],"mem":{"total":16304513024,"free":8023314432},"uptime":7493},"proc":{"uptime":458.002,"mem":{"rss":167256064,"heapTotal":127217664,"heapUsed":116808976,"external":4158387},"delay":0.09423300065100193},"load":{"requests":{"5601":{"total":0,"disconnects":0,"statusCodes":{}}},"responseTimes":{"5601":{"avg":null,"max":0}},"sockets":{"http":{"total":0},"https":{"total":0}}},"message":"memory: 111.4MB uptime: 0:07:38 load: [0.81 0.80 0.76] delay: 0.094"}

od-kibana | {"type":"log","@timestamp":"2021-07-29T14:05:46Z","tags":["error","plugins","opendistroReportsKibana"],"pid":1,"message":"Failed to generate report: TimeoutError: waiting for selector \".visEditor__content\" failed: timeout 60000ms exceeded"}

od-kibana | {"type":"log","@timestamp":"2021-07-29T14:05:46Z","tags":["error","plugins","opendistroReportsKibana"],"pid":1,"message":"{ TimeoutError: waiting for selector \".visEditor__content\" failed: timeout 60000ms exceeded\n    at new WaitTask (/usr/share/kibana/plugins/opendistroReportsKibana/node_modules/puppeteer-core/lib/DOMWorld.js:549:28)\n    at DOMWorld._waitForSelectorOrXPath (/usr/share/kibana/plugins/opendistroReportsKibana/node_modules/puppeteer-core/lib/DOMWorld.js:478:22)\n    at DOMWorld.waitForSelector (/usr/share/kibana/plugins/opendistroReportsKibana/node_modules/puppeteer-core/lib/DOMWorld.js:432:17)\n    at Frame.waitForSelector (/usr/share/kibana/plugins/opendistroReportsKibana/node_modules/puppeteer-core/lib/FrameManager.js:627:47)\n    at Frame.<anonymous> (/usr/share/kibana/plugins/opendistroReportsKibana/node_modules/puppeteer-core/lib/helper.js:112:23)\n    at Page.waitForSelector (/usr/share/kibana/plugins/opendistroReportsKibana/node_modules/puppeteer-core/lib/Page.js:1095:29)\n    at createVisualReport (/usr/share/kibana/plugins/opendistroReportsKibana/server/routes/utils/visual_report/visualReportHelper.ts:134:18) name: 'TimeoutError' }"}

od-kibana | {"type":"error","@timestamp":"2021-07-29T14:04:43Z","tags":[],"pid":1,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Internal Server Error\n    at HapiResponseAdapter.toError (/usr/share/kibana/src/core/server/http/router/response_adapter.js:132:19)\n    at HapiResponseAdapter.toHapiResponse (/usr/share/kibana/src/core/server/http/router/response_adapter.js:86:19)\n    at HapiResponseAdapter.handle (/usr/share/kibana/src/core/server/http/router/response_adapter.js:81:17)\n    at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:164:34)"},"url":{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":null,"search":"?timezone=America/New_York","query":{"timezone":"America/New_York"},"pathname":"/api/reporting/generateReport","path":"/api/reporting/generateReport?timezone=America/New_York","href":"/api/reporting/generateReport?timezone=America/New_York"},"message":"Internal Server Error"}

od-kibana | {"type":"response","@timestamp":"2021-07-29T14:04:43Z","tags":[],"pid":1,"method":"post","statusCode":500,"req":{"url":"/api/reporting/generateReport?timezone=America%2FNew_York","method":"post","headers":{"host":"xx.xxx.xxx.xxx","sec-ch-ua":"\" Not;A Brand\";v=\"99\", \"Google Chrome\";v=\"91\", \"Chromium\";v=\"91\"","pragma":"no-cache","accept-language":"en-US,en;q=0.9,zh-CN;q=0.8,zh;q=0.7,zh-TW;q=0.6","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36","content-type":"application/json","accept":"*/*","kbn-version":"7.10.2","origin":"https://xx.xxx.xxx.xxx","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://xx.xxx.xxx.xxx/kibana/app/visualize","accept-encoding":"gzip, deflate, br","oidc_claim_sub":"8n91kaou-my67-9018-18aj-aosu981773baj","oidc_claim_email_verified":"1","oidc_claim_roles":"default-roles-myrealm,offline_access,test_role,uma_authorization","oidc_claim_name":"Fname Lname","oidc_claim_preferred_username":"flname","oidc_claim_given_name":"Fname","oidc_claim_family_name":"Lname","oidc_claim_email":"fname.lname@group.org.com","oidc_claim_exp":"1627566733","oidc_claim_iat":"1627566433","oidc_claim_auth_time":"1627566433","oidc_claim_jti":"59cc1784-2b29-4d6b-bdb2-f3f94b9f43a3","oidc_claim_iss":"https://xx.xxx.xxx.xxx/auth/realms/myrealm","oidc_claim_aud":"apache-proxy","oidc_claim_typ":"ID","oidc_claim_azp":"apache-proxy","oidc_claim_nonce":"kJIez6KZd08lurC7FjmQoNsusFim2Ull8GvDyBaYU5E","oidc_claim_session_state":"199iurnla-9987-35c9-7n1j-0d7662k13oq444","oidc_claim_at_hash":"4OzdiYhLkFKrg2gwtF7wcA","oidc_claim_acr":"1","oidc_access_token":"eyJ........","oidc_access_token_expires":"1627566733","x-proxy-user":"rlname","x-proxy-roles":"admin","x-forwarded-for":"xxx.xx.xxx.xx","x-forwarded-host":"xx.xxx.xxx.xxx","x-forwarded-server":"xx.x.x.xx","content-length":"871","connection":"Keep-Alive","securitytenant":""},"remoteAddress":"xxx.xx.x.x","userAgent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36","referer":"https://xx.xxx.xxx.xxx/kibana/app/visualize"},"res":{"statusCode":500,"responseTime":62251,"contentLength":9},"message":"POST /api/reporting/generateReport?timezone=America%2FNew_York 500 62251ms - 9.0B"}
4

@anirudha Here are my configuration files, in case that helps.

securityconfig/config.yml:

_meta:
  type: "config"
  config_version: 2

config:
  dynamic:
    http:
      anonymous_auth_enabled: false
      xff:
        enabled: true
        #internalProxies: '192\.168\.0\.10|192\.168\.0\.11' # regex pattern
        internalProxies: '.*' # trust all internal proxies, regex pattern
        remoteIpHeader:  'x-forwarded-for'
        ###### see https://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html for regex help
        ###### more information about XFF https://en.wikipedia.org/wiki/X-Forwarded-For
        ###### and here https://tools.ietf.org/html/rfc7239
        ###### and https://tomcat.apache.org/tomcat-8.0-doc/config/valve.html#Remote_IP_Valve
    authc:
      basic_internal_auth_domain:
        description: "Authenticate via HTTP Basic against internal users database"
        http_enabled: true
        transport_enabled: true
        order: 0
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          type: intern
      
      proxy_auth_domain:
        http_enabled: true
        transport_enabled: true
        order: 1
        http_authenticator:
          type: proxy
          challenge: false
          config:
            user_header: "x-proxy-user"
            roles_header: "x-proxy-roles"
        authentication_backend:
          type: noop

kibana.yml:

logging.verbose: true

server.name: kibana
server.host: "0"
elasticsearch.hosts: https://localhost:9200
elasticsearch.ssl.verificationMode: certificate
elasticsearch.ssl.certificateAuthorities: ["/usr/share/kibana/config/root-ca.pem"]
elasticsearch.username: xxxxxxxx
elasticsearch.password: xxxxxxxx
elasticsearch.requestHeadersWhitelist: ["securitytenant","Authorization","x-forwarded-for","x-proxy-user","x-proxy-roles"]

opendistro_security.multitenancy.enabled: true
opendistro_security.multitenancy.tenants.preferred: ["Private", "Global"]
opendistro_security.readonly_mode.roles: ["kibana_read_only"]

# Use this setting if you are running kibana without https
opendistro_security.cookie.secure: false

newsfeed.enabled: false
telemetry.optIn: false
telemetry.enabled: false
security.showInsecureClusterWarning: false
# To configure a WMS map server for use with Kibana, see:
# https://opendistro.github.io/for-elasticsearch-docs/docs/kibana/maptiles/
map.includeElasticMapsService: false


opendistro_security.auth.type: "proxy"
opendistro_security.proxycache.user_header: "x-proxy-user"
opendistro_security.proxycache.roles_header: "x-proxy-roles"

elasticsearch.yml:

cluster.name: "docker-cluster"
network.host: 0.0.0.0

# # minimum_master_nodes need to be explicitly set when bound on a public IP
# # set to 1 to allow single node clusters
# # Details: https://github.com/elastic/elasticsearch/pull/17288
# discovery.zen.minimum_master_nodes: 1

# # Breaking change in 7.0
# # https://www.elastic.co/guide/en/elasticsearch/reference/7.0/breaking-changes-7.0.html#breaking_70_discovery_changes
# cluster.initial_master_nodes: 
#    - elasticsearch1
#    - docker-test-node-1 

opendistro_security.ssl.transport.pemcert_filepath: node.pem
opendistro_security.ssl.transport.pemkey_filepath: node-key.pem
opendistro_security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
opendistro_security.ssl.transport.enforce_hostname_verification: false
opendistro_security.ssl.http.enabled: true
opendistro_security.ssl.http.pemcert_filepath: node.pem
opendistro_security.ssl.http.pemkey_filepath: node-key.pem
opendistro_security.ssl.http.pemtrustedcas_filepath: root-ca.pem
opendistro_security.allow_unsafe_democertificates: false
opendistro_security.allow_default_init_securityindex: true
opendistro_security.authcz.admin_dn:
  - 'CN=xxxxx,O=xxxxx,L=xxxxx,ST=xxxxx,C=US'

opendistro_security.audit.type: internal_elasticsearch
opendistro_security.enable_snapshot_restore_privilege: true
opendistro_security.check_snapshot_restore_write_privileges: true
opendistro_security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
cluster.routing.allocation.disk.threshold_enabled: false
opendistro_security.audit.config.disabled_rest_categories: NONE
opendistro_security.audit.config.disabled_transport_categories: NONE
opendistro_security.nodes_dn:
  - 'CN=xxxxx,O=xxxxx,L=xxxxx,ST=xxxxx,C=US'
5

Maybe you can try to set your kibana’s ( or your proxy) host ip in kibana.yml rather than 0.0.0.0

server.host: xxx.xxx.xxx.xxx

6

@mudboyzh Thanks for the reply. I gave that a shot but no luck. I’m getting the following error when attempting to use kibana or proxy host ip in kibana.yml:

Error: listen EADDRNOTAVAIL: address not available xx.xxx.xxx.xxx:5601\n

This may be unrelated or not an issue, but it appears that the kibana basepath “/kibana” is not getting appended to the address when I modify the server.host to point to an ip address other than “0”.

7

I’ve reverted to the configuration settings in my original post.

In addition to the 401 error in the Kibana logs that I noted in my original post, I also see the following error in the Chrome dev console when I attempt to create a PDF/PNG report:

Cannot read property 'includes' of undefined

kibana_reporting_error
kibana_reporting_error1754×655 139 KB

This appears to be the exact same error (word-for-word in the stacktrace) on the front end as issue 5618.

However, in the case of issue 5618, it appears that they couldn’t even create a CSV report, whereas my issue occurs only with PDF/PNG reports. I can create and download CSV reports from the Discover tab or from saved searches in the Reporting tab as expected without any errors in Chrome or in the Kibana logs.

8

Hi Ryan,
Did you fix or found a workaround to your issue? I observed the same error when I tried to use ODFE 1.13.2 (with official docker image) I cannot generated a pdf/png report.
Regards

9

Hi @pascal. I was not ever able to resolve this in ODFE 1.13.2. However, I ended up resolving my issue by upgrading to OpenSearch 1.2.4 and doing two additional things:

  1. I had large headers that weren’t being passed through by my reverse-proxy (nginx), and I had to add the line large_client_header_buffers 4 32k; to my server block configuration in nginx.conf. If you are using keycloak with openid connect, you may also need to add set $session_name nginx_session; in your server configuration.
  2. If you are not using a reverse-proxy or keycloak and still seeing the same issue, check your opensearch_dashboards.yml (or kibana.yml) for a line like opensearchDashboards.index: ".opensearch_dashboards", and comment it out.

Here is a working setup for OpenSearch 1.2.4 and OpenSearch Dashboards 1.2.0 in Docker with Keycloak and an NginX reverse proxy - https://github.com/rylativity/opensearch-nginx-keycloak