Cannot Generate PDF / PNG Reports in ODFE 1.13.2 (CSV Works Fine)

I am attempting to generate PDF and PNG reports from the Kibana UI. When I click Download PDF/PNG in the UI, I immediately see a 401 response in the Kibana logs for the endpoint /app/visualize?security_tenant=global (pasted below), and 1 minute later I get a timeout and 500 error. I see the exact same behavior when I use the private tenant, too.

However, I have no problem generating and downloading CSV reports with both the global and private tenants, so I am confused by the 401 error and am not sure where it is coming from.

I am using OpenDistro Elastic & Kibana versions 1.13.2 in docker containers behind an Apache2 reverse proxy. I also have Proxy Auth enabled.
Please see below for the log output (starts at initial download request and ends after timeout; includes 401 error).

od-kibana           | {"type":"response","@timestamp":"2021-07-27T14:04:30Z","tags":[],"pid":1,"method":"get","statusCode":200,"req":{"url":"/api/v1/multitenancy/tenant","method":"get","headers":{"host":"xx.xxx.xxx.xxx","sec-ch-ua":"\" Not;A Brand\";v=\"99\", \"Google Chrome\";v=\"91\", \"Chromium\";v=\"91\"","pragma":"no-cache","accept-language":"en-US,en;q=0.9,zh-CN;q=0.8,zh;q=0.7,zh-TW;q=0.6","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36","content-type":"application/json","accept":"*/*","kbn-version":"7.10.2","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://xx.xxx.xxx.xxx/kibana/app/visualize","accept-encoding":"gzip, deflate, br","oidc_claim_sub":"8adjkf-wip98-1242-90aa-dfjai082","oidc_claim_email_verified":"1","oidc_claim_roles":"default-roles-myrealm,offline_access,test_role,uma_authorization","oidc_claim_name":"Fname Lname","oidc_claim_preferred_username":"flname","oidc_claim_given_name":"Fname","oidc_claim_family_name":"Lname","oidc_claim_email":"Fname.Lname@group.domain.com","oidc_claim_exp":"1627393756","oidc_claim_iat":"1627393456","oidc_claim_auth_time":"1627393257","oidc_claim_jti":"b8908a33-ab9q-183a-a72b-j8quyew88123p12","oidc_claim_iss":"https://xx.xxx.xxx.xxx/auth/realms/myrealm","oidc_claim_aud":"apache-proxy","oidc_claim_typ":"ID","oidc_claim_azp":"apache-proxy","oidc_claim_nonce":"bALqrSKbe2QcU_9yyTUDVKet77cqCBQmXSr-LwnvofQ","oidc_claim_session_state":"q583nf54-ad64-10q9-ac36-b28a90402780","oidc_claim_at_hash":"q5qikAOIJ1281alkKIQPlQAui","oidc_claim_acr":"0","oidc_access_token":"eyJ................","oidc_access_token_expires":"1627393756","x-proxy-user":"flname","x-proxy-roles":"admin","x-forwarded-for":"xxx.xx.xxx.xxx","x-forwarded-host":"xx.xxx.xxx.xxx","x-forwarded-server":"10.0.1.45","connection":"Keep-Alive","securitytenant":""},"remoteAddress":"xxx.xx.xxx.xxx","userAgent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36","referer":"https://xx.xxx.xxx.xxx/kibana/app/visualize"},"res":{"statusCode":200,"responseTime":10,"contentLength":9},"message":"GET /api/v1/multitenancy/tenant 200 10ms - 9.0B"}

od-kibana           | {"type":"log","@timestamp":"2021-07-27T14:04:30Z","tags":["info","plugins","opendistroReportsKibana"],"pid":1,"message":"domain enables security, use session cookie to access"}

od-kibana           | {"type":"log","@timestamp":"2021-07-27T14:04:30Z","tags":["info","plugins","opendistroReportsKibana"],"pid":1,"message":"original queryUrl http://0:5601/kibana/app/visualize?security_tenant=global#/edit/4d739760-eee3-11eb-8f1e-1991bfa88630?type=markdown&_g=(filters:!(),refreshInterval:(pause:!f,value:900000),time:(from:'2021-07-26T14:04:29.203Z',to:'2021-07-27T14:04:29.203Z'))&_a=(filters:!(),linked:!f,query:(language:kuery,query:''),uiState:(),vis:(aggs:!(),params:(fontSize:12,markdown:'Test%20Markdown',openLinksInNewTab:!f),title:Test,type:markdown))"}

od-kibana           | {"type":"response","@timestamp":"2021-07-27T14:04:30Z","tags":[],"pid":1,"method":"get","statusCode":401,"req":{"url":"/app/visualize?security_tenant=global","method":"get","headers":{"host":"0.0.0.0:5601","connection":"keep-alive","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/77.0.3844.0 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3","accept-encoding":"gzip, deflate","accept-language":"en-US"},"remoteAddress":"127.0.0.1","userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/77.0.3844.0 Safari/537.36"},"res":{"statusCode":401,"responseTime":2,"contentLength":9},"message":"GET /app/visualize?security_tenant=global 401 2ms - 9.0B"}

od-kibana           | {"type":"log","@timestamp":"2021-07-27T14:04:31Z","tags":["info","plugins","opendistroReportsKibana"],"pid":1,"message":"page url http://0.0.0.0:5601/kibana/app/visualize?security_tenant=global#/edit/4d739760-eee3-11eb-8f1e-1991bfa88630?type=markdown&_g=(filters:!(),refreshInterval:(pause:!f,value:900000),time:(from:'2021-07-26T14:04:29.203Z',to:'2021-07-27T14:04:29.203Z'))&_a=(filters:!(),linked:!f,query:(language:kuery,query:''),uiState:(),vis:(aggs:!(),params:(fontSize:12,markdown:'Test%20Markdown',openLinksInNewTab:!f),title:Test,type:markdown))"}

od-kibana           | {"type":"log","@timestamp":"2021-07-27T14:05:32Z","tags":["error","plugins","opendistroReportsKibana"],"pid":1,"message":"Failed to generate report: TimeoutError: waiting for selector \".visEditor__content\" failed: timeout 60000ms exceeded"}

od-kibana           | {"type":"log","@timestamp":"2021-07-27T14:05:32Z","tags":["error","plugins","opendistroReportsKibana"],"pid":1,"message":"{ TimeoutError: waiting for selector \".visEditor__content\" failed: timeout 60000ms exceeded\n    at new WaitTask (/usr/share/kibana/plugins/opendistroReportsKibana/node_modules/puppeteer-core/lib/DOMWorld.js:549:28)\n    at DOMWorld._waitForSelectorOrXPath (/usr/share/kibana/plugins/opendistroReportsKibana/node_modules/puppeteer-core/lib/DOMWorld.js:478:22)\n    at DOMWorld.waitForSelector (/usr/share/kibana/plugins/opendistroReportsKibana/node_modules/puppeteer-core/lib/DOMWorld.js:432:17)\n    at Frame.waitForSelector (/usr/share/kibana/plugins/opendistroReportsKibana/node_modules/puppeteer-core/lib/FrameManager.js:627:47)\n    at Frame.<anonymous> (/usr/share/kibana/plugins/opendistroReportsKibana/node_modules/puppeteer-core/lib/helper.js:112:23)\n    at Page.waitForSelector (/usr/share/kibana/plugins/opendistroReportsKibana/node_modules/puppeteer-core/lib/Page.js:1095:29)\n    at createVisualReport (/usr/share/kibana/plugins/opendistroReportsKibana/server/routes/utils/visual_report/visualReportHelper.ts:134:18) name: 'TimeoutError' }"}

od-kibana           | {"type":"error","@timestamp":"2021-07-27T14:04:30Z","tags":[],"pid":1,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Internal Server Error\n    at HapiResponseAdapter.toError (/usr/share/kibana/src/core/server/http/router/response_adapter.js:132:19)\n    at HapiResponseAdapter.toHapiResponse (/usr/share/kibana/src/core/server/http/router/response_adapter.js:86:19)\n    at HapiResponseAdapter.handle (/usr/share/kibana/src/core/server/http/router/response_adapter.js:81:17)\n    at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:164:34)"},"url":{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":null,"search":"?timezone=America/New_York","query":{"timezone":"America/New_York"},"pathname":"/api/reporting/generateReport","path":"/api/reporting/generateReport?timezone=America/New_York","href":"/api/reporting/generateReport?timezone=America/New_York"},"message":"Internal Server Error"}

od-kibana           | {"type":"response","@timestamp":"2021-07-27T14:04:30Z","tags":[],"pid":1,"method":"post","statusCode":500,"req":{"url":"/api/reporting/generateReport?timezone=America%2FNew_York","method":"post","headers":{"host":"xx.xxx.xxx.xxx","sec-ch-ua":"\" Not;A Brand\";v=\"99\", \"Google Chrome\";v=\"91\", \"Chromium\";v=\"91\"","pragma":"no-cache","accept-language":"en-US,en;q=0.9,zh-CN;q=0.8,zh;q=0.7,zh-TW;q=0.6","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36","content-type":"application/json","accept":"*/*","kbn-version":"7.10.2","origin":"https://xx.xxx.xxx.xxx","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://xx.xxx.xxx.xxx/kibana/app/visualize","accept-encoding":"gzip, deflate, br","oidc_claim_sub":"8adjkf-wip98-1242-90aa-dfjai082","oidc_claim_email_verified":"1","oidc_claim_roles":"default-roles-myrealm,offline_access,test_role,uma_authorization","oidc_claim_name":"Fname Lname","oidc_claim_preferred_username":"flname","oidc_claim_given_name":"Fname","oidc_claim_family_name":"Lname","oidc_claim_email":"Fname.Lname@group.domain.com","oidc_claim_exp":"1627393756","oidc_claim_iat":"1627393456","oidc_claim_auth_time":"1627393257","oidc_claim_jti":"b8908a33-ab9q-183a-a72b-j8quyew88123p12","oidc_claim_iss":"https://xx.xxx.xxx.xxx/auth/realms/myrealm","oidc_claim_aud":"apache-proxy","oidc_claim_typ":"ID","oidc_claim_azp":"apache-proxy","oidc_claim_nonce":"bALqrSKbe2QcU_9yyTUDVKet77cqCBQmXSr-LwnvofQ","oidc_claim_session_state":"q583nf54-ad64-10q9-ac36-b28a90402780","oidc_claim_at_hash":"q5qikAOIJ1281alkKIQPlQAui","oidc_claim_acr":"0","oidc_access_token":"eyJ................","oidc_access_token_expires":"1627393756","x-proxy-user":"flname","x-proxy-roles":"admin","x-forwarded-for":"xxx.xx.xxx.xxx","x-forwarded-host":"xx.xxx.xxx.xxx","x-forwarded-server":"10.0.1.45","content-length":"895","connection":"Keep-Alive","securitytenant":""},"remoteAddress":"xxx.xx.xxx.xxx","userAgent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36","referer":"https://xx.xxx.xxx.xxx/kibana/app/visualize"},"res":{"statusCode":500,"responseTime":62131,"contentLength":9},"message":"POST /api/reporting/generateReport?timezone=America%2FNew_York 500 62131ms - 9.0B"}

This is definitely related, can you send more information eg. Any errors from Kibanaog with increased verbosity. Thanks

@anirudha I added “logging.verbose: true” to my kibana.yml, but none of the additional output (pasted at the bottom of this comment) appears to be related to the 401 error.

I also tried adding the following to my elasticsearch log4j2.properties

logger.opendistro_security.name = com.amazon.opendistroforelasticsearch.security
logger.opendistro_security.level = trace
logger.opendistro_security.appenderRef.rolling.ref = rolling
logger.opendistro_security.appenderRef.rolling_old.ref = rolling_old
logger.opendistro_security.additivity = false

Is there another way to increase logging output that may provide additional detail on where the 401 is coming from?

od-kibana | {"type":"log","@timestamp":"2021-07-29T14:04:42Z","tags":["debug","metrics"],"pid":1,"message":"Refreshing metrics"}

od-kibana | {"type":"response","@timestamp":"2021-07-29T14:04:43Z","tags":[],"pid":1,"method":"get","statusCode":200,"req":{"url":"/api/v1/multitenancy/tenant","method":"get","headers":{"host":"xx.xxx.xxx.xxx","sec-ch-ua":"\" Not;A Brand\";v=\"99\", \"Google Chrome\";v=\"91\", \"Chromium\";v=\"91\"","pragma":"no-cache","accept-language":"en-US,en;q=0.9,zh-CN;q=0.8,zh;q=0.7,zh-TW;q=0.6","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36","content-type":"application/json","accept":"*/*","kbn-version":"7.10.2","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://xx.xxx.xxx.xxx/kibana/app/visualize","accept-encoding":"gzip, deflate, br","oidc_claim_sub":"8n91kaou-my67-9018-18aj-aosu981773baj","oidc_claim_email_verified":"1","oidc_claim_roles":"default-roles-myrealm,offline_access,test_role,uma_authorization","oidc_claim_name":"Fname Lname","oidc_claim_preferred_username":"rlname","oidc_claim_given_name":"Fname","oidc_claim_family_name":"Lname","oidc_claim_email":"fname.lname@group.org.com","oidc_claim_exp":"1627566733","oidc_claim_iat":"1627566433","oidc_claim_auth_time":"1627566433","oidc_claim_jti":"59cc1784-2b29-4d6b-bdb2-f3f94b9f43a3","oidc_claim_iss":"https://xx.xxx.xxx.xxx/auth/realms/myrealm","oidc_claim_aud":"apache-proxy","oidc_claim_typ":"ID","oidc_claim_azp":"apache-proxy","oidc_claim_nonce":"kJIez6KZd08lurC7FjmQoNsusFim2Ull8GvDyBaYU5E","oidc_claim_session_state":"199iurnla-9987-35c9-7n1j-0d7662k13oq444","oidc_claim_at_hash":"4OzdiYhLkFKrg2gwtF7wcA","oidc_claim_acr":"1","oidc_access_token":"eyJ........","oidc_access_token_expires":"1627566733","x-proxy-user":"rlname","x-proxy-roles":"admin","x-forwarded-for":"xxx.xx.xxx.xx","x-forwarded-host":"xx.xxx.xxx.xxx","x-forwarded-server":"xx.x.x.xx","connection":"Keep-Alive","securitytenant":""},"remoteAddress":"xxx.xx.x.x","userAgent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36","referer":"https://xx.xxx.xxx.xxx/kibana/app/visualize"},"res":{"statusCode":200,"responseTime":5,"contentLength":9},"message":"GET /api/v1/multitenancy/tenant 200 5ms - 9.0B"}

od-kibana | {"type":"log","@timestamp":"2021-07-29T14:04:44Z","tags":["info","plugins","opendistroReportsKibana"],"pid":1,"message":"domain enables security, use session cookie to access"}

od-kibana | {"type":"log","@timestamp":"2021-07-29T14:04:44Z","tags":["info","plugins","opendistroReportsKibana"],"pid":1,"message":"original queryUrl http://0:5601/kibana/app/visualize?security_tenant=global#/edit/4d739760-eee3-11eb-8f1e-1991bfa88630?_g=(filters:!(),refreshInterval:(pause:!f,value:900000),time:(from:'2021-07-28T14:04:42.671Z',to:'2021-07-29T14:04:42.671Z'))&_a=(filters:!(),linked:!f,query:(language:kuery,query:''),uiState:(),vis:(aggs:!(),params:(fontSize:12,markdown:'Test%20Markdown',openLinksInNewTab:!f),title:Test,type:markdown))"}

od-kibana | {"type":"response","@timestamp":"2021-07-29T14:04:44Z","tags":[],"pid":1,"method":"get","statusCode":401,"req":{"url":"/app/visualize?security_tenant=global","method":"get","headers":{"host":"0.0.0.0:5601","connection":"keep-alive","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/77.0.3844.0 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3","accept-encoding":"gzip, deflate","accept-language":"en-US"},"remoteAddress":"127.0.0.1","userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/77.0.3844.0 Safari/537.36"},"res":{"statusCode":401,"responseTime":3,"contentLength":9},"message":"GET /app/visualize?security_tenant=global 401 3ms - 9.0B"}

od-kibana | {"type":"ops","@timestamp":"2021-07-29T14:04:44Z","tags":[],"pid":1,"os":{"load":[0.8193359375,0.80322265625,0.7607421875],"mem":{"total":16304513024,"free":7904096256},"uptime":7433},"proc":{"uptime":397.984,"mem":{"rss":166649856,"heapTotal":126693376,"heapUsed":116199896,"external":3971302},"delay":0.12289100047200918},"load":{"requests":{"5601":{"total":8,"disconnects":0,"statusCodes":{"200":6,"401":1}}},"responseTimes":{"5601":{"avg":5.857142857142857,"max":11}},"sockets":{"http":{"total":0},"https":{"total":0}}},"message":"memory: 110.8MB uptime: 0:06:38 load: [0.82 0.80 0.76] delay: 0.123"}

od-kibana | {"type":"log","@timestamp":"2021-07-29T14:04:45Z","tags":["info","plugins","opendistroReportsKibana"],"pid":1,"message":"page url http://0.0.0.0:5601/kibana/app/visualize?security_tenant=global#/edit/4d739760-eee3-11eb-8f1e-1991bfa88630?_g=(filters:!(),refreshInterval:(pause:!f,value:900000),time:(from:'2021-07-28T14:04:42.671Z',to:'2021-07-29T14:04:42.671Z'))&_a=(filters:!(),linked:!f,query:(language:kuery,query:''),uiState:(),vis:(aggs:!(),params:(fontSize:12,markdown:'Test%20Markdown',openLinksInNewTab:!f),title:Test,type:markdown))"}

od-kibana | {"type":"response","@timestamp":"2021-07-29T14:04:44Z","tags":[],"pid":1,"method":"post","statusCode":200,"req":{"url":"/api/ui_metric/report","method":"post","headers":{"host":"xx.xxx.xxx.xxx","sec-ch-ua":"\" Not;A Brand\";v=\"99\", \"Google Chrome\";v=\"91\", \"Chromium\";v=\"91\"","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36","kbn-version":"7.10.2","content-type":"application/json","accept":"*/*","origin":"https://xx.xxx.xxx.xxx","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://xx.xxx.xxx.xxx/kibana/app/visualize","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9","oidc_claim_sub":"8n91kaou-my67-9018-18aj-aosu981773baj","oidc_claim_email_verified":"1","oidc_claim_roles":"default-roles-myrealm,offline_access,test_role,uma_authorization","oidc_claim_name":"Fname Lname","oidc_claim_preferred_username":"rlname","oidc_claim_given_name":"Fname","oidc_claim_family_name":"Lname","oidc_claim_email":"fname.lname@group.org.com","oidc_claim_exp":"1627566733","oidc_claim_iat":"1627566433","oidc_claim_auth_time":"1627566433","oidc_claim_jti":"59cc1784-2b29-4d6b-bdb2-f3f94b9f43a3","oidc_claim_iss":"https://xx.xxx.xxx.xxx/auth/realms/myrealm","oidc_claim_aud":"apache-proxy","oidc_claim_typ":"ID","oidc_claim_azp":"apache-proxy","oidc_claim_nonce":"kJIez6KZd08lurC7FjmQoNsusFim2Ull8GvDyBaYU5E","oidc_claim_session_state":"199iurnla-9987-35c9-7n1j-0d7662k13oq444","oidc_claim_at_hash":"4OzdiYhLkFKrg2gwtF7wcA","oidc_claim_acr":"1","oidc_access_token":"eyJ........","oidc_access_token_expires":"1627566733","x-proxy-user":"rlname","x-proxy-roles":"admin","x-forwarded-for":"xxx.xx.xxx.xx","x-forwarded-host":"xx.xxx.xxx.xxx","x-forwarded-server":"xx.x.x.xx","content-length":"169","connection":"Keep-Alive","securitytenant":""},"remoteAddress":"xxx.xx.x.x","userAgent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36","referer":"https://xx.xxx.xxx.xxx/kibana/app/visualize"},"res":{"statusCode":200,"responseTime":152,"contentLength":9},"message":"POST /api/ui_metric/report 200 152ms - 9.0B"}

od-kibana | {"type":"log","@timestamp":"2021-07-29T14:04:47Z","tags":["debug","metrics"],"pid":1,"message":"Refreshing metrics"}

od-kibana | {"type":"ops","@timestamp":"2021-07-29T14:04:49Z","tags":[],"pid":1,"os":{"load":[0.75341796875,0.78955078125,0.75634765625],"mem":{"total":16304513024,"free":8088416256},"uptime":7438},"proc":{"uptime":402.986,"mem":{"rss":166371328,"heapTotal":126693376,"heapUsed":115955488,"external":4101906},"delay":0.0932820001617074},"load":{"requests":{"5601":{"total":0,"disconnects":0,"statusCodes":{"200":1}}},"responseTimes":{"5601":{"avg":152,"max":152}},"sockets":{"http":{"total":0},"https":{"total":0}}},"message":"memory: 110.6MB uptime: 0:06:43 load: [0.75 0.79 0.76] delay: 0.093"}

od-kibana | {"type":"log","@timestamp":"2021-07-29T14:04:52Z","tags":["debug","metrics"],"pid":1,"message":"Refreshing metrics"}

od-kibana | {"type":"ops","@timestamp":"2021-07-29T14:04:54Z","tags":[],"pid":1,"os":{"load":[0.69287109375,0.7763671875,0.751953125],"mem":{"total":16304513024,"free":8089174016},"uptime":7443},"proc":{"uptime":407.987,"mem":{"rss":166612992,"heapTotal":126693376,"heapUsed":116202512,"external":4241598},"delay":0.09310999978333712},"load":{"requests":{"5601":{"total":0,"disconnects":0,"statusCodes":{}}},"responseTimes":{"5601":{"avg":null,"max":0}},"sockets":{"http":{"total":0},"https":{"total":0}}},"message":"memory: 110.8MB uptime: 0:06:48 load: [0.69 0.78 0.75] delay: 0.093"}

od-kibana | {"type":"log","@timestamp":"2021-07-29T14:04:57Z","tags":["debug","metrics"],"pid":1,"message":"Refreshing metrics"}

od-kibana | {"type":"ops","@timestamp":"2021-07-29T14:04:59Z","tags":[],"pid":1,"os":{"load":[0.7978515625,0.796875,0.7587890625],"mem":{"total":16304513024,"free":8088657920},"uptime":7448},"proc":{"uptime":412.989,"mem":{"rss":166850560,"heapTotal":126693376,"heapUsed":116436416,"external":4373098},"delay":0.09290800057351589},"load":{"requests":{"5601":{"total":0,"disconnects":0,"statusCodes":{}}},"responseTimes":{"5601":{"avg":null,"max":0}},"sockets":{"http":{"total":0},"https":{"total":0}}},"message":"memory: 111.0MB uptime: 0:06:53 load: [0.80 0.80 0.76] delay: 0.093"}

od-kibana | {"type":"log","@timestamp":"2021-07-29T14:05:02Z","tags":["debug","metrics"],"pid":1,"message":"Refreshing metrics"}

od-kibana | {"type":"ops","@timestamp":"2021-07-29T14:05:04Z","tags":[],"pid":1,"os":{"load":[0.814453125,0.80029296875,0.76025390625],"mem":{"total":16304513024,"free":8089948160},"uptime":7453},"proc":{"uptime":417.991,"mem":{"rss":167133184,"heapTotal":126693376,"heapUsed":116670168,"external":4504598},"delay":0.09504599962383509},"load":{"requests":{"5601":{"total":0,"disconnects":0,"statusCodes":{}}},"responseTimes":{"5601":{"avg":null,"max":0}},"sockets":{"http":{"total":0},"https":{"total":0}}},"message":"memory: 111.3MB uptime: 0:06:58 load: [0.81 0.80 0.76] delay: 0.095"}

od-kibana | {"type":"log","@timestamp":"2021-07-29T14:05:07Z","tags":["debug","metrics"],"pid":1,"message":"Refreshing metrics"}

od-kibana | {"type":"ops","@timestamp":"2021-07-29T14:05:09Z","tags":[],"pid":1,"os":{"load":[0.7490234375,0.78662109375,0.755859375],"mem":{"total":16304513024,"free":8039424000},"uptime":7458},"proc":{"uptime":422.991,"mem":{"rss":166412288,"heapTotal":127217664,"heapUsed":116026952,"external":4101962},"delay":0.09433500003069639},"load":{"requests":{"5601":{"total":0,"disconnects":0,"statusCodes":{}}},"responseTimes":{"5601":{"avg":null,"max":0}},"sockets":{"http":{"total":0},"https":{"total":0}}},"message":"memory: 110.7MB uptime: 0:07:03 load: [0.75 0.79 0.76] delay: 0.094"}

od-kibana | {"type":"log","@timestamp":"2021-07-29T14:05:12Z","tags":["debug","metrics"],"pid":1,"message":"Refreshing metrics"}

od-kibana | {"type":"ops","@timestamp":"2021-07-29T14:05:15Z","tags":[],"pid":1,"os":{"load":[0.68896484375,0.7734375,0.75146484375],"mem":{"total":16304513024,"free":8089157632},"uptime":7463},"proc":{"uptime":427.993,"mem":{"rss":166727680,"heapTotal":127217664,"heapUsed":116272248,"external":4233462},"delay":0.08964299969375134},"load":{"requests":{"5601":{"total":0,"disconnects":0,"statusCodes":{}}},"responseTimes":{"5601":{"avg":null,"max":0}},"sockets":{"http":{"total":0},"https":{"total":0}}},"message":"memory: 110.9MB uptime: 0:07:08 load: [0.69 0.77 0.75] delay: 0.090"}

od-kibana | {"type":"log","@timestamp":"2021-07-29T14:05:17Z","tags":["debug","metrics"],"pid":1,"message":"Refreshing metrics"}

od-kibana | {"type":"ops","@timestamp":"2021-07-29T14:05:20Z","tags":[],"pid":1,"os":{"load":[0.6337890625,0.76025390625,0.7470703125],"mem":{"total":16304513024,"free":8018616320},"uptime":7468},"proc":{"uptime":432.995,"mem":{"rss":166916096,"heapTotal":127217664,"heapUsed":116505328,"external":4364962},"delay":0.16557299997657537},"load":{"requests":{"5601":{"total":0,"disconnects":0,"statusCodes":{}}},"responseTimes":{"5601":{"avg":null,"max":0}},"sockets":{"http":{"total":0},"https":{"total":0}}},"message":"memory: 111.1MB uptime: 0:07:13 load: [0.63 0.76 0.75] delay: 0.166"}

od-kibana | {"type":"log","@timestamp":"2021-07-29T14:05:22Z","tags":["debug","metrics"],"pid":1,"message":"Refreshing metrics"}

od-kibana | {"type":"ops","@timestamp":"2021-07-29T14:05:25Z","tags":[],"pid":1,"os":{"load":[0.66357421875,0.7646484375,0.74853515625],"mem":{"total":16304513024,"free":8088109056},"uptime":7473},"proc":{"uptime":437.996,"mem":{"rss":167096320,"heapTotal":127217664,"heapUsed":116738416,"external":4496462},"delay":0.10125199984759092},"load":{"requests":{"5601":{"total":0,"disconnects":0,"statusCodes":{}}},"responseTimes":{"5601":{"avg":null,"max":0}},"sockets":{"http":{"total":0},"https":{"total":0}}},"message":"memory: 111.3MB uptime: 0:07:18 load: [0.66 0.76 0.75] delay: 0.101"}

od-kibana | {"type":"log","@timestamp":"2021-07-29T14:05:27Z","tags":["debug","metrics"],"pid":1,"message":"Refreshing metrics"}

od-kibana | {"type":"ops","@timestamp":"2021-07-29T14:05:30Z","tags":[],"pid":1,"os":{"load":[0.6103515625,0.751953125,0.744140625],"mem":{"total":16304513024,"free":8090431488},"uptime":7478},"proc":{"uptime":442.998,"mem":{"rss":166510592,"heapTotal":127217664,"heapUsed":116097144,"external":4102018},"delay":0.09809799958020449},"load":{"requests":{"5601":{"total":0,"disconnects":0,"statusCodes":{}}},"responseTimes":{"5601":{"avg":null,"max":0}},"sockets":{"http":{"total":0},"https":{"total":0}}},"message":"memory: 110.7MB uptime: 0:07:23 load: [0.61 0.75 0.74] delay: 0.098"}

od-kibana | {"type":"log","@timestamp":"2021-07-29T14:05:32Z","tags":["debug","metrics"],"pid":1,"message":"Refreshing metrics"}

od-kibana | {"type":"ops","@timestamp":"2021-07-29T14:05:35Z","tags":[],"pid":1,"os":{"load":[0.9619140625,0.82275390625,0.76708984375],"mem":{"total":16304513024,"free":8045264896},"uptime":7483},"proc":{"uptime":448,"mem":{"rss":166768640,"heapTotal":127217664,"heapUsed":116330248,"external":4233518},"delay":0.09495500009506941},"load":{"requests":{"5601":{"total":0,"disconnects":0,"statusCodes":{}}},"responseTimes":{"5601":{"avg":null,"max":0}},"sockets":{"http":{"total":0},"https":{"total":0}}},"message":"memory: 110.9MB uptime: 0:07:28 load: [0.96 0.82 0.77] delay: 0.095"}

od-kibana | {"type":"log","@timestamp":"2021-07-29T14:05:37Z","tags":["debug","metrics"],"pid":1,"message":"Refreshing metrics"}

od-kibana | {"type":"ops","@timestamp":"2021-07-29T14:05:40Z","tags":[],"pid":1,"os":{"load":[0.884765625,0.80908203125,0.7626953125],"mem":{"total":16304513024,"free":8044232704},"uptime":7488},"proc":{"uptime":453.002,"mem":{"rss":166768640,"heapTotal":127217664,"heapUsed":116563376,"external":4365018},"delay":0.09859300032258034},"load":{"requests":{"5601":{"total":0,"disconnects":0,"statusCodes":{}}},"responseTimes":{"5601":{"avg":null,"max":0}},"sockets":{"http":{"total":0},"https":{"total":0}}},"message":"memory: 111.2MB uptime: 0:07:33 load: [0.88 0.81 0.76] delay: 0.099"}

od-kibana | {"type":"log","@timestamp":"2021-07-29T14:05:42Z","tags":["debug","metrics"],"pid":1,"message":"Refreshing metrics"}

od-kibana | {"type":"ops","@timestamp":"2021-07-29T14:05:45Z","tags":[],"pid":1,"os":{"load":[0.8134765625,0.79541015625,0.75830078125],"mem":{"total":16304513024,"free":8023314432},"uptime":7493},"proc":{"uptime":458.002,"mem":{"rss":167256064,"heapTotal":127217664,"heapUsed":116808976,"external":4158387},"delay":0.09423300065100193},"load":{"requests":{"5601":{"total":0,"disconnects":0,"statusCodes":{}}},"responseTimes":{"5601":{"avg":null,"max":0}},"sockets":{"http":{"total":0},"https":{"total":0}}},"message":"memory: 111.4MB uptime: 0:07:38 load: [0.81 0.80 0.76] delay: 0.094"}

od-kibana | {"type":"log","@timestamp":"2021-07-29T14:05:46Z","tags":["error","plugins","opendistroReportsKibana"],"pid":1,"message":"Failed to generate report: TimeoutError: waiting for selector \".visEditor__content\" failed: timeout 60000ms exceeded"}

od-kibana | {"type":"log","@timestamp":"2021-07-29T14:05:46Z","tags":["error","plugins","opendistroReportsKibana"],"pid":1,"message":"{ TimeoutError: waiting for selector \".visEditor__content\" failed: timeout 60000ms exceeded\n    at new WaitTask (/usr/share/kibana/plugins/opendistroReportsKibana/node_modules/puppeteer-core/lib/DOMWorld.js:549:28)\n    at DOMWorld._waitForSelectorOrXPath (/usr/share/kibana/plugins/opendistroReportsKibana/node_modules/puppeteer-core/lib/DOMWorld.js:478:22)\n    at DOMWorld.waitForSelector (/usr/share/kibana/plugins/opendistroReportsKibana/node_modules/puppeteer-core/lib/DOMWorld.js:432:17)\n    at Frame.waitForSelector (/usr/share/kibana/plugins/opendistroReportsKibana/node_modules/puppeteer-core/lib/FrameManager.js:627:47)\n    at Frame.<anonymous> (/usr/share/kibana/plugins/opendistroReportsKibana/node_modules/puppeteer-core/lib/helper.js:112:23)\n    at Page.waitForSelector (/usr/share/kibana/plugins/opendistroReportsKibana/node_modules/puppeteer-core/lib/Page.js:1095:29)\n    at createVisualReport (/usr/share/kibana/plugins/opendistroReportsKibana/server/routes/utils/visual_report/visualReportHelper.ts:134:18) name: 'TimeoutError' }"}

od-kibana | {"type":"error","@timestamp":"2021-07-29T14:04:43Z","tags":[],"pid":1,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Internal Server Error\n    at HapiResponseAdapter.toError (/usr/share/kibana/src/core/server/http/router/response_adapter.js:132:19)\n    at HapiResponseAdapter.toHapiResponse (/usr/share/kibana/src/core/server/http/router/response_adapter.js:86:19)\n    at HapiResponseAdapter.handle (/usr/share/kibana/src/core/server/http/router/response_adapter.js:81:17)\n    at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:164:34)"},"url":{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":null,"search":"?timezone=America/New_York","query":{"timezone":"America/New_York"},"pathname":"/api/reporting/generateReport","path":"/api/reporting/generateReport?timezone=America/New_York","href":"/api/reporting/generateReport?timezone=America/New_York"},"message":"Internal Server Error"}

od-kibana | {"type":"response","@timestamp":"2021-07-29T14:04:43Z","tags":[],"pid":1,"method":"post","statusCode":500,"req":{"url":"/api/reporting/generateReport?timezone=America%2FNew_York","method":"post","headers":{"host":"xx.xxx.xxx.xxx","sec-ch-ua":"\" Not;A Brand\";v=\"99\", \"Google Chrome\";v=\"91\", \"Chromium\";v=\"91\"","pragma":"no-cache","accept-language":"en-US,en;q=0.9,zh-CN;q=0.8,zh;q=0.7,zh-TW;q=0.6","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36","content-type":"application/json","accept":"*/*","kbn-version":"7.10.2","origin":"https://xx.xxx.xxx.xxx","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://xx.xxx.xxx.xxx/kibana/app/visualize","accept-encoding":"gzip, deflate, br","oidc_claim_sub":"8n91kaou-my67-9018-18aj-aosu981773baj","oidc_claim_email_verified":"1","oidc_claim_roles":"default-roles-myrealm,offline_access,test_role,uma_authorization","oidc_claim_name":"Fname Lname","oidc_claim_preferred_username":"flname","oidc_claim_given_name":"Fname","oidc_claim_family_name":"Lname","oidc_claim_email":"fname.lname@group.org.com","oidc_claim_exp":"1627566733","oidc_claim_iat":"1627566433","oidc_claim_auth_time":"1627566433","oidc_claim_jti":"59cc1784-2b29-4d6b-bdb2-f3f94b9f43a3","oidc_claim_iss":"https://xx.xxx.xxx.xxx/auth/realms/myrealm","oidc_claim_aud":"apache-proxy","oidc_claim_typ":"ID","oidc_claim_azp":"apache-proxy","oidc_claim_nonce":"kJIez6KZd08lurC7FjmQoNsusFim2Ull8GvDyBaYU5E","oidc_claim_session_state":"199iurnla-9987-35c9-7n1j-0d7662k13oq444","oidc_claim_at_hash":"4OzdiYhLkFKrg2gwtF7wcA","oidc_claim_acr":"1","oidc_access_token":"eyJ........","oidc_access_token_expires":"1627566733","x-proxy-user":"rlname","x-proxy-roles":"admin","x-forwarded-for":"xxx.xx.xxx.xx","x-forwarded-host":"xx.xxx.xxx.xxx","x-forwarded-server":"xx.x.x.xx","content-length":"871","connection":"Keep-Alive","securitytenant":""},"remoteAddress":"xxx.xx.x.x","userAgent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36","referer":"https://xx.xxx.xxx.xxx/kibana/app/visualize"},"res":{"statusCode":500,"responseTime":62251,"contentLength":9},"message":"POST /api/reporting/generateReport?timezone=America%2FNew_York 500 62251ms - 9.0B"}

@anirudha Here are my configuration files, in case that helps.

securityconfig/config.yml:

_meta:
  type: "config"
  config_version: 2

config:
  dynamic:
    http:
      anonymous_auth_enabled: false
      xff:
        enabled: true
        #internalProxies: '192\.168\.0\.10|192\.168\.0\.11' # regex pattern
        internalProxies: '.*' # trust all internal proxies, regex pattern
        remoteIpHeader:  'x-forwarded-for'
        ###### see https://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html for regex help
        ###### more information about XFF https://en.wikipedia.org/wiki/X-Forwarded-For
        ###### and here https://tools.ietf.org/html/rfc7239
        ###### and https://tomcat.apache.org/tomcat-8.0-doc/config/valve.html#Remote_IP_Valve
    authc:
      basic_internal_auth_domain:
        description: "Authenticate via HTTP Basic against internal users database"
        http_enabled: true
        transport_enabled: true
        order: 0
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          type: intern
      
      proxy_auth_domain:
        http_enabled: true
        transport_enabled: true
        order: 1
        http_authenticator:
          type: proxy
          challenge: false
          config:
            user_header: "x-proxy-user"
            roles_header: "x-proxy-roles"
        authentication_backend:
          type: noop

kibana.yml:

logging.verbose: true

server.name: kibana
server.host: "0"
elasticsearch.hosts: https://localhost:9200
elasticsearch.ssl.verificationMode: certificate
elasticsearch.ssl.certificateAuthorities: ["/usr/share/kibana/config/root-ca.pem"]
elasticsearch.username: xxxxxxxx
elasticsearch.password: xxxxxxxx
elasticsearch.requestHeadersWhitelist: ["securitytenant","Authorization","x-forwarded-for","x-proxy-user","x-proxy-roles"]

opendistro_security.multitenancy.enabled: true
opendistro_security.multitenancy.tenants.preferred: ["Private", "Global"]
opendistro_security.readonly_mode.roles: ["kibana_read_only"]

# Use this setting if you are running kibana without https
opendistro_security.cookie.secure: false

newsfeed.enabled: false
telemetry.optIn: false
telemetry.enabled: false
security.showInsecureClusterWarning: false
# To configure a WMS map server for use with Kibana, see:
# https://opendistro.github.io/for-elasticsearch-docs/docs/kibana/maptiles/
map.includeElasticMapsService: false


opendistro_security.auth.type: "proxy"
opendistro_security.proxycache.user_header: "x-proxy-user"
opendistro_security.proxycache.roles_header: "x-proxy-roles"

elasticsearch.yml:

cluster.name: "docker-cluster"
network.host: 0.0.0.0

# # minimum_master_nodes need to be explicitly set when bound on a public IP
# # set to 1 to allow single node clusters
# # Details: https://github.com/elastic/elasticsearch/pull/17288
# discovery.zen.minimum_master_nodes: 1

# # Breaking change in 7.0
# # https://www.elastic.co/guide/en/elasticsearch/reference/7.0/breaking-changes-7.0.html#breaking_70_discovery_changes
# cluster.initial_master_nodes: 
#    - elasticsearch1
#    - docker-test-node-1 

opendistro_security.ssl.transport.pemcert_filepath: node.pem
opendistro_security.ssl.transport.pemkey_filepath: node-key.pem
opendistro_security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
opendistro_security.ssl.transport.enforce_hostname_verification: false
opendistro_security.ssl.http.enabled: true
opendistro_security.ssl.http.pemcert_filepath: node.pem
opendistro_security.ssl.http.pemkey_filepath: node-key.pem
opendistro_security.ssl.http.pemtrustedcas_filepath: root-ca.pem
opendistro_security.allow_unsafe_democertificates: false
opendistro_security.allow_default_init_securityindex: true
opendistro_security.authcz.admin_dn:
  - 'CN=xxxxx,O=xxxxx,L=xxxxx,ST=xxxxx,C=US'

opendistro_security.audit.type: internal_elasticsearch
opendistro_security.enable_snapshot_restore_privilege: true
opendistro_security.check_snapshot_restore_write_privileges: true
opendistro_security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
cluster.routing.allocation.disk.threshold_enabled: false
opendistro_security.audit.config.disabled_rest_categories: NONE
opendistro_security.audit.config.disabled_transport_categories: NONE
opendistro_security.nodes_dn:
  - 'CN=xxxxx,O=xxxxx,L=xxxxx,ST=xxxxx,C=US'

Maybe you can try to set your kibana’s ( or your proxy) host ip in kibana.yml rather than 0.0.0.0

server.host: xxx.xxx.xxx.xxx

@mudboyzh Thanks for the reply. I gave that a shot but no luck. I’m getting the following error when attempting to use kibana or proxy host ip in kibana.yml:

Error: listen EADDRNOTAVAIL: address not available xx.xxx.xxx.xxx:5601\n

This may be unrelated or not an issue, but it appears that the kibana basepath “/kibana” is not getting appended to the address when I modify the server.host to point to an ip address other than “0”.

I’ve reverted to the configuration settings in my original post.

In addition to the 401 error in the Kibana logs that I noted in my original post, I also see the following error in the Chrome dev console when I attempt to create a PDF/PNG report:

Cannot read property 'includes' of undefined

This appears to be the exact same error (word-for-word in the stacktrace) on the front end as issue 5618.

However, in the case of issue 5618, it appears that they couldn’t even create a CSV report, whereas my issue occurs only with PDF/PNG reports. I can create and download CSV reports from the Discover tab or from saved searches in the Reporting tab as expected without any errors in Chrome or in the Kibana logs.

Hi Ryan,
Did you fix or found a workaround to your issue? I observed the same error when I tried to use ODFE 1.13.2 (with official docker image) I cannot generated a pdf/png report.
Regards

Hi @pascal. I was not ever able to resolve this in ODFE 1.13.2. However, I ended up resolving my issue by upgrading to OpenSearch 1.2.4 and doing two additional things:

  1. I had large headers that weren’t being passed through by my reverse-proxy (nginx), and I had to add the line large_client_header_buffers 4 32k; to my server block configuration in nginx.conf. If you are using keycloak with openid connect, you may also need to add set $session_name nginx_session; in your server configuration.
  2. If you are not using a reverse-proxy or keycloak and still seeing the same issue, check your opensearch_dashboards.yml (or kibana.yml) for a line like opensearchDashboards.index: ".opensearch_dashboards", and comment it out.

Here is a working setup for OpenSearch 1.2.4 and OpenSearch Dashboards 1.2.0 in Docker with Keycloak and an NginX reverse proxy - https://github.com/rylativity/opensearch-nginx-keycloak