# Once in a while receiving "security\_exception" form opensearch using aiven kafka connector **URL:** \ **Category:** Security\ **Tags:** troubleshoot\ **Created:** [May 31, 2024, 9:57am UTC](https://forum.opensearch.org/t/once-in-a-while-receiving-security-exception-form-opensearch-using-aiven-kafka-connector/19652 "2024-05-31T09:57:00Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![javad](https://avatars.discourse-cdn.com/v4/letter/j/cdc98d/32.png) [@javad](https://forum.opensearch.org/u/javad)\ **Post date:** [May 31, 2024, 9:57am UTC](https://forum.opensearch.org/t/once-in-a-while-receiving-security-exception-form-opensearch-using-aiven-kafka-connector/19652/1 "2024-05-31T09:57:00Z")
**Versions** (relevant - OpenSearch/Dashboard/Server OS/Browser): v 2.12.0 **Describe the issue** : I’m using aiven kafka connector to opensearch (version 2.12.0 ), and using data stream to ingest data to different indeices, it works well however once in a while kakfa connector tasks fails with error: “security\_exception”, when I stop the connector and resume it, it starts working untill receiving the same error after some days. this error seems to only affect indices with millions of records (high velocity); indices with fewer records don’t encounter this exception. **Configuration** : my cluster has 5 nodes, 1 master, 1 master-eligible & data and 3 data nodes running on AWS EC2 machines, using https with AWS ALB with public certificate which is then forwarded to target port 9200 with self-signed certifcate. **Relevant Logs or Screenshots** : `{ "name": "iot-node-data-sink", "connector": { "state": "RUNNING", "worker_id": "ip-172-xx-xx-xx.eu-central-1.compute.internal:8083" }, "tasks": [{ "id": 0, "state": "FAILED", "worker_id": "ip-172-xx-xx-xx.eu-central-1.compute.internal:8083", "trace": "org.apache.kafka.connect.errors.ConnectException: Exiting WorkerSinkTask due to unrecoverable exception.\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:632)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:350)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:250)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:219)\n\tat org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:204)\n\tat org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:259)\n\tat org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:236)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:829)\nCaused by: org.apache.kafka.connect.errors.ConnectException: Non-repeatable exception trown by bulk processing\n\tat io.aiven.kafka.connect.opensearch.RetryUtil.callWithRetry(RetryUtil.java:127)\n\tat io.aiven.kafka.connect.opensearch.BulkProcessor$BulkTask.execute(BulkProcessor.java:398)\n\tat io.aiven.kafka.connect.opensearch.BulkProcessor$BulkTask.call(BulkProcessor.java:370)\n\tat io.aiven.kafka.connect.opensearch.BulkProcessor$BulkTask.call(BulkProcessor.java:351)\n\t... 4 more\nCaused by: OpenSearchStatusException[OpenSearch exception [type=security_exception, reason=Unexpected exception indices:data/write/bulk]]\n\tat org.opensearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:209)\n\tat org.opensearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2235)\n\tat org.opensearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2212)\n\tat org.opensearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1931)\n\tat org.opensearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1884)\n\tat org.opensearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1852)\n\tat org.opensearch.client.RestHighLevelClient.bulk(RestHighLevelClient.java:371)\n\tat io.aiven.kafka.connect.opensearch.BulkProcessor$BulkTask.lambda$execute$0(BulkProcessor.java:401)\n\tat io.aiven.kafka.connect.opensearch.RetryUtil.callWithRetry(RetryUtil.java:119)\n\t... 7 more\n\tSuppressed: org.opensearch.client.ResponseException: method [POST], host [https://opensearch-nodes.xxxxxx.de], URI [/_bulk?timeout=1m], status line [HTTP/1.1 500 Internal Server Error]\n{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"Unexpected exception indices:data/write/bulk\"}],\"type\":\"security_exception\",\"reason\":\"Unexpected exception indices:data/write/bulk\"},\"status\":500}\n\t\tat org.opensearch.client.RestClient.convertResponse(RestClient.java:375)\n\t\tat org.opensearch.client.RestClient.performRequest(RestClient.java:345)\n\t\tat org.opensearch.client.RestClient.performRequest(RestClient.java:320)\n\t\tat org.opensearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1918)\n\t\t... 12 more\n" } ], "type": "sink" }` other task error: `{ "name": "mqtt-opensearch-sink", "connector": { "state": "RUNNING", "worker_id": "ip-172-xx-xx-xx.eu-central-1.compute.internal:8083" }, "tasks": [{ "id": 0, "state": "FAILED", "worker_id": "ip-172-xx-xx-xx.eu-central-1.compute.internal:8083", "trace": "org.apache.kafka.connect.errors.ConnectException: Exiting WorkerSinkTask due to unrecoverable exception.\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:632)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:350)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:250)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:219)\n\tat org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:204)\n\tat org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:259)\n\tat org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:236)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:829)\nCaused by: org.apache.kafka.connect.errors.ConnectException: Non-repeatable exception trown by bulk processing\n\tat io.aiven.kafka.connect.opensearch.RetryUtil.callWithRetry(RetryUtil.java:127)\n\tat io.aiven.kafka.connect.opensearch.BulkProcessor$BulkTask.execute(BulkProcessor.java:398)\n\tat io.aiven.kafka.connect.opensearch.BulkProcessor$BulkTask.call(BulkProcessor.java:370)\n\tat io.aiven.kafka.connect.opensearch.BulkProcessor$BulkTask.call(BulkProcessor.java:351)\n\t... 4 more\nCaused by: OpenSearchStatusException[OpenSearch exception [type=security_exception, reason=Unexpected exception indices:data/write/bulk]]\n\tat org.opensearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:209)\n\tat org.opensearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2235)\n\tat org.opensearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2212)\n\tat org.opensearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1931)\n\tat org.opensearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1884)\n\tat org.opensearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1852)\n\tat org.opensearch.client.RestHighLevelClient.bulk(RestHighLevelClient.java:371)\n\tat io.aiven.kafka.connect.opensearch.BulkProcessor$BulkTask.lambda$execute$0(BulkProcessor.java:401)\n\tat io.aiven.kafka.connect.opensearch.RetryUtil.callWithRetry(RetryUtil.java:119)\n\t... 7 more\n\tSuppressed: org.opensearch.client.ResponseException: method [POST], host [https://opensearch-nodes.xxxxxx.de], URI [/_bulk?timeout=1m], status line [HTTP/1.1 500 Internal Server Error]\n{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"Unexpected exception indices:data/write/bulk\"}],\"type\":\"security_exception\",\"reason\":\"Unexpected exception indices:data/write/bulk\"},\"status\":500}\n\t\tat org.opensearch.client.RestClient.convertResponse(RestClient.java:375)\n\t\tat org.opensearch.client.RestClient.performRequest(RestClient.java:345)\n\t\tat org.opensearch.client.RestClient.performRequest(RestClient.java:320)\n\t\tat org.opensearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1918)\n\t\t... 12 more\n" } ], "type": "sink" }` ---
**Author:** ![pablo](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/pablo/32/10363_2.png) [@pablo](https://forum.opensearch.org/u/pablo)\ **Post date:** [June 5, 2024, 1:51pm UTC](https://forum.opensearch.org/t/once-in-a-while-receiving-security-exception-form-opensearch-using-aiven-kafka-connector/19652/2 "2024-06-05T13:51:05Z")
> [@javad](#): > > HTTP/1.1 500 Internal Server Error @javad Could you share the OpenSearch logs corresponding to the timestamp of the HTTP 500 error? ---
**Author:** ![javad](https://avatars.discourse-cdn.com/v4/letter/j/cdc98d/32.png) [@javad](https://forum.opensearch.org/u/javad)\ **Post date:** [June 5, 2024, 10:10pm UTC](https://forum.opensearch.org/t/once-in-a-while-receiving-security-exception-form-opensearch-using-aiven-kafka-connector/19652/3 "2024-06-05T22:10:10Z")
I deployed OS with docker compose and this issue happens days ago, with docker as it is pointed out in the OS document and this page: [Opensearch console log for docker - #5 by radu.gheorghe](https://forum.opensearch.org/t/opensearch-console-log-for-docker/15787/5) I do not know how can I get logs on the file! inside container there is only these logs: PerformanceAnalyzer.log performance\_analyzer\_agent\_stats.log.2024-05-24-1.gz gc.log performance\_analyzer\_agent\_stats.log.2024-05-25-1.gz gc.log.00 performance\_analyzer\_agent\_stats.log.2024-05-26-1.gz gc.log.01 performance\_analyzer\_agent\_stats.log.2024-05-27-1.gz performance\_analyzer\_agent\_stats.log and I can see console log with this command: sudo docker logs --timestamps os01 2\>&1 | grep “$(date +%Y-%m-%d)T0[8]:” Is there a way I can persist logs in file when using docker? Do you think this exception can occur due to rate limit API call, e.g. this: > **[API rate limiting](https://opensearch.org/docs/latest/security/configuration/api-rate-limiting/)** > > API rate limiting or circuit-breaker ---
**Author:** ![pablo](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/pablo/32/10363_2.png) [@pablo](https://forum.opensearch.org/u/pablo)\ **Post date:** [June 5, 2024, 11:28pm UTC](https://forum.opensearch.org/t/once-in-a-while-receiving-security-exception-form-opensearch-using-aiven-kafka-connector/19652/4 "2024-06-05T23:28:10Z")
@javad Try using log4j2.properties. You need to do a few things to get the logs in a file. 1. Set the path.log in each OpenSearch node i.e. ```auto path.logs: /var/log/opensearch ``` > **[Configuration and system settings](https://opensearch.org/docs/latest/install-and-configure/configuring-opensearch/configuration-system/)** > > Configuration and system settings 2. Create a `log` folder in the docker host (i.e. in the docker-compose folder) 3. Change user and group of the `log` folder to user and group with ID 1000 (this corresponds with an opensearch user and group inside the container) 4. Mount log4j2.properties and log folder as a volume in docker-compose 5. Use the below log4j2.properties example to write the logs to both console and a file. ```auto status = error appender.console.type = Console appender.console.name = console appender.console.layout.type = PatternLayout appender.console.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n appender.rolling.type = RollingFile appender.rolling.name = rolling appender.rolling.fileName = ${sys:opensearch.logs.base_path}/opensearch.log appender.rolling.layout.type = PatternLayout appender.rolling.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n appender.rolling.filePattern = ${sys:opensearch.logs.base_path}/opensearch-%d{yyyy-MM-dd}.log.gz appender.rolling.policies.type = Policies appender.rolling.policies.time.type = TimeBasedTriggeringPolicy appender.rolling.policies.time.interval = 1 appender.rolling.policies.time.modulate = true appender.rolling.policies.size.type = SizeBasedTriggeringPolicy appender.rolling.policies.size.size = 256MB rootLogger.level = info rootLogger.appenderRef.console.ref = console rootLogger.appenderRef.rolling.ref = rolling ``` ---
**Author:** ![pablo](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/pablo/32/10363_2.png) [@pablo](https://forum.opensearch.org/u/pablo)\ **Post date:** [June 5, 2024, 11:37pm UTC](https://forum.opensearch.org/t/once-in-a-while-receiving-security-exception-form-opensearch-using-aiven-kafka-connector/19652/5 "2024-06-05T23:37:14Z")
@javad I’m not sure if the API rate limiting is causing it. The error you’ve provided is a general Internal server error. Sometimes API clients won’t catch all the errors and will reply with the last one. Maybe there are some errors preceding HTTP 500 in the OpenSearch logs. Have you checked the resource utilization in OpenSearch nodes when that problematic \_bulk is executed? What resources did you assign to your EC2 machines? Did you install OpenSearch as a Docker or a service? What is the backing storage of EC2? Did you confirm that the issue is related to only those large indices (millions of documents)? ---
**Author:** ![javad](https://avatars.discourse-cdn.com/v4/letter/j/cdc98d/32.png) [@javad](https://forum.opensearch.org/u/javad)\ **Post date:** [June 7, 2024, 9:17am UTC](https://forum.opensearch.org/t/once-in-a-while-receiving-security-exception-form-opensearch-using-aiven-kafka-connector/19652/6 "2024-06-07T09:17:16Z")
Unfortunately, our Docker daemon defaulted to using the logging driver with a file rotation of 1, which means I don’t have access to old logs. I’ve now activated logging to be written both to the console and the file system, with the folder mapped. I’m waiting for the exception to occur again, and once it does, I’ll share the logs with you. For those who want to access the logs in files, here are the steps: You can check your Docker container logs here: get the : > docker ps -a > log file is in this path: /var/lib/docker/containers/ > you can check log rotatin and ur container log driver: > docker inspect \ | grep -A 5 LogConfig > “LogConfig”: { > “Type”: “json-file”, > “Config”: { > “max-file”: “1”, > “max-size”: “100m” > } for activating wrting logs to file: do the steps @pablo pointed out in previous posts, for me the default path of log is in: “path”: { “data”: , “logs”: “/usr/share/opensearch/logs”, “shared\_data”: “”, “home”: “/usr/share/opensearch”, “repo”: } you can check it in devTools: GET /\_cluster/settings?include\_defaults=true put these in log4j2.properties ```auto appender.console.type = Console appender.console.name = console appender.console.layout.type = PatternLayout appender.console.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n appender.rolling.type = RollingFile appender.rolling.name = rolling appender.rolling.fileName = ${sys:opensearch.logs.base_path}/opensearch.log appender.rolling.layout.type = PatternLayout appender.rolling.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n appender.rolling.filePattern = ${sys:opensearch.logs.base_path}/opensearch-%d{yyyy-MM-dd}.log.gz appender.rolling.policies.type = Policies appender.rolling.policies.time.type = TimeBasedTriggeringPolicy appender.rolling.policies.time.interval = 1 appender.rolling.policies.time.modulate = true appender.rolling.policies.size.type = SizeBasedTriggeringPolicy appender.rolling.policies.size.size = 256MB rootLogger.level = info rootLogger.appenderRef.console.ref = console rootLogger.appenderRef.rolling.ref = rolling ``` and maped the volumes in ur compose file: - “/home/xxxx/opensearch-cluster/opensearch-node1/config/log4j2.properties:/usr/share/opensearch/config/log4j2.properties” - “/home/xxxx/opensearch-cluster/opensearch-node1/logs:/usr/share/opensearch/logs” ---
**Author:** ![javad](https://avatars.discourse-cdn.com/v4/letter/j/cdc98d/32.png) [@javad](https://forum.opensearch.org/u/javad)\ **Post date:** [June 7, 2024, 9:46am UTC](https://forum.opensearch.org/t/once-in-a-while-receiving-security-exception-form-opensearch-using-aiven-kafka-connector/19652/7 "2024-06-07T09:46:00Z")
yes, I confirm that only happens to larger indices, I am using same username and password for both kafka connectors tasks using data stream. I have 5 nodes uses AWS EBS storage and deployed with docker compose with 16GB of RAM (t3a.xlarge EC2 instances). ```auto ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role node.roles cluster_manager name 172.24.x.x 18 97 8 0.80 0.80 0.91 d data - os04 172.24.x.x 12 84 16 0.67 0.84 0.92 d cluster_manager_eligible,data - os02 172.24.x.x 44 67 10 0.83 0.95 1.02 d data - os05 172.24.x.x 11 49 2 0.18 0.15 0.10 m cluster_manager * os01 172.24.x.x 15 90 7 0.79 0.78 0.78 d data - os03 ``` 4 voulmes of following type for data node machines: nvme0n1 259:0 0 250G 0 disk ├─nvme0n1p1 259:1 0 249.9G 0 part / ├─nvme0n1p14 259:2 0 4M 0 part └─nvme0n1p15 259:3 0 106M 0 part /boot/efi and for master node: nvme0n1 259:0 0 30G 0 disk ├─nvme0n1p1 259:1 0 29.9G 0 part / ├─nvme0n1p14 259:2 0 4M 0 part └─nvme0n1p15 259:3 0 106M 0 part /boot/efi ---
**Author:** ![javad](https://avatars.discourse-cdn.com/v4/letter/j/cdc98d/32.png) [@javad](https://forum.opensearch.org/u/javad)\ **Post date:** [June 11, 2024, 7:55am UTC](https://forum.opensearch.org/t/once-in-a-while-receiving-security-exception-form-opensearch-using-aiven-kafka-connector/19652/8 "2024-06-11T07:55:35Z")
It happened again after nearly 6 days, and once again on the same large index. Fortunately, this time I have the log. I have included the error details showing exactly when it occurred, including both the master node log and the Aiven Kafka Connect failed task log. The exception is: > Unexpected exception java.util.ConcurrentModificationException. This is OS master-node log: ````auto [2024-06-11T03:14:28,034][ERROR][o.o.s.f.SecurityFilter] [os01] Unexpected exception java.util.ConcurrentModificationException java.util.ConcurrentModificationException: null at java.base/java.util.HashMap$HashIterator.nextNode(HashMap.java:1605) ~[?:?] at java.base/java.util.HashMap$KeyIterator.next(HashMap.java:1628) ~[?:?] at java.base/java.util.Collections$UnmodifiableCollection$1.next(Collections.java:1078) ~[?:?] at java.base/java.util.AbstractCollection.addAll(AbstractCollection.java:337) ~[?:?] at java.base/java.util.HashSet.(HashSet.java:121) ~[?:?] at org.opensearch.security.securityconf.ConfigModelV7$RoleMappingHolder.map(ConfigModelV7.java:1304) ~[opensearch-security-2.12.0.0.jar:2.12.0.0] at org.opensearch.security.securityconf.ConfigModelV7.mapSecurityRoles(ConfigModelV7.java:1368) ~[opensearch-security-2.12.0.0.jar:2.12.0.0] at org.opensearch.security.privileges.PrivilegesEvaluator.mapRoles(PrivilegesEvaluator.java:584) ~[opensearch-security-2.12.0.0.jar:2.12.0.0] at org.opensearch.security.privileges.PrivilegesEvaluator.evaluate(PrivilegesEvaluator.java:254) ~[opensearch-security-2.12.0.0.jar:2.12.0.0] at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:381) [opensearch-security-2.12.0.0.jar:2.12.0.0] at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:165) [opensearch-security-2.12.0.0.jar:2.12.0.0] at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.12.0.jar:2.12.0] at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) [opensearch-2.12.0.jar:2.12.0] at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) [opensearch-2.12.0.jar:2.12.0] at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) [opensearch-2.12.0.jar:2.12.0] at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) [opensearch-2.12.0.jar:2.12.0] at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:476) [opensearch-2.12.0.jar:2.12.0] at org.opensearch.client.support.AbstractClient.bulk(AbstractClient.java:552) [opensearch-2.12.0.jar:2.12.0] at org.opensearch.rest.action.document.RestBulkAction.lambda$prepareRequest$0(RestBulkAction.java:111) [opensearch-2.12.0.jar:2.12.0] at org.opensearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:128) [opensearch-2.12.0.jar:2.12.0] at org.opensearch.security.filter.SecurityRestFilter$AuthczRestHandler.handleRequest(SecurityRestFilter.java:182) [opensearch-security-2.12.0.0.jar:2.12.0.0] at org.opensearch.rest.RestController.dispatchRequest(RestController.java:334) [opensearch-2.12.0.jar:2.12.0] at org.opensearch.rest.RestController.tryAllHandlers(RestController.java:425) [opensearch-2.12.0.jar:2.12.0] at org.opensearch.rest.RestController.dispatchRequest(RestController.java:263) [opensearch-2.12.0.jar:2.12.0] at org.opensearch.security.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:69) [opensearch-security-2.12.0.0.jar:2.12.0.0] at org.opensearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:387) [opensearch-2.12.0.jar:2.12.0] at org.opensearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:468) [opensearch-2.12.0.jar:2.12.0] at org.opensearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:370) [opensearch-2.12.0.jar:2.12.0] at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:56) [transport-netty4-client-2.12.0.jar:2.12.0] at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:42) [transport-netty4-client-2.12.0.jar:2.12.0] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at org.opensearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:72) [transport-netty4-client-2.12.0.jar:2.12.0] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:102) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) [netty-codec-4.1.106.Final.jar:4.1.106.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) [netty-codec-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1475) [netty-handler-4.1.106.Final.jar:4.1.106.Final] at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1338) [netty-handler-4.1.106.Final.jar:4.1.106.Final] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1387) [netty-handler-4.1.106.Final.jar:4.1.106.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) [netty-codec-4.1.106.Final.jar:4.1.106.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) [netty-codec-4.1.106.Final.jar:4.1.106.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) [netty-codec-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.106.Final.jar:4.1.106.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.106.Final.jar:4.1.106.Final] at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] [2024-06-11T03:14:28,037][WARN][r.suppressed] [os01] path: /_bulk, params: {timeout=1m} org.opensearch.OpenSearchSecurityException: Unexpected exception indices:data/write/bulk at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:479) [opensearch-security-2.12.0.0.jar:2.12.0.0] at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:165) [opensearch-security-2.12.0.0.jar:2.12.0.0] at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.12.0.jar:2.12.0] at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) [opensearch-2.12.0.jar:2.12.0] at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) [opensearch-2.12.0.jar:2.12.0] at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) [opensearch-2.12.0.jar:2.12.0] at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) [opensearch-2.12.0.jar:2.12.0] at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:476) [opensearch-2.12.0.jar:2.12.0] at org.opensearch.client.support.AbstractClient.bulk(AbstractClient.java:552) [opensearch-2.12.0.jar:2.12.0] at org.opensearch.rest.action.document.RestBulkAction.lambda$prepareRequest$0(RestBulkAction.java:111) [opensearch-2.12.0.jar:2.12.0] at org.opensearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:128) [opensearch-2.12.0.jar:2.12.0] at org.opensearch.security.filter.SecurityRestFilter$AuthczRestHandler.handleRequest(SecurityRestFilter.java:182) [opensearch-security-2.12.0.0.jar:2.12.0.0] at org.opensearch.rest.RestController.dispatchRequest(RestController.java:334) [opensearch-2.12.0.jar:2.12.0] at org.opensearch.rest.RestController.tryAllHandlers(RestController.java:425) [opensearch-2.12.0.jar:2.12.0] at org.opensearch.rest.RestController.dispatchRequest(RestController.java:263) [opensearch-2.12.0.jar:2.12.0] at org.opensearch.security.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:69) [opensearch-security-2.12.0.0.jar:2.12.0.0] at org.opensearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:387) [opensearch-2.12.0.jar:2.12.0] at org.opensearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:468) [opensearch-2.12.0.jar:2.12.0] at org.opensearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:370) [opensearch-2.12.0.jar:2.12.0] at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:56) [transport-netty4-client-2.12.0.jar:2.12.0] at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:42) [transport-netty4-client-2.12.0.jar:2.12.0] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at org.opensearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:72) [transport-netty4-client-2.12.0.jar:2.12.0] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:102) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) [netty-codec-4.1.106.Final.jar:4.1.106.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) [netty-codec-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1475) [netty-handler-4.1.106.Final.jar:4.1.106.Final] at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1338) [netty-handler-4.1.106.Final.jar:4.1.106.Final] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1387) [netty-handler-4.1.106.Final.jar:4.1.106.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) [netty-codec-4.1.106.Final.jar:4.1.106.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) [netty-codec-4.1.106.Final.jar:4.1.106.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) [netty-codec-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.106.Final.jar:4.1.106.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.106.Final.jar:4.1.106.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.106.Final.jar:4.1.106.Final] at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]``` This is aiven kafka-connect error log of tasks failed: ```` ``` "tasks": [ { "id": 0, "state": "FAILED", "worker_id": "ip-172-24-xx-xx.eu-central-1.compute.internal:8083", "trace": "org.apache.kafka.connect.errors.ConnectException: Exiting WorkerSinkTask due to unrecoverable exception.\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:632)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:350)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:250)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:219)\n\tat org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:204)\n\tat org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:259)\n\tat org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:236)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:829)\nCaused by: org.apache.kafka.connect.errors.ConnectException: Non-repeatable exception trown by bulk processing\n\tat io.aiven.kafka.connect.opensearch.RetryUtil.callWithRetry(RetryUtil.java:127)\n\tat io.aiven.kafka.connect.opensearch.BulkProcessor$BulkTask.execute(BulkProcessor.java:398)\n\tat io.aiven.kafka.connect.opensearch.BulkProcessor$BulkTask.call(BulkProcessor.java:370)\n\tat io.aiven.kafka.connect.opensearch.BulkProcessor$BulkTask.call(BulkProcessor.java:351)\n\t... 4 more \nCaused by: OpenSearchStatusException[OpenSearch exception [type=security_exception, reason=Unexpected exception indices:data/write/bulk]]\n\tat org.opensearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:209)\n\tat org.opensearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2235)\n\tat org.opensearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2212)\n\tat org.opensearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1931)\n\tat org.opensearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1884)\n\tat org.opensearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1852)\n\tat org.opensearch.client.RestHighLevelClient.bulk(RestHighLevelClient.java:371)\n\tat io.aiven.kafka.connect.opensearch.BulkProcessor$BulkTask.lambda$execute$0(BulkProcessor.java:401)\n\tat io.aiven.kafka.connect.opensearch.RetryUtil.callWithRetry(RetryUtil.java:119)\n\t... 7 more\n\t Suppressed: org.opensearch.client.ResponseException: method [POST], host [https://opensearch-node.xxxx.de], URI [/_bulk?timeout=1m], status line [HTTP/1.1 500 Internal Server Error]\n{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"Unexpected exception indices:data/write/bulk\"}], \"type\":\"security_exception\",\"reason\":\"Unexpected exception indices:data/write/bulk\"},\"status\":500}\n\t\tat org.opensearch.client.RestClient.convertResponse(RestClient.java:375)\n\t\tat org.opensearch.client.RestClient.performRequest(RestClient.java:345)\n\t\tat org.opensearch.client.RestClient.performRequest(RestClient.java:320)\n\t\tat org.opensearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1918)\n\t\t... 12 more\n" } ], "type": "sink" ``` }``` ---
**Author:** ![reta](https://avatars.discourse-cdn.com/v4/letter/r/cdc98d/32.png) [@reta](https://forum.opensearch.org/u/reta)\ **Post date:** [June 11, 2024, 12:48pm UTC](https://forum.opensearch.org/t/once-in-a-while-receiving-security-exception-form-opensearch-using-aiven-kafka-connector/19652/9 "2024-06-11T12:48:10Z")
Hm … it is supposed to be fixed [1] but it apparently is not, could you please comment out on the issue so it could be reopened? Thank you. [1] [[BUG] ConcurrentModificationException in security package · Issue #3404 · opensearch-project/security · GitHub](https://github.com/opensearch-project/security/issues/3404) ---
**Author:** ![pablo](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/pablo/32/10363_2.png) [@pablo](https://forum.opensearch.org/u/pablo)\ **Post date:** [June 19, 2024, 1:37pm UTC](https://forum.opensearch.org/t/once-in-a-while-receiving-security-exception-form-opensearch-using-aiven-kafka-connector/19652/10 "2024-06-19T13:37:01Z")
Updating with a link to a new GitHub issue created by @reta > > > I'm using OS v 2.12.0, and when ingesting data with kafka-connector after some d…ays, get this exception. I explained issue with details in follwoing link: (https://forum.opensearch.org/t/once-in-a-while-receiving-security-exception-form-opensearch-using-aiven-kafka-connector/19652) and you can refer to stack trace of exception: > > \`\[2024-06-11T03:14:28,034\]\[ERROR\]\[o.o.s.f.SecurityFilter \] \[os01\] Unexpected exception java.util.ConcurrentModificationException > java.util.ConcurrentModificationException: null > at java.base/java.util.HashMap$HashIterator.nextNode(HashMap.java:1605) ~\[?:?\] > at java.base/java.util.HashMap$KeyIterator.next(HashMap.java:1628) ~\[?:?\] > at java.base/java.util.Collections$UnmodifiableCollection$1.next(Collections.java:1078) ~\[?:?\] > at java.base/java.util.AbstractCollection.addAll(AbstractCollection.java:337) ~\[?:?\] > at java.base/java.util.HashSet.\(HashSet.java:121) ~\[?:?\] > at org.opensearch.security.securityconf.ConfigModelV7$RoleMappingHolder.map(ConfigModelV7.java:1304) ~\[opensearch-security-2.12.0.0.jar:2.12.0.0\] > at org.opensearch.security.securityconf.ConfigModelV7.mapSecurityRoles(ConfigModelV7.java:1368) ~\[opensearch-security-2.12.0.0.jar:2.12.0.0\] > at org.opensearch.security.privileges.PrivilegesEvaluator.mapRoles(PrivilegesEvaluator.java:584) ~\[opensearch-security-2.12.0.0.jar:2.12.0.0\] > at org.opensearch.security.privileges.PrivilegesEvaluator.evaluate(PrivilegesEvaluator.java:254) ~\[opensearch-security-2.12.0.0.jar:2.12.0.0\] > at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:381) \[opensearch-security-2.12.0.0.jar:2.12.0.0\] > at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:165) \[opensearch-security-2.12.0.0.jar:2.12.0.0\] > at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) \[opensearch-2.12.0.jar:2.12.0\] > at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) \[opensearch-2.12.0.jar:2.12.0\] > at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) \[opensearch-2.12.0.jar:2.12.0\] > at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) \[opensearch-2.12.0.jar:2.12.0\] > at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) \[opensearch-2.12.0.jar:2.12.0\] > at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:476) \[opensearch-2.12.0.jar:2.12.0\] > at org.opensearch.client.support.AbstractClient.bulk(AbstractClient.java:552) \[opensearch-2.12.0.jar:2.12.0\] > at org.opensearch.rest.action.document.RestBulkAction.lambda$prepareRequest$0(RestBulkAction.java:111) \[opensearch-2.12.0.jar:2.12.0\] > at org.opensearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:128) \[opensearch-2.12.0.jar:2.12.0\] > at org.opensearch.security.filter.SecurityRestFilter$AuthczRestHandler.handleRequest(SecurityRestFilter.java:182) \[opensearch-security-2.12.0.0.jar:2.12.0.0\] > at org.opensearch.rest.RestController.dispatchRequest(RestController.java:334) \[opensearch-2.12.0.jar:2.12.0\] > at org.opensearch.rest.RestController.tryAllHandlers(RestController.java:425) \[opensearch-2.12.0.jar:2.12.0\] > at org.opensearch.rest.RestController.dispatchRequest(RestController.java:263) \[opensearch-2.12.0.jar:2.12.0\] > at org.opensearch.security.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:69) \[opensearch-security-2.12.0.0.jar:2.12.0.0\] > at org.opensearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:387) \[opensearch-2.12.0.jar:2.12.0\] > at org.opensearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:468) \[opensearch-2.12.0.jar:2.12.0\] > at org.opensearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:370) \[opensearch-2.12.0.jar:2.12.0\] > at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:56) \[transport-netty4-client-2.12.0.jar:2.12.0\] > at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:42) \[transport-netty4-client-2.12.0.jar:2.12.0\] > at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at org.opensearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:72) \[transport-netty4-client-2.12.0.jar:2.12.0\] > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) \[netty-codec-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) \[netty-codec-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) \[netty-codec-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:102) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) \[netty-codec-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) \[netty-codec-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) \[netty-handler-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) \[netty-codec-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1475) \[netty-handler-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1338) \[netty-handler-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1387) \[netty-handler-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) \[netty-codec-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) \[netty-codec-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) \[netty-codec-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) \[netty-common-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) \[netty-common-4.1.106.Final.jar:4.1.106.Final\] > at java.base/java.lang.Thread.run(Thread.java:1583) \[?:?\] > \[2024-06-11T03:14:28,037\]\[WARN \]\[r.suppressed \] \[os01\] path: /\_bulk, params: {timeout=1m} > org.opensearch.OpenSearchSecurityException: Unexpected exception indices:data/write/bulk > at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:479) \[opensearch-security-2.12.0.0.jar:2.12.0.0\] > at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:165) \[opensearch-security-2.12.0.0.jar:2.12.0.0\] > at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) \[opensearch-2.12.0.jar:2.12.0\] > at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) \[opensearch-2.12.0.jar:2.12.0\] > at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) \[opensearch-2.12.0.jar:2.12.0\] > at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) \[opensearch-2.12.0.jar:2.12.0\] > at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) \[opensearch-2.12.0.jar:2.12.0\] > at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:476) \[opensearch-2.12.0.jar:2.12.0\] > at org.opensearch.client.support.AbstractClient.bulk(AbstractClient.java:552) \[opensearch-2.12.0.jar:2.12.0\] > at org.opensearch.rest.action.document.RestBulkAction.lambda$prepareRequest$0(RestBulkAction.java:111) \[opensearch-2.12.0.jar:2.12.0\] > at org.opensearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:128) \[opensearch-2.12.0.jar:2.12.0\] > at org.opensearch.security.filter.SecurityRestFilter$AuthczRestHandler.handleRequest(SecurityRestFilter.java:182) \[opensearch-security-2.12.0.0.jar:2.12.0.0\] > at org.opensearch.rest.RestController.dispatchRequest(RestController.java:334) \[opensearch-2.12.0.jar:2.12.0\] > at org.opensearch.rest.RestController.tryAllHandlers(RestController.java:425) \[opensearch-2.12.0.jar:2.12.0\] > at org.opensearch.rest.RestController.dispatchRequest(RestController.java:263) \[opensearch-2.12.0.jar:2.12.0\] > at org.opensearch.security.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:69) \[opensearch-security-2.12.0.0.jar:2.12.0.0\] > at org.opensearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:387) \[opensearch-2.12.0.jar:2.12.0\] > at org.opensearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:468) \[opensearch-2.12.0.jar:2.12.0\] > at org.opensearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:370) \[opensearch-2.12.0.jar:2.12.0\] > at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:56) \[transport-netty4-client-2.12.0.jar:2.12.0\] > at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:42) \[transport-netty4-client-2.12.0.jar:2.12.0\] > at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at org.opensearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:72) \[transport-netty4-client-2.12.0.jar:2.12.0\] > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) \[netty-codec-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) \[netty-codec-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) \[netty-codec-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:102) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) \[netty-codec-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) \[netty-codec-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) \[netty-handler-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) \[netty-codec-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1475) \[netty-handler-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1338) \[netty-handler-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1387) \[netty-handler-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) \[netty-codec-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) \[netty-codec-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) \[netty-codec-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) \[netty-transport-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) \[netty-common-4.1.106.Final.jar:4.1.106.Final\] > at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) \[netty-common-4.1.106.Final.jar:4.1.106.Final\] > at java.base/java.lang.Thread.run(Thread.java:1583) \[?:?\]\` > > > > > > @javad87 Could you open up a new issue with the call stack that you encountered this issue on and any other details? > > \_Originally posted by @peternied in https://github.com/opensearch-project/security/issues/3404#issuecomment-2161426729\_