Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Describe the issue:
Except communication with OpenSearch Dashboards and OpenSearch Cluster, I need my plugin to communicate with external service which is currently hosted on localhost:18880 (for development). I’m using Apache Http Client 5 for this purpose and it fails to connect to the service with java.security.AccessControlException: access denied ("java.net.SocketPermission" "127.0.0.1:18880" "connect,resolve").
I can solve this issue locally by providing .policy file and specyfing JVM argument -Djava.security.policy to the running OpenSearch instance. But how do I resolve this in production where my plugin is shipped as zip with other plugins. Is there a way directly in plugin to allow connecting to external sockets, so OpenSearch can pick it up on plugin load? Or is it native way in OpenSearch client library to communicate with other services?
As I understood OpenSearch creates it’s own SecurityManager and it fails if i disable it, so policies must be set in some way, but I did not find a way to do it in plugin or plugin configs, because SecurityManager instantiation is private and and i didn’t find exposed api
Configuration:
Not changed from standard
Relevant Logs or Screenshots:
Full error log when pinging external service
java.security.AccessControlException: access denied ("java.net.SocketPermission" "127.0.0.1:18880" "connect,resolve")
at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:488) ~[?:?]
at java.base/java.security.AccessController.checkPermission(AccessController.java:1071) ~[?:?]
at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:411) ~[?:?]
at java.base/java.lang.SecurityManager.checkConnect(SecurityManager.java:905) ~[?:?]
at java.base/java.net.Socket.connect(Socket.java:747) ~[?:?]
at org.apache.hc.client5.http.socket.PlainConnectionSocketFactory.lambda$connectSocket$0(PlainConnectionSocketFactory.java:91) ~[httpclient5-5.3.1.jar:5.3.1]
at java.base/java.security.AccessController.doPrivileged(AccessController.java:571) ~[?:?]
at org.apache.hc.client5.http.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:90) ~[httpclient5-5.3.1.jar:5.3.1]
at org.apache.hc.client5.http.socket.ConnectionSocketFactory.connectSocket(ConnectionSocketFactory.java:123) ~[httpclient5-5.3.1.jar:5.3.1]
at org.apache.hc.client5.http.impl.io.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:189) ~[httpclient5-5.3.1.jar:5.3.1]
at org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:450) ~[httpclient5-5.3.1.jar:5.3.1]
at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.connectEndpoint(InternalExecRuntime.java:162) ~[httpclient5-5.3.1.jar:5.3.1]
at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.connectEndpoint(InternalExecRuntime.java:172) ~[httpclient5-5.3.1.jar:5.3.1]
at org.apache.hc.client5.http.impl.classic.ConnectExec.execute(ConnectExec.java:142) ~[httpclient5-5.3.1.jar:5.3.1]
at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[httpclient5-5.3.1.jar:5.3.1]
at org.apache.hc.client5.http.impl.classic.ProtocolExec.execute(ProtocolExec.java:192) ~[httpclient5-5.3.1.jar:5.3.1]
at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[httpclient5-5.3.1.jar:5.3.1]
at org.apache.hc.client5.http.impl.classic.HttpRequestRetryExec.execute(HttpRequestRetryExec.java:113) ~[httpclient5-5.3.1.jar:5.3.1]
at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[httpclient5-5.3.1.jar:5.3.1]
at org.apache.hc.client5.http.impl.classic.ContentCompressionExec.execute(ContentCompressionExec.java:152) ~[httpclient5-5.3.1.jar:5.3.1]
at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[httpclient5-5.3.1.jar:5.3.1]
at org.apache.hc.client5.http.impl.classic.RedirectExec.execute(RedirectExec.java:116) ~[httpclient5-5.3.1.jar:5.3.1]
at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) ~[httpclient5-5.3.1.jar:5.3.1]
at org.apache.hc.client5.http.impl.classic.InternalHttpClient.doExecute(InternalHttpClient.java:170) ~[httpclient5-5.3.1.jar:5.3.1]
at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:87) ~[httpclient5-5.3.1.jar:5.3.1]
at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:55) ~[httpclient5-5.3.1.jar:5.3.1]
at org.apache.hc.client5.http.classic.HttpClient.executeOpen(HttpClient.java:183) ~[httpclient5-5.3.1.jar:5.3.1]
at at.risedev.mlbase.idp.ad.service.external.AnomalyDetectionService.busy(AnomalyDetectionService.java:423) ~[IdpAdOpenSearchPlugin-2.19.1.0-SNAPSHOT.jar:2.19.1.0-SNAPSHOT]
at at.risedev.mlbase.idp.ad.service.PollService.pollAdService(PollService.java:55) [IdpAdOpenSearchPlugin-2.19.1.0-SNAPSHOT.jar:2.19.1.0-SNAPSHOT]
at at.risedev.mlbase.idp.ad.rest.RestPollAdServiceAction.lambda$prepareRequest$0(RestPollAdServiceAction.java:69) [IdpAdOpenSearchPlugin-2.19.1.0-SNAPSHOT.jar:2.19.1.0-SNAPSHOT]
at org.opensearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:127) [opensearch-2.19.1.jar:2.19.1]
at org.opensearch.rest.RestController.dispatchRequest(RestController.java:381) [opensearch-2.19.1.jar:2.19.1]
at org.opensearch.rest.RestController.tryAllHandlers(RestController.java:467) [opensearch-2.19.1.jar:2.19.1]
at org.opensearch.rest.RestController.dispatchRequest(RestController.java:287) [opensearch-2.19.1.jar:2.19.1]
at org.opensearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:374) [opensearch-2.19.1.jar:2.19.1]
at org.opensearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:482) [opensearch-2.19.1.jar:2.19.1]
at org.opensearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:357) [opensearch-2.19.1.jar:2.19.1]
at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:56) [transport-netty4-client-2.19.1.jar:2.19.1]
at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:42) [transport-netty4-client-2.19.1.jar:2.19.1]
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
at org.opensearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:72) [transport-netty4-client-2.19.1.jar:2.19.1]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:107) [netty-codec-4.1.118.Final.jar:4.1.118.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:107) [netty-codec-4.1.118.Final.jar:4.1.118.Final]
at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:120) [netty-codec-4.1.118.Final.jar:4.1.118.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:107) [netty-codec-4.1.118.Final.jar:4.1.118.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:107) [netty-codec-4.1.118.Final.jar:4.1.118.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) [netty-codec-4.1.118.Final.jar:4.1.118.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) [netty-codec-4.1.118.Final.jar:4.1.118.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:289) [netty-handler-4.1.118.Final.jar:4.1.118.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:107) [netty-codec-4.1.118.Final.jar:4.1.118.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:796) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:697) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:660) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.118.Final.jar:4.1.118.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998) [netty-common-4.1.118.Final.jar:4.1.118.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.118.Final.jar:4.1.118.Final]
If i force disable SecurityManager with -Djava.security.manager=disallow
Exec output and error:
| Output for ./bin/opensearch-plugin:warning: no-jdk distributions that do not bundle a JDK are deprecated and will be removed in a future release
| Error occurred during initialization of VM
| java.lang.InternalError: Could not create SecurityManager
| at java.lang.System.initPhase3(java.base@21.0.6/System.java:2305)
| Caused by: java.lang.ClassNotFoundException: false
| at jdk.internal.loader.BuiltinClassLoader.loadClass(java.base@21.0.6/BuiltinClassLoader.java:641)
| at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(java.base@21.0.6/ClassLoaders.java:188)
| at java.lang.ClassLoader.loadClass(java.base@21.0.6/ClassLoader.java:526)
| at java.lang.Class.forName0(java.base@21.0.6/Native Method)
| at java.lang.Class.forName(java.base@21.0.6/Class.java:534)
| at java.lang.Class.forName(java.base@21.0.6/Class.java:513)
| at java.lang.System.initPhase3(java.base@21.0.6/System.java:2289)