Hi,
I installed opensearch cluster by yum method.
If I set plugins.security.disabled: true
cluster is up and I can make curl to API.
Now I would like to enable security. I prepared certs for every node, made setting by instruction, but in that case - cluster not initialized.
my opensearch.yml
config
cluster.name: opensearch-cluster
node.name: opensearch-0
node.roles: [ data, ingest, cluster_manager ]
path.data: /opensearch
path.logs: /var/log/opensearch
bootstrap.memory_lock: true
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: ["192.168.11.211", "192.168.11.212" , "192.168.11.213"]
cluster.initial_cluster_manager_nodes: ["192.168.11.211", "192.168.11.212" , "192.168.11.213"]
# manual settings
cluster.max_shards_per_node: 100
action.auto_create_index: false
logger.level: "info"
# (!)
plugins.security.disabled: false
# Transport layer TLS
plugins.security.ssl.transport.pemkey_filepath: opensearch-0-key.pem
plugins.security.ssl.transport.pemcert_filepath: opensearch-0.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
# REST layer TLS
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemkey_filepath: opensearch-0-key.pem
plugins.security.ssl.http.pemcert_filepath: opensearch-0.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
# To identify inter-cluster requests
plugins.security.nodes_dn:
- 'CN=opensearch-0.example.local,OU=UNIT,O=ORG,L=GRUN,C=UR'
- 'CN=*.example.local,OU=UNIT,O=ORG,L=GRUN,C=UR'
# Admin certificate
plugins.security.authcz.admin_dn:
- "CN=ADMIN,OU=UNIT,O=ORG,L=GRUN,ST=TAKO,C=UR"
# (Advanced) OpenSSL
plugins.security.ssl.transport.enable_openssl_if_available: true
plugins.security.ssl.http.enable_openssl_if_available: true
# (Advanced) Hostname verification and DNS lookup
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.transport.resolve_hostname: false
# (Advanced) Client authentication
plugins.security.ssl.http.clientauth_mode: OPTIONAL
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled:
- "all_access"
- "security_rest_api_access"
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices:
- ".plugins-ml-model"
- ".plugins-ml-task"
- ".opendistro-alerting-config"
- ".opendistro-alerting-alert*"
- ".opendistro-anomaly-results*"
- ".opendistro-anomaly-detector*"
- ".opendistro-anomaly-checkpoints"
- ".opendistro-anomaly-detection-state"
- ".opendistro-reports-*"
- ".opensearch-notifications-*"
- ".opensearch-notebooks"
- ".opensearch-observability"
- ".opendistro-asynchronous-search-response*"
- ".replication-metadata-store"
node.max_local_storage_nodes: 3
log shows me
[2022-06-08T23:30:09,785][WARN ][o.o.d.HandshakingTransportAddressConnector] [opensearch-0] handshake failed for [connectToRemoteMasterNode[192.168.11.213:9300]]
org.opensearch.transport.RemoteTransportException: [opensearch-2][192.168.11.213:9300][internal:transport/handshake]
Caused by: org.opensearch.OpenSearchException: Transport client authentication no longer supported.
at org.opensearch.security.ssl.util.ExceptionUtils.createTransportClientNoLongerSupportedException(ExceptionUtils.java:63) ~[?:?]
at org.opensearch.security.transport.SecurityRequestHandler.messageReceivedDecorate(SecurityRequestHandler.java:270) ~[?:?]
at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceived(SecuritySSLRequestHandler.java:153) ~[?:?]
at org.opensearch.security.OpenSearchSecurityPlugin$7$1.messageReceived(OpenSearchSecurityPlugin.java:651) ~[?:?]
at org.opensearch.indexmanagement.rollup.interceptor.RollupInterceptor$interceptHandler$1.messageReceived(RollupInterceptor.kt:118) ~[?:?]
at org.opensearch.performanceanalyzer.transport.PerformanceAnalyzerTransportRequestHandler.messageReceived(PerformanceAnalyzerTransportRequestHandler.java:43) ~[?:?]
at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:103) ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.transport.InboundHandler.handleRequest(InboundHandler.java:249) ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.transport.InboundHandler.messageReceived(InboundHandler.java:132) ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:114) ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:769) ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:175) ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:150) ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:115) ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:94) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1371) ~[?:?]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1234) ~[?:?]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1283) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
[2022-06-08T23:30:09,787][WARN ][o.o.d.HandshakingTransportAddressConnector] [opensearch-0] handshake failed for [connectToRemoteMasterNode[192.168.11.212:9300]]
org.opensearch.transport.RemoteTransportException: [opensearch-1][192.168.11.212:9300][internal:transport/handshake]
Caused by: org.opensearch.OpenSearchException: Transport client authentication no longer supported.
at org.opensearch.security.ssl.util.ExceptionUtils.createTransportClientNoLongerSupportedException(ExceptionUtils.java:63) ~[?:?]
at org.opensearch.security.transport.SecurityRequestHandler.messageReceivedDecorate(SecurityRequestHandler.java:270) ~[?:?]
at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceived(SecuritySSLRequestHandler.java:153) ~[?:?]
at org.opensearch.security.OpenSearchSecurityPlugin$7$1.messageReceived(OpenSearchSecurityPlugin.java:651) ~[?:?]
at org.opensearch.indexmanagement.rollup.interceptor.RollupInterceptor$interceptHandler$1.messageReceived(RollupInterceptor.kt:118) ~[?:?]
at org.opensearch.performanceanalyzer.transport.PerformanceAnalyzerTransportRequestHandler.messageReceived(PerformanceAnalyzerTransportRequestHandler.java:43) ~[?:?]
at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:103) ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.transport.InboundHandler.handleRequest(InboundHandler.java:249) ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.transport.InboundHandler.messageReceived(InboundHandler.java:132) ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:114) ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:769) ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:175) ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:150) ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:115) ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:94) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1371) ~[?:?]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1234) ~[?:?]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1283) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
[2022-06-08T23:30:09,881][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-0] Exception while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, AUDIT] (index=.opendistro_security)
org.opensearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
at org.opensearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:204) ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:190) ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:81) ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:58) ~[opensearch-2.0.0.jar:2.0.0]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:204) [opensearch-2.0.0.jar:2.0.0]
at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118) [opensearch-index-management-2.0.0.0.jar:2.0.0.0]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:202) [opensearch-2.0.0.jar:2.0.0]
at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:78) [opensearch-performance-analyzer-2.0.0.0.jar:2.0.0.0]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:202) [opensearch-2.0.0.jar:2.0.0]
at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:240) [opensearch-security-2.0.0.0.jar:2.0.0.0]
at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:157) [opensearch-security-2.0.0.0.jar:2.0.0.0]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:202) [opensearch-2.0.0.jar:2.0.0]
at org.opensearch.action.support.TransportAction.execute(TransportAction.java:174) [opensearch-2.0.0.jar:2.0.0]
at org.opensearch.action.support.TransportAction.execute(TransportAction.java:102) [opensearch-2.0.0.jar:2.0.0]
at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) [opensearch-2.0.0.jar:2.0.0]
at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) [opensearch-2.0.0.jar:2.0.0]
at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:423) [opensearch-2.0.0.jar:2.0.0]
at org.opensearch.client.support.AbstractClient.multiGet(AbstractClient.java:539) [opensearch-2.0.0.jar:2.0.0]
at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.loadAsync(ConfigurationLoaderSecurity7.java:211) [opensearch-security-2.0.0.0.jar:2.0.0.0]
at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.load(ConfigurationLoaderSecurity7.java:102) [opensearch-security-2.0.0.0.jar:2.0.0.0]
at org.opensearch.security.configuration.ConfigurationRepository.getConfigurationsFromIndex(ConfigurationRepository.java:375) [opensearch-security-2.0.0.0.jar:2.0.0.0]
at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration0(ConfigurationRepository.java:321) [opensearch-security-2.0.0.0.jar:2.0.0.0]
at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:306) [opensearch-security-2.0.0.0.jar:2.0.0.0]
at org.opensearch.security.configuration.ConfigurationRepository$1.run(ConfigurationRepository.java:166) [opensearch-security-2.0.0.0.jar:2.0.0.0]
at java.lang.Thread.run(Thread.java:833) [?:?]
[2022-06-08T23:30:09,953][ERROR][o.o.s.t.SecurityRequestHandler] [opensearch-0] OpenSearchException[Transport client authentication no longer supported.]
[2022-06-08T23:30:10,032][ERROR][o.o.s.t.SecurityRequestHandler] [opensearch-0] OpenSearchException[Transport client authentication no longer supported.]
I’ve tryed to set transport_enabled
to false
, but it did not help
cat /etc/opensearch/opensearch-security/config.yml
http_enabled: true
transport_enabled: false
Maybe someone can help me to start cluster with enabled security