I’m trying to form an opensearch app on kubernetes and having trouble setting up the certificates.
I’m getting this error on startup (which doesnt fail the pod itself):
The certificates were generated using the bash supplied here:
https://opensearch.org/docs/latest/security-plugin/configuration/generate-certificates/#sample-script
This means that one node is trying to connect to another with
a non-node certificate (no OID or security.nodes_dn incorrect configured) or that someone
is spoofing requests. Check your TLS certificate setup as described here: See https://opendistro.github.io/for-elasticsearch-docs/docs/troubleshoot/tls/]
if I’m making requests from inside the pod it works. but from outside it’s not.
when I make a request from another pod ( I have a service setup)
I get the following error:
io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: <value>
my opensearch.yml:
---
apiVersion: v1
kind: ConfigMap
metadata:
name: opensearch-config
data:
opensearch.yml: |
cluster.name: opensearch-cluster
network.host: "0.0.0.0"
transport.host: "0.0.0.0"
transport.tcp.port: 9300
http.host: "0.0.0.0"
http.port: 9200
action.auto_create_index: true
bootstrap.memory_lock: true
plugins.security.authcz.admin_dn:
- 'subject= CN=ADMIN,OU=UNIT,O=ORG,L=TA,ST=IL,C=CA'
plugins.security.nodes_dn:
- 'subject= CN=*.opensearch-master-headless.search.svc.cluster.local,OU=UNIT,O=ORG,L=TA,ST=IL,C=CA'
- 'subject= CN=*.opensearch-master-headless.search.svc.cluster.local,OU=UNIT,O=ORG,L=TA,ST=IL,C=CA'
- 'subject= CN=*.opensearch-master-headless.search.svc.cluster.local,OU=UNIT,O=ORG,L=TA,ST=IL,C=CA'
plugins.security.allow_unsafe_democertificates: true
plugins.security.ssl.http.enabled: "true"
plugins.security.ssl.http.pemtrustedcas_filepath: {{ .Values.rootCertPathOnMachine }}
plugins.security.ssl.http.pemcert_filepath: {{ .Values.certPathOnMachine }}
plugins.security.ssl.http.pemkey_filepath: {{ .Values.privateKeyPathOnMachine }}
plugins.security.ssl.transport.enabled: "true"
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.transport.resolve_hostname: false
plugins.security.ssl.transport.pemtrustedcas_filepath: {{ .Values.rootCertPathOnMachine }}
plugins.security.ssl.transport.pemcert_filepath: {{ .Values.certPathOnMachine }}
plugins.security.ssl.transport.pemkey_filepath: {{ .Values.privateKeyPathOnMachine }}
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.allow_default_init_securityindex: true
plugins.security.restapi.roles_enabled:
["all_access", "security_rest_api_access"]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices:
[
".opendistro-alerting-config",
".opendistro-alerting-alert*",
".opendistro-anomaly-results*",
".opendistro-anomaly-detector*",
".opendistro-anomaly-checkpoints",
".opendistro-anomaly-detection-state",
".opendistro-reports-*",
".opendistro-notifications-*",
".opendistro-notebooks",
".opendistro-asynchronous-search-response*",
".replication-metadata-store",
]
my deployment:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: {{ .Values.global.name }}
namespace: {{ .Values.global.namespace }}
clusterName: {{ .Values.global.clusterName }}
labels:
app: {{ .Values.global.name }}
annotations:
majorVersion: "{{ include "opensearch.majorVersion" . }}"
spec:
serviceName: "{{ .Values.global.name }}-master-headless"
selector:
matchLabels:
app: {{ .Values.global.name }}
replicas: {{ .Values.replicas }}
template:
metadata:
name: {{ .Values.global.name }}
labels:
app: {{ .Values.global.name }}
role: master
spec:
initContainers:
- name: init-sysctl
image: busybox:latest
command:
- sysctl
- -w
- vm.max_map_count=262144
securityContext:
privileged: true
containers:
- name: "{{.Values.global.name }}-master"
image: opensearchproject/opensearch:1.2.3
imagePullPolicy: IfNotPresent
resources:
limits:
cpu: 1000m
requests:
cpu: 100m
ports:
- containerPort: 9200
name: rest
protocol: TCP
- containerPort: 9300
name: inter-node
protocol: TCP
env:
- name: node.name
value: "{{ .Values.global.name }}"
- name: node.master
value: "true"
- name: cluster.initial_master_nodes
value: "{{ .Values.global.name }}"
- name: discovery.seed_hosts
value: "{{ .Values.global.name }}"
- name: cluster.name
value: "{{ .Values.global.clusterName }}"
- name: ES_JAVA_OPTS
value: "-Xms512m -Xmx512m"
volumeMounts:
- name: {{ .Values.global.name }}
mountPath: /etc/opensearch/data
- name: config
mountPath: /usr/share/opensearch/config/opensearch.yml
subPath: opensearch.yml
- name: node-key
mountPath: {{ .Values.privateKeyPathOnMachine }}
subPath: node-key.pem
readOnly: true
- name: node
mountPath: {{ .Values.certPathOnMachine }}
subPath: node.pem
readOnly: true
- name: root-ca
mountPath: {{ .Values.rootCertPathOnMachine }}
subPath: root-ca.pem
volumes:
- name: config
configMap:
name: opensearch-config
- name: node
secret:
secretName: node
items:
- key: node.pem
path: node.pem
- name: node-key
secret:
secretName: node-key
items:
- key: node-key.pem
path: node-key.pem
- name: root-ca
secret:
secretName: root-ca
items:
- key: root-ca.pem
path: root-ca.pem
volumeClaimTemplates:
- metadata:
name: {{ .Values.global.name }}
labels:
app: {{ .Values.global.name }}
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 10Gi
certificates seems to load ok from secrets.