Opensearch failing to start

OpenSearch
,
1

Hello All,

Wondering if you could assist with an RPM-based installation that is failing to start after the basic install recommendations and TLS configuration. Relevant logs and configuration included below, any assistance is much appreciated,

Thanks in advance,
David

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

Rocky Linux release 9.1 (Blue Onyx)
opensearch-2.5.0-1.x86_64

/usr/share/opensearch/jdk/bin/java

openjdk version “17.0.5” 2022-10-18
OpenJDK Runtime Environment Temurin-17.0.5+8 (build 17.0.5+8)
OpenJDK 64-Bit Server VM Temurin-17.0.5+8 (build 17.0.5+8, mixed mode, sharing)

Describe the issue:

When trying to start opensearch.service via systemctl, it fails with

[2023-02-07T17:05:08,743][WARN ][o.o.d.SeedHostsResolver ] [dm-in-odfe-01] failed to resolve host [dm-ineos-odfe-02]
java.net.UnknownHostException: dm-ineos-odfe-02
at java.net.InetAddress$CachedAddresses.get(InetAddress.java:801) ~[?:?]
at java.net.InetAddress.getAllByName0(InetAddress.java:1519) ~[?:?]
at java.net.InetAddress.getAllByName(InetAddress.java:1377) ~[?:?]
at java.net.InetAddress.getAllByName(InetAddress.java:1305) ~[?:?]
…

  • Using RPM based installation as per request (customer doesn’t want docker and offline environment)
  • SELinux disabled
  • Firewalld disabled for testing
  • Name resolution successful between hosts

dm-in-odfe-01 opensearch]$ ping dm-in-odfe-02 -c4
PING dm-ineos-odfe-02 (10.30.0.21) 56(84) bytes of data.
64 bytes from dm-in-odfe-02 (10.30.0.21): icmp_seq=1 ttl=64 time=0.207 ms
64 bytes from dm-in-odfe-02 (10.30.0.21): icmp_seq=2 ttl=64 time=0.201 ms
64 bytes from dm-in-odfe-02 (10.30.0.21): icmp_seq=3 ttl=64 time=0.201 ms
64 bytes from dm-in-odfe-02 (10.30.0.21): icmp_seq=4 ttl=64 time=0.201 ms

Configuration:

cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

10.30.0.20 dm-in-odfe-01
10.30.0.21 dm-in-odfe-02

ip addr | grep inet
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
inet 10.30.0.20/28 brd 10.30.0.31 scope global noprefixroute ens192
inet6 fe80::250:56ff:fea3:e8ce/64 scope link noprefixroute

/etc/opensearch/opensearch.yml

BASE

path.data: /opensearch-data
path.logs: /var/log/opensearch

cluster.name: FPS-DMZ-ELK

node.roles: [ cluster_manager ]
node.name: dm-in-odfe-01
network.host: 10.30.0.20
discovery.seed_hosts: [“dm-in-odfe-01”, “dm-in-odfe-02”]

SECURITY

plugins.security.ssl.transport.pemcert_filepath: /etc/opensearch/certs/odfe-01.pem
plugins.security.ssl.transport.pemkey_filepath: /etc/opensearch/certs/in-odfe-01.key
plugins.security.ssl.transport.pemtrustedcas_filepath: /etc/opensearch/certs/root-ca.pem
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: /etc/opensearch/certs/in-odfe-01.pem
plugins.security.ssl.http.pemkey_filepath: /etc/opensearch/certs/in-odfe-01.key
plugins.security.ssl.http.pemtrustedcas_filepath: /etc/opensearch/certs/root-ca.pem
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn: “- ‘CN=GB,OU=IT,O=IN,L=,ST=FA,C=GB’”
plugins.security.nodes_dn: “- ‘CN=in-odfe-01.in.com,OU=OT,O=IN,L=GR,ST=FA,C=GB’”
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]

Relevant Logs or Screenshots:

/var/log/opensearch/opensearch.log

[2023-02-07T17:05:07,743][WARN ][o.o.d.SeedHostsResolver ] [dm-in-odfe-01] failed to resolve host [dm-in-odfe-02] java.net.UnknownHostException: dm-in-odfe-02

/var/log/opensearch/FPS-DMZ-ELK.log

Caused by: org.opensearch.OpenSearchSecurityException: Error while initializing http SSL layer from PEM: java.security.cert.CertificateParsingException: signed fields invalid
at org.opensearch.security.ssl.DefaultSecurityKeyStore.initHttpSSLConfig(DefaultSecurityKeyStore.java:535) ~[?:?]
at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:262) ~[?:?]
at org.opensearch.security.ssl.DefaultSecurityKeyStore.(DefaultSecurityKeyStore.java:177) ~[?:?]
at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.(OpenSearchSecuritySSLPlugin.java:218) ~[?:?]
at org.opensearch.security.OpenSearchSecurityPlugin.(OpenSearchSecurityPlugin.java:263) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:782) ~[opensearch-2.5.0.jar:2.5.0]
at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:731) ~[opensearch-2.5.0.jar:2.5.0]
at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:533) ~[opensearch-2.5.0.jar:2.5.0]
at org.opensearch.plugins.PluginsService.(PluginsService.java:195) ~[opensearch-2.5.0.jar:2.5.0]
at org.opensearch.node.Node.(Node.java:429) ~[opensearch-2.5.0.jar:2.5.0]
at org.opensearch.node.Node.(Node.java:356) ~[opensearch-2.5.0.jar:2.5.0]
at org.opensearch.bootstrap.Bootstrap$5.(Bootstrap.java:242) ~[opensearch-2.5.0.jar:2.5.0]
at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.5.0.jar:2.5.0]
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.5.0.jar:2.5.0]
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.5.0.jar:2.5.0]
… 6 more
Caused by: java.security.cert.CertificateParsingException: signed fields invalid
at sun.security.x509.X509CertImpl.parse(X509CertImpl.java:1772) ~[?:?]
at sun.security.x509.X509CertImpl.(X509CertImpl.java:183) ~[?:?]
at sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:105) ~[?:?]
at java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:355) ~[?:?]
at org.opensearch.security.support.PemKeyReader.loadCertificateFromFile(PemKeyReader.java:212) ~[?:?]
at org.opensearch.security.ssl.util.CertFromFile.(CertFromFile.java:59) ~[?:?]
at org.opensearch.security.ssl.DefaultSecurityKeyStore.initHttpSSLConfig(DefaultSecurityKeyStore.java:522) ~[?:?]
at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:262) ~[?:?]
at org.opensearch.security.ssl.DefaultSecurityKeyStore.(DefaultSecurityKeyStore.java:177) ~[?:?]
at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.(OpenSearchSecuritySSLPlugin.java:218) ~[?:?]
at org.opensearch.security.OpenSearchSecurityPlugin.(OpenSearchSecurityPlugin.java:263) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:782) ~[opensearch-2.5.0.jar:2.5.0]
at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:731) ~[opensearch-2.5.0.jar:2.5.0]
at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:533) ~[opensearch-2.5.0.jar:2.5.0]
at org.opensearch.plugins.PluginsService.(PluginsService.java:195) ~[opensearch-2.5.0.jar:2.5.0]
at org.opensearch.node.Node.(Node.java:429) ~[opensearch-2.5.0.jar:2.5.0]
at org.opensearch.node.Node.(Node.java:356) ~[opensearch-2.5.0.jar:2.5.0]
at org.opensearch.bootstrap.Bootstrap$5.(Bootstrap.java:242) ~[opensearch-2.5.0.jar:2.5.0]
at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.5.0.jar:2.5.0]
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.5.0.jar:2.5.0]
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.5.0.jar:2.5.0]
… 6 more

2

Hi @dmonagh4n,

First of all, did you edit the log to anonymize the real hostname or is this the original log?
I ask because it is a different hostname than defined in the discovery.seed_hosts variable.

A second thought is that you could disable ipv6 and see if that helps.

Third, you can try an FQDN in the hosts file and in the OpenSearch configuration yml.
If that doesn’t work either, you can set the DNS name that is configured on the DNS server. Maybe OpenSearch will use the DNS server that is configured in your resolve.conf.

And for the last step, I would test if it works with the previous version 2.4.1. Maybe it is a small bug.

Translated with DeepL Translate: The world's most accurate translator (free version)

3

Hi vi4life,

Thanks for the response.

Looking back at the icmp tests displayed - they aren’t a proper reflection of what was in place when testing sorry. I was trying different hostnames along with ones that were actually implemented so it looks like there is a bit of configuration bleed in there.

I can confirm that for testing, I have performed the following with regards to host connectivity

  • Updated discovery.seed_hosts to use both fqdn, single hostname and IP address (I’ve had the most success with the later)
  • Updated /etc/hosts to include both nodes internal IP addresses and all upper,lowercase and fqdn combinations
  • Successfully tested connectivity between nodes with icmp
  • Permitted firewalld services/ports, then disabled firewalld completely during testing
  • Disabled SELinux completely during testing

It does appear to be a strange result between the TLS errors and the FQDN errors.

Also, what I have tried with much more success is; start again with a dnf/yum installation and try the demo configuration and I see the daemons successfully start. This includes with dashboards and nginx configurations without issue, which makes think there could potentially be an issue with the TLS certs used in the cluster, either with regards to my placement within configuration, missing configuration or the certificates themselves. I stuck closely to the documentation for the creation process on purpose, with the exception of the names of the certificates.

Best Regards,
David

4

Hi @dmonagh4n,
what success did you had with setting the IP Address instead of the Hostnames?
Did you used the provided Script from OpenSearch or did you made more changes than the Hostnames?
Could you please share the Certificate itself? It is fine if it is anonymized, i just wanted to have a look at the DN of the Cert and the SAN. Maybe you have to disable Hostname verification just for testing.
plugins.security.ssl.transport.enforce_hostname_verification: false

kind regards

1 Like
5

Hi vi4life,

>> what success did you had with setting the IP Address instead of the Hostnames?

I didn’t see any change there, even to the error message about being unable to resolve the hostname which was pre-dnf/yum install testing with the demo environment. I tried both hostname, fqdn and IP there as well.

>> Did you used the provided Script from OpenSearch or did you made more changes than the Hostnames?

I only made the hostname adjustments along with the configuration listed above to see if there was any fault there.

>> Could you please share the Certificate itself? It is fine if it is anonymized, i just wanted to have a look at the DN of the Cert and the SAN. Maybe you have to disable Hostname verification just for testing.

Granted these aren’t giving you the base64 but hopefully you can see where I was going with them. I can give them over private message or email if it comes to that but I suspect it’s the hostname verification mentioned because these are all going to be self-signed certificates / made with local CA as per the guidance, we won’t be able to do the likes of DNS verification that letsencrypt offer and update the environment every 3 months.

Certificate:
Data:
Version: 3 (0x2)
Serial Number:
4c:ca:33:ce:11:70:94:d6:04:59:ef:1e:b9:34:a9:9a:49:f3:fb:e8
Signature Algorithm: sha256WithRSAEncryption
Issuer: C = , ST = , L = , O = , OU = UNIT, CN = ROOT
Validity
Not Before: Feb 8 12:06:49 2023 GMT
Not After : Feb 7 12:06:49 2025 GMT
Subject: C = , ST = , L = , O = , OU = UNIT, CN = ROOT
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (4096 bit)
Modulus:
00:b7:c0:d4:a8:d8:16:7f:5a:da:1e:d1:21:8b:ad:
d2:53:5b:ef:96:ca:8f:44:c9:b7:67:4d:cb:3b:ab:
df:bc:0b:16:a5:c4:72:ac:d9:ea:48:0b:08:88:85:
89:19:58:8e:07:b9:9d:1b:cd:38:af:bf:e0:e0:35:
82:35:c8:79:9b:22:9e:b3:ba:b9:f9:8b:4f:6a:c3:
7e:f4:2a:f9:66:75:60:8a:74:fc:98:cc:d0:ae:36:
b4:8d:99:a3:6c:ce:3f:20:61:13:a4:f4:c4:9c:a4:
fb:23:ef:71:0c:56:2f:d7:56:15:36:a8:20:c1:f1:
e0:06:03:0c:9a:4a:d3:c2:45:40:a2:68:be:43:35:
ba:52:69:37:36:e5:5c:fc:8b:93:29:9b:1b:ff:65:
28:14:ac:f3:54:d0:d2:c9:75:c0:ea:1c:94:fd:6c:
04:cd:71:21:03:ad:c9:f8:81:45:4a:f7:67:8b:41:
2b:f0:f8:88:20:2a:6c:89:11:79:fb:2a:9d:cd:e6:
67:48:82:85:ac:ca:e9:88:10:d5:89:e7:8b:96:27:
6b:bd:f9:0c:29:ca:d2:c1:82:49:43:ff:f5:21:1d:
7b:52:63:e1:f6:fd:90:08:93:53:ec:44:8c:37:d5:
20:96:19:6a:44:b8:c3:a6:4c:89:ef:3a:56:b5:bc:
a1:18:1f:f9:ab:b9:03:44:95:5c:91:a9:d1:c2:95:
b0:47:5b:b4:df:60:54:a0:ba:34:78:fa:8f:0e:d6:
7b:6e:2b:d3:19:48:51:f1:4d:d3:fd:3c:b0:62:4a:
8c:85:bc:e1:2b:b7:95:c8:d5:27:08:72:7a:16:2b:
cb:66:94:f0:b5:24:99:07:2e:a8:11:3c:71:4d:13:
9d:77:31:a0:ff:c2:2f:33:9e:bb:19:9d:7e:00:69:
17:5d:29:66:a9:85:fd:84:81:51:32:2a:0f:df:e3:
d5:d9:59:4e:d0:0d:d8:29:e5:1c:55:92:78:42:5c:
e5:fe:18:47:81:13:14:6f:79:28:51:be:c8:9a:97:
41:91:19:cd:c3:11:0a:de:af:e4:5b:67:7e:8a:bf:
1c:54:b8:77:41:d5:37:e5:d5:90:a1:85:5a:ce:4d:
a6:cf:f4:12:60:9d:47:66:c3:82:7f:6e:e2:9b:0d:
4e:ae:9c:18:92:c5:8f:6a:88:73:9e:c7:9a:db:fb:
35:a7:ef:43:07:97:a3:c4:3b:35:09:19:5e:7d:f4:
6d:cb:3f:21:cc:43:47:14:97:96:2b:96:83:a7:8b:
e9:43:a4:ca:e2:bc:a5:08:5b:64:53:bc:d7:ea:dc:
63:6c:8e:5d:03:cd:c3:93:c8:f4:ae:e3:3b:16:96:
a0:ed:c5
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Subject Alternative Name:
DNS:in-odfe-01.in.com
X509v3 Subject Key Identifier:
0E:4F:7C:05:F4:E9:65:AE:67:E0:F8:5D:49:1C:E9:3B:C3:56:BC:7D
X509v3 Authority Key Identifier:
CF:0D:5E:63:8E:78:66:7A:E7:7E:67:61:50:E4:C7:95:80:0D:BA:0A
Signature Algorithm: sha256WithRSAEncryption
Signature Value:
01:27:b3:c0:2a:77:b3:af:6b:bf:b8:67:10:2d:34:d1:08:f7:
b6:6a:b2:96:1c:ef:69:31:4d:32:63:49:b2:36:33:e0:90:34:
ce:4a:3f:49:75:c2:b3:8e:b6:ee:8a:fb:21:0a:82:0c:94:d5:
cd:19:d9:4f:c5:7f:01:d2:c9:ff:70:0b:1e:d0:bb:2e:44:b3:
15:c0:a2:55:04:93:37:54:1e:c4:23:96:a1:9f:6a:fc:84:f6:
a3:af:43:04:81:76:65:cb:35:96:1a:d2:5e:cc:40:f0:60:2e:
f5:7f:66:0d:be:09:9a:45:54:d2:6a:25:1b:a5:8f:c1:2f:6a:
2a:57:88:f1:f4:13:02:e6:85:f5:a4:94:dd:66:27:07:d4:9c:
38:57:b7:16:a4:0d:c4:57:87:ce:e8:bd:62:20:2a:4c:63:48:
5f:4e:e8:0e:ba:47:49:2c:2b:7a:77:4f:c8:5e:b5:a8:4e:6c:
d6:12:f7:06:e8:c0:74:a5:ee:4d:b4:6a:8c:50:08:04:25:66:
3e:b2:4f:5b:56:3d:b7:19:58:f2:41:94:f3:68:ef:d7:31:ef:
4e:c5:b3:6a:dd:02:1b:68:8b:f8:3b:a1:0a:81:de:f3:fb:5e:
b2:55:51:8e:a2:ce:87:39:b7:fb:27:76:d7:a4:8c:a4:67:55:
78:c9:5e:a5:c5:cc:fd:dc:7b:62:51:f4:25:04:14:74:d9:8f:
d1:fe:fd:aa:44:69:50:cb:2b:93:b4:a2:cb:4a:f5:49:be:0e:
52:4d:8b:d5:7a:a5:f8:c5:4f:f4:1c:96:74:c6:8c:2f:33:5b:
c6:d1:03:f6:46:b9:d8:6c:5e:5a:e4:5b:a2:91:70:6b:05:f4:
88:9b:17:5f:70:33:b7:0a:ac:1b:04:e1:fd:71:53:01:bd:6a:
10:cc:bf:be:34:1a:72:9a:4c:ad:b6:5f:f3:f5:e2:76:8e:d4:
6f:90:a2:2b:4e:22:dd:25:f0:58:3a:52:f0:f5:cc:b3:61:61:
bf:0f:e9:6d:e0:72:64:9e:fe:2d:af:fb:ba:6d:da:aa:7d:6c:
1b:9d:ae:d8:1a:24:67:81:9f:8e:79:6f:81:83:33:3b:1c:24:
21:77:9b:d4:73:ff:68:20:82:fc:8b:c6:dc:63:d6:a5:81:ec:
f9:c8:bf:8f:87:12:6f:48:9b:f9:6c:74:20:c3:8c:a1:ce:80:
af:d6:bc:17:00:51:10:03:67:7b:6d:d3:1b:d6:e4:6b:b4:58:
8f:fe:18:52:06:da:ea:a9:1e:ce:1e:a0:a5:2e:8b:b2:74:46:
25:9b:6c:11:ce:ac:5a:c3:fd:bc:d2:75:6e:8d:89:d8:04:fb:
bb:51:18:f6:4b:43:22:c2

Best Regards,
David