Opensearch Timeout upon systemctl start

nardixcx · January 30, 2024, 6:33am

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Opensearch 2.3.0

Describe the issue:
Not able to start opensearch service using “systemctl start opensearch”

Configuration:
opensearch.yml
→

network.host: 0.0.0.0
discovery.type: single-node

path.data: /data/opensearch/data
path.logs: /var/log/opensearch

Relevant Logs or Screenshots:

[2024-01-29T21:22:12,124][INFO ][o.o.n.Node               ] [instance001] version[2.3.0-SNAPSHOT], pid[25633], build[tar/unknown/2024-01-29T04:37:31.160217Z], OS[Linux/5.4.17-2136.316.7.el7uek.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/11.0.20/11.0.20+9-LTS-256]
[2024-01-29T21:22:12,126][INFO ][o.o.n.Node               ] [instance001] JVM home [/opt/jdk11], using bundled JDK [false]
[2024-01-29T21:22:12,126][INFO ][o.o.n.Node               ] [instance001] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms30720m, -Xmx30720m, -Djdk.tls.ephemeralDHKeySize=2048, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -XX:+UseG1GC, -XX:G1ReservePercent=15, -XX:InitiatingHeapOccupancyPercent=30, -XX:+AlwaysPreTouch, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/data/opensearch/heapdump, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.formatMsgNoLookups=true, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/data/opensearch/tmp, -XX:ErrorFile=/var/log/opensearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/opensearch/gc.log:utctime,pid,tags:filecount=2,filesize=32m, -Djava.locale.providers=COMPAT, -XX:+FlightRecorder, -XX:MaxDirectMemorySize=16106127360, -Dopensearch.path.home=/usr/share/opensearch, -Dopensearch.path.conf=/etc/opensearch, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]
[2024-01-29T21:22:12,126][WARN ][o.o.n.Node               ] [instance001] version [2.3.0-SNAPSHOT] is a pre-release version of OpenSearch and is not suitable for production
[2024-01-29T21:22:12,864][INFO ][o.o.i.r.ReindexPlugin    ] [instance001] ReindexPlugin reloadSPI called
[2024-01-29T21:22:12,865][INFO ][o.o.i.r.ReindexPlugin    ] [instance001] Unable to find any implementation for RemoteReindexExtension
[2024-01-29T21:22:12,877][INFO ][o.o.p.PluginsService     ] [instance001] loaded module [aggs-matrix-stats]
[2024-01-29T21:22:12,877][INFO ][o.o.p.PluginsService     ] [instance001] loaded module [analysis-common]
[2024-01-29T21:22:12,877][INFO ][o.o.p.PluginsService     ] [instance001] loaded module [geo]
[2024-01-29T21:22:12,877][INFO ][o.o.p.PluginsService     ] [instance001] loaded module [ingest-common]
[2024-01-29T21:22:12,877][INFO ][o.o.p.PluginsService     ] [instance001] loaded module [ingest-geoip]
[2024-01-29T21:22:12,877][INFO ][o.o.p.PluginsService     ] [instance001] loaded module [ingest-user-agent]
[2024-01-29T21:22:12,877][INFO ][o.o.p.PluginsService     ] [instance001] loaded module [lang-expression]
[2024-01-29T21:22:12,878][INFO ][o.o.p.PluginsService     ] [instance001] loaded module [lang-mustache]
[2024-01-29T21:22:12,878][INFO ][o.o.p.PluginsService     ] [instance001] loaded module [lang-painless]
[2024-01-29T21:22:12,878][INFO ][o.o.p.PluginsService     ] [instance001] loaded module [mapper-extras]
[2024-01-29T21:22:12,878][INFO ][o.o.p.PluginsService     ] [instance001] loaded module [opensearch-dashboards]
[2024-01-29T21:22:12,878][INFO ][o.o.p.PluginsService     ] [instance001] loaded module [parent-join]
[2024-01-29T21:22:12,878][INFO ][o.o.p.PluginsService     ] [instance001] loaded module [percolator]
[2024-01-29T21:22:12,878][INFO ][o.o.p.PluginsService     ] [instance001] loaded module [rank-eval]
[2024-01-29T21:22:12,878][INFO ][o.o.p.PluginsService     ] [instance001] loaded module [reindex]
[2024-01-29T21:22:12,878][INFO ][o.o.p.PluginsService     ] [instance001] loaded module [repository-url]
[2024-01-29T21:22:12,879][INFO ][o.o.p.PluginsService     ] [instance001] loaded module [systemd]
[2024-01-29T21:22:12,879][INFO ][o.o.p.PluginsService     ] [instance001] loaded module [test-delayed-aggs]
[2024-01-29T21:22:12,879][INFO ][o.o.p.PluginsService     ] [instance001] loaded module [transport-netty4]
[2024-01-29T21:22:12,880][INFO ][o.o.p.PluginsService     ] [instance001] no plugins loaded
[2024-01-29T21:22:12,924][INFO ][o.o.e.NodeEnvironment    ] [instance001] using [1] data paths, mounts [[/local/volume/data (/dev/sdc)]], net usable_space [466gb], net total_space [491gb], types [ext4]
[2024-01-29T21:22:12,925][INFO ][o.o.e.NodeEnvironment    ] [instance001] heap size [30gb], compressed ordinary object pointers [true]
[2024-01-29T21:22:13,040][INFO ][o.o.n.Node               ] [instance001] node name [instance001], node ID [kHo4T_nQRj6h-LzAwBGuDQ], cluster name [opensearch], roles [ingest, remote_cluster_client, data, cluster_manager]
[2024-01-29T21:22:16,351][INFO ][o.o.t.NettyAllocator     ] [instance001] creating NettyAllocator with the following configs: [name=opensearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={opensearch.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=8mb}]
[2024-01-29T21:22:16,469][INFO ][o.o.d.DiscoveryModule    ] [instance001] using discovery type [single-node] and seed hosts providers [settings]
[2024-01-29T21:22:16,783][WARN ][o.o.g.DanglingIndicesState] [instance001] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2024-01-29T21:22:16,974][INFO ][o.o.n.Node               ] [instance001] initialized
[2024-01-29T21:22:16,974][INFO ][o.o.n.Node               ] [instance001] starting ...
[2024-01-29T21:22:17,116][INFO ][o.o.t.TransportService   ] [instance001] publish_address {10.56.12.32:9300}, bound_addresses {0.0.0.0:9300}
[2024-01-29T21:22:17,309][INFO ][o.o.c.c.Coordinator      ] [instance001] cluster UUID [Ma7A9F2aQiS0DrMapO9j0Q]
[2024-01-29T21:22:17,461][INFO ][o.o.c.s.MasterService    ] [instance001] elected-as-cluster-manager ([1] nodes joined)[{instance001}{kHo4T_nQRj6h-LzAwBGuDQ}{xnVziUBUQ4aDfXgVaZyW8Q}{10.56.12.32}{10.56.12.32:9300}{dimr}{shard_indexing_pressure_enabled=true} elect leader, _BECOME_CLUSTER_MANAGER_TASK_, _FINISH_ELECTION_], term: 9, version: 17, delta: cluster-manager node changed {previous [], current [{instance001}{kHo4T_nQRj6h-LzAwBGuDQ}{xnVziUBUQ4aDfXgVaZyW8Q}{10.56.12.32}{10.56.12.32:9300}{dimr}{shard_indexing_pressure_enabled=true}]}
[2024-01-29T21:22:17,512][INFO ][o.o.c.s.ClusterApplierService] [instance001] cluster-manager node changed {previous [], current [{instance001}{kHo4T_nQRj6h-LzAwBGuDQ}{xnVziUBUQ4aDfXgVaZyW8Q}{10.56.12.32}{10.56.12.32:9300}{dimr}{shard_indexing_pressure_enabled=true}]}, term: 9, version: 17, reason: Publication{term=9, version=17}
[2024-01-29T21:22:17,538][INFO ][o.o.h.AbstractHttpServerTransport] [instance001] publish_address {10.56.12.32:9200}, bound_addresses {0.0.0.0:9200}
[2024-01-29T21:22:17,538][INFO ][o.o.n.Node               ] [instance001] started
[2024-01-29T21:22:17,555][INFO ][o.o.g.GatewayService     ] [instance001] recovered [0] indices into cluster_state
[2024-01-29T21:25:06,379][INFO ][o.o.n.Node               ] [instance001] stopping ...
[2024-01-29T21:25:06,396][INFO ][o.o.n.Node               ] [instance001] stopped
[2024-01-29T21:25:06,396][INFO ][o.o.n.Node               ] [instance001] closing ...
[2024-01-29T21:25:06,406][INFO ][o.o.n.Node               ] [instance001] closed

Jan 29 21:41:01 instance001 kernel: audit: type=1006 audit(1706593261.354:214638): pid=30452 uid=0 old-auid=4294967295 auid=728 tty=(none) old-ses=4294967295 ses=23873 res=1
Jan 29 21:41:01 instance001 kernel: audit: type=1105 audit(1706593261.354:214639): pid=30451 uid=0 auid=728 ses=23872 msg='op=PAM:session_open grantors=pam_loginuid,pam_mkhomedir,pam_keyinit,pam_limits,pam_ldap acct="svc_cmdb_agent" exe="/
Jan 29 21:41:01 instance001 kernel: audit: type=1110 audit(1706593261.355:214640): pid=30451 uid=0 auid=728 ses=23872 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="svc_cmdb_agent" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron
Jan 29 21:41:01 instance001 kernel: audit: type=1105 audit(1706593261.355:214641): pid=30452 uid=0 auid=728 ses=23873 msg='op=PAM:session_open grantors=pam_loginuid,pam_mkhomedir,pam_keyinit,pam_limits,pam_ldap acct="svc_cmdb_agent" exe="/
Jan 29 21:41:01 instance001 kernel: audit: type=1110 audit(1706593261.355:214642): pid=30452 uid=0 auid=728 ses=23873 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="svc_cmdb_agent" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron
Jan 29 21:41:10 instance001 kernel: kauditd_printk_skb: 2 callbacks suppressed
Jan 29 21:41:10 instance001 kernel: audit: type=1123 audit(1706593270.371:214645): pid=30561 uid=996 auid=4294967295 ses=4294967295 msg='cwd="/var/lib/oracle-cloud-agent" cmd=79756D202D2D64697361626C657265706F3D2A20696E7374616C6C202D79202F
Jan 29 21:41:10 instance001 kernel: audit: type=1123 audit(1706593270.393:214646): pid=30562 uid=996 auid=4294967295 ses=4294967295 msg='cwd="/var/lib/oracle-cloud-agent" cmd=2F62696E2F73797374656D63746C2063617420756E69666965642D6D6F6E6974
Jan 29 21:41:27 instance001 rsyslogd[9184]: cannot resolve hostname 'syslog001.lab.li.nsgbu.internal' [v8.2208.0 try https://www.rsyslog.com/e/2027 ]
Jan 29 21:41:27 instance001 rsyslogd[9184]: cannot resolve hostname 'syslog.lab.li.nsgbu.internal' [v8.2208.0 try https://www.rsyslog.com/e/2027 ]
Jan 29 21:41:27 instance001 rsyslogd[9184]: cannot resolve hostname 'syslog001.lab.li.nsgbu.internal' [v8.2208.0 try https://www.rsyslog.com/e/2027 ]
Jan 29 21:41:31 instance001 kernel: audit: type=1104 audit(1706593291.359:214647): pid=30451 uid=0 auid=728 ses=23872 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="svc_cmdb_agent" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron
Jan 29 21:41:31 instance001 kernel: audit: type=1106 audit(1706593291.360:214648): pid=30451 uid=0 auid=728 ses=23872 msg='op=PAM:session_close grantors=pam_loginuid,pam_mkhomedir,pam_keyinit,pam_limits,pam_ldap acct="svc_cmdb_agent" exe="
Jan 29 21:41:57 instance001 rsyslogd[9184]: cannot resolve hostname 'syslog001.lab.li.nsgbu.internal' [v8.2208.0 try https://www.rsyslog.com/e/2027 ]
Jan 29 21:41:57 instance001 rsyslogd[9184]: cannot resolve hostname 'syslog001.lab.li.nsgbu.internal' [v8.2208.0 try https://www.rsyslog.com/e/2027 ]
Jan 29 21:42:01 instance001 CROND[30701]: (root) CMD (/opt/opsta/cmdb-agent/watchdog.sh cmdb_agent)
Jan 29 21:42:01 instance001 CROND[30702]: (svc_cmdb_agent) CMD (source /opt/opsta/cmdb-agent/scripts/monitoring_utils.sh && resend_cached_data)
Jan 29 21:42:01 instance001 CROND[30703]: (svc_cmdb_agent) CMD (sleep 30 && source /opt/opsta/cmdb-agent/scripts/monitoring_utils.sh && resend_cached_data)
Jan 29 21:42:01 instance001 kernel: audit: type=1101 audit(1706593321.364:214649): pid=30698 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:accounting grantors=pam_access,pam_unix,pam_localuser acct="root" exe="/usr/sbin/crond" hostname=
Jan 29 21:42:01 instance001 kernel: audit: type=1101 audit(1706593321.365:214650): pid=30699 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:accounting grantors=pam_access,pam_unix,pam_localuser acct="svc_cmdb_agent" exe="/usr/sbin/crond"
Jan 29 21:42:01 instance001 kernel: audit: type=1103 audit(1706593321.365:214651): pid=30698 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cr
Jan 29 21:42:01 instance001 kernel: audit: type=1103 audit(1706593321.365:214652): pid=30699 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="svc_cmdb_agent" exe="/usr/sbin/crond" hostname=? addr=? t
Jan 29 21:42:01 instance001 kernel: audit: type=1101 audit(1706593321.365:214653): pid=30700 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:accounting grantors=pam_access,pam_unix,pam_localuser acct="svc_cmdb_agent" exe="/usr/sbin/crond"
Jan 29 21:42:01 instance001 kernel: audit: type=1103 audit(1706593321.365:214654): pid=30700 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="svc_cmdb_agent" exe="/usr/sbin/crond" hostname=? addr=? t
Jan 29 21:42:01 instance001 kernel: audit: type=1006 audit(1706593321.365:214655): pid=30699 uid=0 old-auid=4294967295 auid=728 tty=(none) old-ses=4294967295 ses=23874 res=1
Jan 29 21:42:01 instance001 kernel: audit: type=1006 audit(1706593321.365:214656): pid=30698 uid=0 old-auid=4294967295 auid=0 tty=(none) old-ses=4294967295 ses=23875 res=1
Jan 29 21:42:01 instance001 kernel: audit: type=1006 audit(1706593321.365:214657): pid=30700 uid=0 old-auid=4294967295 auid=728 tty=(none) old-ses=4294967295 ses=23876 res=1
Jan 29 21:42:01 instance001 kernel: audit: type=1105 audit(1706593321.365:214658): pid=30699 uid=0 auid=728 ses=23874 msg='op=PAM:session_open grantors=pam_loginuid,pam_mkhomedir,pam_keyinit,pam_limits,pam_ldap acct="svc_cmdb_agent" exe="/
Jan 29 21:42:27 instance001 rsyslogd[9184]: cannot resolve hostname 'syslog001.lab.li.nsgbu.internal' [v8.2208.0 try https://www.rsyslog.com/e/2027 ]
Jan 29 21:42:27 instance001 rsyslogd[9184]: cannot resolve hostname 'syslog001.lab.li.nsgbu.internal' [v8.2208.0 try https://www.rsyslog.com/e/2027 ]
Jan 29 21:42:31 instance001 kernel: kauditd_printk_skb: 9 callbacks suppressed
Jan 29 21:42:31 instance001 kernel: audit: type=1104 audit(1706593351.371:214668): pid=30699 uid=0 auid=728 ses=23874 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="svc_cmdb_agent" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron
Jan 29 21:42:31 instance001 kernel: audit: type=1106 audit(1706593351.371:214669): pid=30699 uid=0 auid=728 ses=23874 msg='op=PAM:session_close grantors=pam_loginuid,pam_mkhomedir,pam_keyinit,pam_limits,pam_ldap acct="svc_cmdb_agent" exe="
Jan 29 21:42:43 instance001 CROND[27605]: (root) CMDOUT (local:)
Jan 29 21:42:43 instance001 CROND[27605]: (root) CMDOUT (    None)
Jan 29 21:42:44 instance001 kernel: audit: type=1104 audit(1706593364.378:214670): pid=27605 uid=0 auid=0 ses=23843 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success
Jan 29 21:42:44 instance001 kernel: audit: type=1106 audit(1706593364.378:214671): pid=27605 uid=0 auid=0 ses=23843 msg='op=PAM:session_close grantors=pam_loginuid,pam_mkhomedir,pam_keyinit,pam_limits,pam_ldap acct="root" exe="/usr/sbin/cr
Jan 29 21:42:57 instance001 rsyslogd[9184]: cannot resolve hostname 'syslog001.lab.li.nsgbu.internal' [v8.2208.0 try https://www.rsyslog.com/e/2027 ]
Jan 29 21:42:57 instance001 rsyslogd[9184]: cannot resolve hostname 'syslog.lab.li.nsgbu.internal' [v8.2208.0 try https://www.rsyslog.com/e/2027 ]
Jan 29 21:42:57 instance001 rsyslogd[9184]: cannot resolve hostname 'syslog001.lab.li.nsgbu.internal' [v8.2208.0 try https://www.rsyslog.com/e/2027 ]
Jan 29 21:43:01 instance001 CROND[30981]: (svc_cmdb_agent) CMD (source /opt/opsta/cmdb-agent/scripts/monitoring_utils.sh && resend_cached_data)
Jan 29 21:43:01 instance001 CROND[30980]: (svc_cmdb_agent) CMD (sleep 30 && source /opt/opsta/cmdb-agent/scripts/monitoring_utils.sh && resend_cached_data)
Jan 29 21:43:01 instance001 kernel: audit: type=1101 audit(1706593381.382:214672): pid=30978 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:accounting grantors=pam_access,pam_unix,pam_localuser acct="svc_cmdb_agent" exe="/usr/sbin/crond"
Jan 29 21:43:01 instance001 kernel: audit: type=1101 audit(1706593381.382:214673): pid=30979 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:accounting grantors=pam_access,pam_unix,pam_localuser acct="svc_cmdb_agent" exe="/usr/sbin/crond"
Jan 29 21:43:01 instance001 kernel: audit: type=1103 audit(1706593381.382:214674): pid=30978 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="svc_cmdb_agent" exe="/usr/sbin/crond" hostname=? addr=? t
Jan 29 21:43:01 instance001 kernel: audit: type=1103 audit(1706593381.382:214675): pid=30979 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="svc_cmdb_agent" exe="/usr/sbin/crond" hostname=? addr=? t
Jan 29 21:43:01 instance001 kernel: audit: type=1006 audit(1706593381.382:214676): pid=30978 uid=0 old-auid=4294967295 auid=728 tty=(none) old-ses=4294967295 ses=23877 res=1
Jan 29 21:43:01 instance001 kernel: audit: type=1006 audit(1706593381.382:214677): pid=30979 uid=0 old-auid=4294967295 auid=728 tty=(none) old-ses=4294967295 ses=23878 res=1
Jan 29 21:43:01 instance001 kernel: audit: type=1105 audit(1706593381.383:214678): pid=30978 uid=0 auid=728 ses=23877 msg='op=PAM:session_open grantors=pam_loginuid,pam_mkhomedir,pam_keyinit,pam_limits,pam_ldap acct="svc_cmdb_agent" exe="/
Jan 29 21:43:01 instance001 kernel: audit: type=1105 audit(1706593381.383:214679): pid=30979 uid=0 auid=728 ses=23878 msg='op=PAM:session_open grantors=pam_loginuid,pam_mkhomedir,pam_keyinit,pam_limits,pam_ldap acct="svc_cmdb_agent" exe="/
Jan 29 21:43:01 instance001 kernel: audit: type=1110 audit(1706593381.383:214680): pid=30978 uid=0 auid=728 ses=23877 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="svc_cmdb_agent" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron
Jan 29 21:43:01 instance001 kernel: audit: type=1110 audit(1706593381.383:214681): pid=30979 uid=0 auid=728 ses=23878 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="svc_cmdb_agent" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron
Jan 29 21:43:27 instance001 rsyslogd[9184]: cannot resolve hostname 'syslog001.lab.li.nsgbu.internal' [v8.2208.0 try https://www.rsyslog.com/e/2027 ]
Jan 29 21:43:27 instance001 rsyslogd[9184]: cannot resolve hostname 'syslog001.lab.li.nsgbu.internal' [v8.2208.0 try https://www.rsyslog.com/e/2027 ]
Jan 29 21:43:31 instance001 kernel: kauditd_printk_skb: 2 callbacks suppressed
Jan 29 21:43:31 instance001 kernel: audit: type=1104 audit(1706593411.388:214684): pid=30978 uid=0 auid=728 ses=23877 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="svc_cmdb_agent" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron
Jan 29 21:43:31 instance001 kernel: audit: type=1106 audit(1706593411.389:214685): pid=30978 uid=0 auid=728 ses=23877 msg='op=PAM:session_close grantors=pam_loginuid,pam_mkhomedir,pam_keyinit,pam_limits,pam_ldap acct="svc_cmdb_agent" exe="
Jan 29 21:43:52 instance001 rsyslogd[9184]: -- MARK --
Jan 29 21:43:54 instance001 systemd[1]: opensearch.service start operation timed out. Terminating.
lines 957-1015/1025 100%

pablo · January 30, 2024, 9:27am

@nardixcx The syslog timestamps don’t match the ones in the OpenSearch logs.

Is this a full opensearch.yml file? Did you disable the security plugin?

nardixcx · February 1, 2024, 1:01am

yes I have. I’m trying to run it as simple as possible. Im using min version of opensearch btw

nardixcx · February 1, 2024, 1:03am

@pablo sorry for the timestamp. upon checking, its the same output even i get the results in one time

pablo · February 1, 2024, 3:21pm

@nardixcx Is this is a fresh installation? If yes, could you share all the commands you’ve used to install the OpenSearch node?

Which Linux distro did you use?

nardixcx · February 5, 2024, 10:16pm

@pablo I’m using ubuntu. the command is yum localinstall opensearch.rpm

pablo · March 5, 2024, 11:29am

nardixcx:

[2024-01-29T21:22:17,555][INFO ][o.o.g.GatewayService     ] [instance001] recovered [0] indices into cluster_state
[2024-01-29T21:25:06,379][INFO ][o.o.n.Node               ] [instance001] stopping ...

I see a gap of 3 minutes between these two entries. OpenSearch service is usually very chatty. Did you truncate the logs? Is it nothing happening for 3 minutes?

nardixcx · May 5, 2024, 5:51am

I fixed this by updating opensearch.service for Execstart

ymzhangco · May 28, 2024, 8:06pm

How did you fix it? I am having the same thing. Systemd timed out for opensearch service.

Thanks,

Topic		Replies	Views
Opensearch.service: Failed with result 'timeout' OpenSearch troubleshoot	4	144	September 25, 2024
Job for opensearch.service failed because a timeout was exceeded OpenSearch troubleshoot	6	1805	March 1, 2024
Opensearch experimental installation all in one host server OpenSearch troubleshoot	2	645	March 11, 2024
Opensearch.service: Main process exited, code=exited, status=1/FAILURE \| opensearch.service: Failed with result 'exit-code'. \|Failed to start OpenSearch Community	0	1275	May 21, 2024
Opensearch failing to start OpenSearch troubleshoot , install	4	6016	February 15, 2023

Opensearch Timeout upon systemctl start

Related topics