Systemd-entrypoint DefaultDispatcher-worker ERROR Could not define attribute view on path "/var/log/opensearch/opensearch_server.json"

opoplawski · August 11, 2023, 10:45pm

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

opensearch-2.9.0-1.x86_64
Alma Linux 8.8

Describe the issue:

Getting these messages in the logs a few times a day

Aug 10 00:00:00 systemd-entrypoint[697]: 2023-08-10 00:00:00,433 DefaultDispatcher-worker-5 ERROR Could not define attribute view on path "/var/log/opensearch/opensearch_server.json" got access denied ("java.lang.RuntimePermission" "accessUserInformation") java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "accessUserInformation")
Aug 10 00:00:00 systemd-entrypoint[697]:         at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:485)
Aug 10 00:00:00 systemd-entrypoint[697]:         at java.base/java.security.AccessController.checkPermission(AccessController.java:1068)
Aug 10 00:00:00 systemd-entrypoint[697]:         at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:416)
Aug 10 00:00:00 systemd-entrypoint[697]:         at java.base/sun.nio.fs.UnixFileAttributeViews$Posix.checkWriteExtended(UnixFileAttributeViews.java:195)
Aug 10 00:00:00 systemd-entrypoint[697]:         at java.base/sun.nio.fs.UnixFileAttributeViews$Posix.setMode(UnixFileAttributeViews.java:264)
Aug 10 00:00:00 systemd-entrypoint[697]:         at java.base/sun.nio.fs.UnixFileAttributeViews$Posix.setPermissions(UnixFileAttributeViews.java:299)
Aug 10 00:00:00 systemd-entrypoint[697]:         at org.apache.logging.log4j.core.util.FileUtils.defineFilePosixAttributeView(FileUtils.java:177)
Aug 10 00:00:00 systemd-entrypoint[697]:         at org.apache.logging.log4j.core.appender.FileManager.defineAttributeView(FileManager.java:215)
Aug 10 00:00:00 systemd-entrypoint[697]:         at org.apache.logging.log4j.core.appender.FileManager.createOutputStream(FileManager.java:202)
Aug 10 00:00:00 systemd-entrypoint[697]:         at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.createFileAfterRollover(RollingFileManager.java:419)
Aug 10 00:00:00 systemd-entrypoint[697]:         at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.rollover(RollingFileManager.java:396)
Aug 10 00:00:00 systemd-entrypoint[697]:         at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.checkRollover(RollingFileManager.java:308)
Aug 10 00:00:00 systemd-entrypoint[697]:         at org.apache.logging.log4j.core.appender.RollingFileAppender.append(RollingFileAppender.java:311)
Aug 10 00:00:00 systemd-entrypoint[697]:         at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:161)
Aug 10 00:00:00 systemd-entrypoint[697]:         at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:134)
Aug 10 00:00:00 systemd-entrypoint[697]:         at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:125)
Aug 10 00:00:00 systemd-entrypoint[697]:         at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:89)
Aug 10 00:00:00 systemd-entrypoint[697]:         at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:542)
Aug 10 00:00:00 systemd-entrypoint[697]:         at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:500)
Aug 10 00:00:00 systemd-entrypoint[697]:         at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:483)
Aug 10 00:00:00 systemd-entrypoint[697]:         at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:417)
Aug 10 00:00:00 systemd-entrypoint[697]:         at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:82)
Aug 10 00:00:00 systemd-entrypoint[697]:         at org.apache.logging.log4j.core.Logger.log(Logger.java:161)
Aug 10 00:00:00 systemd-entrypoint[697]:         at org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2205)
Aug 10 00:00:00 systemd-entrypoint[697]:         at org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2159)
Aug 10 00:00:00 systemd-entrypoint[697]:         at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2142)
Aug 10 00:00:00 systemd-entrypoint[697]:         at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2017)
Aug 10 00:00:00 systemd-entrypoint[697]:         at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1983)
Aug 10 00:00:00 systemd-entrypoint[697]:         at org.apache.logging.log4j.spi.AbstractLogger.in
fo(AbstractLogger.java:1320)
Aug 10 00:00:00 systemd-entrypoint[697]:         at org.opensearch.indexmanagement.spi.indexstatemanagement.Step.preExecute(Step.kt:24)
Aug 10 00:00:00 systemd-entrypoint[697]:         at org.opensearch.indexmanagement.indexstatemanagement.ManagedIndexRunner$runManagedIndexConfig$2.invokeSuspend(ManagedIndexRunner.kt:432)
Aug 10 00:00:00 systemd-entrypoint[697]:         at org.opensearch.indexmanagement.indexstatemanagement.ManagedIndexRunner$runManagedIndexConfig$2.invoke(ManagedIndexRunner.kt)
Aug 10 00:00:00 systemd-entrypoint[697]:         at org.opensearch.indexmanagement.opensearchapi.OpenSearchExtensionsKt$withClosableContext$2.invokeSuspend(OpenSearchExtensions.kt:316)
Aug 10 00:00:00 systemd-entrypoint[697]:         at org.opensearch.indexmanagement.opensearchapi.OpenSearchExtensionsKt$withClosableContext$2.invoke(OpenSearchExtensions.kt)
Aug 10 00:00:00 systemd-entrypoint[697]:         at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturn(Undispatched.kt:91)
Aug 10 00:00:00 systemd-entrypoint[697]:         at kotlinx.coroutines.BuildersKt__Builders_commonKt.withContext(Builders.common.kt:160)
Aug 10 00:00:00 systemd-entrypoint[697]:         at kotlinx.coroutines.BuildersKt.withContext(Unknown Source)
Aug 10 00:00:00 systemd-entrypoint[697]:         at org.opensearch.indexmanagement.opensearchapi.OpenSearchExtensionsKt.withClosableContext(OpenSearchExtensions.kt:316)
Aug 10 00:00:00 systemd-entrypoint[697]:         at org.opensearch.indexmanagement.indexstatemanagement.ManagedIndexRunner.runManagedIndexConfig(ManagedIndexRunner.kt:427)
Aug 10 00:00:00 systemd-entrypoint[697]:         at org.opensearch.indexmanagement.indexstatemanagement.ManagedIndexRunner.access$runManagedIndexConfig(ManagedIndexRunner.kt:108)
Aug 10 00:00:00 systemd-entrypoint[697]:         at org.opensearch.indexmanagement.indexstatemanagement.ManagedIndexRunner$runManagedIndexConfig$1.invokeSuspend(ManagedIndexRunner.kt)
Aug 10 00:00:00 systemd-entrypoint[697]:         at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
Aug 10 00:00:00 systemd-entrypoint[697]:         at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
Aug 10 00:00:00 systemd-entrypoint[697]:         at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
Aug 10 00:00:00 systemd-entrypoint[697]:         at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738)
Aug 10 00:00:00 systemd-entrypoint[697]:         at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
Aug 10 00:00:00 systemd-entrypoint[697]:         at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)

ls -l /var/log/opensearch/opensearch_server.json
-rw-r--r--. 1 opensearch opensearch 74537139 Aug 11 15:44 /var/log/opensearch/opensearch_server.json

Gsmitt · August 12, 2023, 4:39am

Hey @opoplawski

what permission do you have on /var/log ? is that the only log file opensearch can not access?

becks · August 14, 2023, 7:47am

Getting the same messages, once for the opensearch.log and for opensearch_server.json.
ERROR Could not define attribute view ... got access denied

It seems that it happens during the daily log rollover when the files are in locked state probably?
And yes these are the only files opensearch gets an access denied here for us.

ls -l opensearch.log opensearch_server.json
-rw-r--r-- 1 opensearch opensearch    25761 Aug 14 08:54 opensearch.log
-rw-r--r-- 1 opensearch opensearch    67733 Aug 14 08:54 opensearch_server.json

Permissions on /var/log/opensearch directory

drwxr-xr-x  2 opensearch opensearch  69632 Aug 14 00:04 opensearch

Permissions on /var/log directory

drwxrwxr-x 17 root syslog  4096 Aug 14 05:28 log

opoplawski · August 14, 2023, 2:37pm

Permissions are like becks’:

drwxr-sr-x. 2 opensearch opensearch 28672 Aug 14 03:09 /var/log/opensearch//
drwxr-xr-x. 17 root root 8192 Aug 13 03:35 /var/log//

And yes, I also see it for opensearch.log as well.

becks · September 8, 2023, 6:35am

After many tries, for me the solution was to add and grant

grant {
  java.lang.RuntimePermission "accessUserInformation" 
};

to the jave security policy file under
/etc/opensearch/opensearch-performance-analyzer/opensearch_security.policy

tejashu · November 29, 2024, 4:20am

What if we are using the min OpenSearch version and we do not have this folder.
Where can we add this grant to resolve the issue?

Topic		Replies	Views
Job for opensearch.service failed because a timeout was exceeded. See "systemctl status opensearch.service" and "journalctl -xeu opensearch.service" for details OpenSearch troubleshoot	7	3236	January 10, 2024
Opensearch-dashboards errors OpenSearch troubleshoot , configure	7	1273	July 21, 2022
Stuck on first start OpenSearch install	4	476	January 22, 2024
Deb install systemd service fails to start - Could not initialize class com.sun.jna.Native OpenSearch troubleshoot , install	1	1106	April 4, 2024
Do we know if there are any known vulnerabilities in opensearch version 2.8.0? OpenSearch cve	3	582	July 13, 2023

Systemd-entrypoint DefaultDispatcher-worker ERROR Could not define attribute view on path "/var/log/opensearch/opensearch_server.json"

Related topics