I’m migrating my project from Open Distro to OpenSearch and am having an issue I didn’t have before.
This is using opensearch in a docker image based on opensearchproject/opensearch:1.1.0, the main difference being that I’m removing the opensearch-security plugin.
I’m using the API to create some detectors. As far as I can tell from the output, there are no errors in creating the detectors.
I can see the .opendistro-anomaly-detectors index with the _cat API:
bash-5.1# curl -XGET http://opensearch:9200/_cat/indices?expand_wildcards=all | sort
green open .kibana_1 bl6RECCfQ0WzvuvTneASmw 1 0 1865 265 655.9kb 655.9kb
green open arkime_dstats_v30 DRpMlzt5ThCx2RaUudtObw 2 0 16 4 186.9kb 186.9kb
green open arkime_fields_v30 ZlhsGU0aSRO6s0UOZ4VAAQ 1 0 1163 37 272.2kb 272.2kb
green open arkime_files_v30 vlwmFJrWRPyzc7khYStDTw 2 0 6 0 29.1kb 29.1kb
green open arkime_history_v1-21w46 QfS_sJcURhG57fVNBC6wBw 1 0 1 0 5.2kb 5.2kb
green open arkime_hunts_v30 RljBCva3SSGtq4abwEaG1A 1 0 0 0 208b 208b
green open arkime_lookups_v30 OivEyO0YSgq43hpkGMu3Mg 1 0 0 0 208b 208b
green open arkime_queries_v30 o55HBsmzQg-9l8aykqCiMw 1 0 0 0 208b 208b
green open arkime_sequence_v30 -OjhMqaQSkSW4V2-mfm7YA 1 0 1 2 13.3kb 13.3kb
green open arkime_sessions3-200428 OtwfYG8TTUWbV7_cm6zMlw 1 0 165416 30418 93.1mb 93.1mb
green open arkime_sessions3-211122 O_ILfY3QS4i1iyx9yNqKdg 1 0 14872 5490 17mb 17mb
green open arkime_stats_v30 EJHssTzYQZOwBwvBiV-kUg 1 0 1 11 98.5kb 98.5kb
green open arkime_users_v30 fveZyV8CSUu6HHc4ehXh4Q 1 0 2 5 204.5kb 204.5kb
yellow open .opendistro-anomaly-detectors rEgXGb6uSxaAQGIjVgo7nA 1 1 7 0 32.7kb 32.7kb
yellow open .opendistro-ism-config CgqYlqd0Tqi_uvDJTWERKA 1 1 7 2 89.3kb 89.3kb
yellow open .opendistro-ism-managed-index-history-2021.11.22-1 pyUOj7g4Rpef_ySflc-JDg 1 1 6 0 13.7kb 13.7kb
yellow open .opendistro-job-scheduler-lock T3SBe837TY-yGAOYkLuORg 1 1 2 3 24.7kb 24.7kb
And I can also do a _search operation and see them:
bash-5.1# curl -XPOST "http://opensearch:9200/_plugins/_anomaly_detection/detectors/_search" -H 'osd-xsrf:true' -H 'Content-type:application/json' -d "@/tmp/arkimesearch.json" | jq
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 3130 100 3030 100 100 200k 6779 --:--:-- --:--:-- --:--:-- 218k
{
"took": 4,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 3,
"relation": "eq"
},
"max_score": 1,
"hits": [
{
"_index": ".opendistro-anomaly-detectors",
"_type": "_doc",
"_id": "QdZ3SX0BVwteEm6PYOmp",
"_version": 1,
"_seq_no": 0,
"_primary_term": 1,
"_score": 1,
"_source": {
"name": "total_bytes_AD",
"description": "Detect anomalies based on total bytes, summed",
"time_field": "firstPacket",
"indices": [
"arkime_sessions3-*"
],
"filter_query": {
"bool": {
"filter": [
{
"exists": {
"field": "network.bytes",
"boost": 1
}
}
],
"adjust_pure_negative": true,
"boost": 1
}
},
"detection_interval": {
"period": {
"interval": 10,
"unit": "Minutes"
}
},
"window_delay": {
"period": {
"interval": 1,
"unit": "Minutes"
}
},
"shingle_size": 8,
"schema_version": 0,
"feature_attributes": [
{
"feature_id": "QNZ3SX0BVwteEm6PYOkL",
"feature_name": "total_bytes",
"feature_enabled": true,
"aggregation_query": {
"total_bytes": {
"sum": {
"field": "network.bytes"
}
}
}
}
],
"last_update_time": 1637615100072,
"detector_type": "SINGLE_ENTITY"
}
},
{
"_index": ".opendistro-anomaly-detectors",
"_type": "_doc",
"_id": "Q9Z3SX0BVwteEm6PYOnK",
"_version": 1,
"_seq_no": 1,
"_primary_term": 1,
"_score": 1,
"_source": {
"name": "file_mime_type_AD",
"description": "Detect anomalies based on transferred file type",
"time_field": "firstPacket",
"indices": [
"arkime_sessions3-*"
],
"filter_query": {
"bool": {
"filter": [
{
"exists": {
"field": "file.mime_type",
"boost": 1
}
}
],
"adjust_pure_negative": true,
"boost": 1
}
},
"detection_interval": {
"period": {
"interval": 10,
"unit": "Minutes"
}
},
"window_delay": {
"period": {
"interval": 1,
"unit": "Minutes"
}
},
"shingle_size": 8,
"schema_version": 0,
"feature_attributes": [
{
"feature_id": "QtZ3SX0BVwteEm6PYOm8",
"feature_name": "file_mime_type",
"feature_enabled": true,
"aggregation_query": {
"file_mime_type": {
"value_count": {
"field": "file.mime_type"
}
}
}
}
],
"last_update_time": 1637615100106,
"detector_type": "SINGLE_ENTITY"
}
},
{
"_index": ".opendistro-anomaly-detectors",
"_type": "_doc",
"_id": "RtZ3SX0BVwteEm6PYOnl",
"_version": 1,
"_seq_no": 2,
"_primary_term": 1,
"_score": 1,
"_source": {
"name": "zeek_service_AD",
"description": "Detect anomalies based application protocol, aggregated by action and by result",
"time_field": "firstPacket",
"indices": [
"arkime_sessions3-*"
],
"filter_query": {
"bool": {
"filter": [
{
"exists": {
"field": "network.protocol",
"boost": 1
}
},
{
"exists": {
"field": "event.action",
"boost": 1
}
},
{
"exists": {
"field": "event.result",
"boost": 1
}
}
],
"adjust_pure_negative": true,
"boost": 1
}
},
"detection_interval": {
"period": {
"interval": 10,
"unit": "Minutes"
}
},
"window_delay": {
"period": {
"interval": 1,
"unit": "Minutes"
}
},
"shingle_size": 8,
"schema_version": 0,
"feature_attributes": [
{
"feature_id": "RNZ3SX0BVwteEm6PYOnb",
"feature_name": "event_action",
"feature_enabled": true,
"aggregation_query": {
"event_action": {
"value_count": {
"field": "event.action"
}
}
}
},
{
"feature_id": "RdZ3SX0BVwteEm6PYOnb",
"feature_name": "event_result",
"feature_enabled": true,
"aggregation_query": {
"event_result": {
"value_count": {
"field": "event.result"
}
}
}
}
],
"last_update_time": 1637615100133,
"category_field": [
"network.protocol"
],
"detector_type": "MULTI_ENTITY"
}
}
]
}
}
I’m also running OpenSearch dashboards in docker, based on opensearchproject/opensearch-dashboards:1.1.0, the only modification of any interest being that the securityDashboards has been removed.
When I navigate to the Anomaly Detectors plugin UI in dashboards, I get the following in the console:
org.opensearch.index.IndexNotFoundException: no such index [.opendistro-anomaly-detection-state]
at org.opensearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.indexNotFoundException(IndexNameExpressionResolver.java:959) ~[opensearch-1.1.0.jar:1.1.0]
at org.opensearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.innerResolve(IndexNameExpressionResolver.java:896) ~[opensearch-1.1.0.jar:1.1.0]
at org.opensearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.resolve(IndexNameExpressionResolver.java:852) ~[opensearch-1.1.0.jar:1.1.0]
at org.opensearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:231) ~[opensearch-1.1.0.jar:1.1.0]
at org.opensearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:207) ~[opensearch-1.1.0.jar:1.1.0]
at org.opensearch.action.search.TransportSearchAction.resolveLocalIndices(TransportSearchAction.java:634) ~[opensearch-1.1.0.jar:1.1.0]
at org.opensearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:662) ~[opensearch-1.1.0.jar:1.1.0]
at org.opensearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:541) ~[opensearch-1.1.0.jar:1.1.0]
at org.opensearch.action.search.TransportSearchAction.lambda$executeRequest$3(TransportSearchAction.java:333) ~[opensearch-1.1.0.jar:1.1.0]
at org.opensearch.action.ActionListener$1.onResponse(ActionListener.java:76) [opensearch-1.1.0.jar:1.1.0]
at org.opensearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:127) [opensearch-1.1.0.jar:1.1.0]
at org.opensearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:100) [opensearch-1.1.0.jar:1.1.0]
at org.opensearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:376) [opensearch-1.1.0.jar:1.1.0]
at org.opensearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:259) [opensearch-1.1.0.jar:1.1.0]
at org.opensearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:120) [opensearch-1.1.0.jar:1.1.0]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:192) [opensearch-1.1.0.jar:1.1.0]
at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:141) [opensearch-index-management-1.1.0.0.jar:1.1.0.0]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:190) [opensearch-1.1.0.jar:1.1.0]
at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:99) [opensearch-performance-analyzer-1.1.0.0.jar:1.1.0.0]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:190) [opensearch-1.1.0.jar:1.1.0]
at org.opensearch.action.support.TransportAction.execute(TransportAction.java:168) [opensearch-1.1.0.jar:1.1.0]
at org.opensearch.action.support.TransportAction.execute(TransportAction.java:96) [opensearch-1.1.0.jar:1.1.0]
at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:99) [opensearch-1.1.0.jar:1.1.0]
at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:88) [opensearch-1.1.0.jar:1.1.0]
at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:428) [opensearch-1.1.0.jar:1.1.0]
at org.opensearch.client.support.AbstractClient.search(AbstractClient.java:561) [opensearch-1.1.0.jar:1.1.0]
at org.opensearch.ad.transport.handler.ADSearchHandler.validateRole(ADSearchHandler.java:84) [opensearch-anomaly-detection-1.1.0.0.jar:1.1.0.0]
at org.opensearch.ad.transport.handler.ADSearchHandler.search(ADSearchHandler.java:72) [opensearch-anomaly-detection-1.1.0.0.jar:1.1.0.0]
at org.opensearch.ad.transport.SearchADTasksTransportAction.doExecute(SearchADTasksTransportAction.java:50) [opensearch-anomaly-detection-1.1.0.0.jar:1.1.0.0]
at org.opensearch.ad.transport.SearchADTasksTransportAction.doExecute(SearchADTasksTransportAction.java:39) [opensearch-anomaly-detection-1.1.0.0.jar:1.1.0.0]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:192) [opensearch-1.1.0.jar:1.1.0]
at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:141) [opensearch-index-management-1.1.0.0.jar:1.1.0.0]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:190) [opensearch-1.1.0.jar:1.1.0]
Anomaly detector - Unable to search detectors { Error: [index_not_found_exception] no such index [.opendistro-anomaly-detection-state], with { index=".opendistro-anomaly-detection-state" & resource.id=".opendistro-anomaly-detection-state" & resource.type="index_or_alias" & index_uuid="_na_" }
at respond (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:349:15)
at checkRespForFailure (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:306:7)
at HttpConnector.<anonymous> (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)
at IncomingMessage.wrapper (/usr/share/opensearch-dashboards/node_modules/lodash/lodash.js:4991:19)
at IncomingMessage.emit (events.js:203:15)
at endReadableNT (_stream_readable.js:1145:12)
at process._tickCallback (internal/process/next_tick.js:63:19)
status: 404,
displayName: 'NotFound',
message:
'[index_not_found_exception] no such index [.opendistro-anomaly-detection-state], with { index=".opendistro-anomaly-detection-state" & resource.id=".opendistro-anomaly-detection-state" & resource.type="index_or_alias" & index_uuid="_na_" }',
path: '/_plugins/_anomaly_detection/detectors/tasks/_search',
query: {},
body:
{ error:
{ root_cause: [Array],
type: 'index_not_found_exception',
reason: 'no such index [.opendistro-anomaly-detection-state]',
index: '.opendistro-anomaly-detection-state',
at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:99) [opensearch-performance-analyzer-1.1.0.0.jar:1.1.0.0]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:190) [opensearch-1.1.0.jar:1.1.0]
'resource.id': '.opendistro-anomaly-detection-state',
'resource.type': 'index_or_alias',
index_uuid: '_na_' },
status: 404 },
statusCode: 404,
response:
'{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [.opendistro-anomaly-detection-state]","index":".opendistro-anomaly-detection-state","resource.id":".opendistro-anomaly-detection-state","resource.type":"index_or_alias","index_uuid":"_na_"}],"type":"index_not_found_exception","reason":"no such index [.opendistro-anomaly-detection-state]","index":".opendistro-anomaly-detection-state","resource.id":".opendistro-anomaly-detection-state","resource.type":"index_or_alias","index_uuid":"_na_"},"status":404}',
toString: [Function],
toJSON: [Function] }
at org.opensearch.action.support.TransportAction.execute(TransportAction.java:168) [opensearch-1.1.0.jar:1.1.0]
at org.opensearch.action.support.TransportAction.execute(TransportAction.java:96) [opensearch-1.1.0.jar:1.1.0]
at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:99) [opensearch-1.1.0.jar:1.1.0]
at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:88) [opensearch-1.1.0.jar:1.1.0]
at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:428) [opensearch-1.1.0.jar:1.1.0]
at org.opensearch.ad.rest.AbstractSearchAction.lambda$prepareRequest$0(AbstractSearchAction.java:101) [opensearch-anomaly-detection-1.1.0.0.jar:1.1.0.0]
at org.opensearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:123) [opensearch-1.1.0.jar:1.1.0]
at org.opensearch.rest.RestController.dispatchRequest(RestController.java:271) [opensearch-1.1.0.jar:1.1.0]
at org.opensearch.rest.RestController.tryAllHandlers(RestController.java:353) [opensearch-1.1.0.jar:1.1.0]
at org.opensearch.rest.RestController.dispatchRequest(RestController.java:204) [opensearch-1.1.0.jar:1.1.0]
at org.opensearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:332) [opensearch-1.1.0.jar:1.1.0]
at org.opensearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:397) [opensearch-1.1.0.jar:1.1.0]
at org.opensearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:322) [opensearch-1.1.0.jar:1.1.0]
at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:55) [transport-netty4-client-1.1.0.jar:1.1.0]
at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:41) [transport-netty4-client-1.1.0.jar:1.1.0]
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at org.opensearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:71) [transport-netty4-client-1.1.0.jar:1.1.0]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.59.Final.jar:4.1.59.Final]
172.22.8.1 - tlacuache [22/Nov/2021:21:25:51 +0000] "GET /dashboards/api/anomaly_detectors/_indices?index=*sample-* HTTP/1.1" 200 37 "https://localhost/dashboards/app/anomaly-detection-dashboards" "Mozilla/5.0 (X11; Linux x86_64; rv:94.0) Gecko/20100101 Firefox/94.0"
172.22.8.1 - tlacuache [22/Nov/2021:21:25:51 +0000] "GET /dashboards/api/anomaly_detectors/detectors?from=0&search=sample&indices=&size=1000&sortDirection=asc&sortField=name HTTP/1.1" 200 61 "https://localhost/dashboards/app/anomaly-detection-dashboards" "Mozilla/5.0 (X11; Linux x86_64; rv:94.0) Gecko/20100101 Firefox/94.0"
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.59.Final.jar:4.1.59.Final]
at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) [netty-codec-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) [netty-codec-4.1.59.Final.jar:4.1.59.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) [netty-codec-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.59.Final.jar:4.1.59.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.59.Final.jar:4.1.59.Final]
at java.lang.Thread.run(Thread.java:832) [?:?]
Seems to be complaining about .opendistro-anomaly-detection-state not existing.
Back last year I had a somewhat similar problem with OpenDistro, which may have been addressed in this bug. Does this look like the same thing? Is this a new bug or a regression, or am I just doing something wrong?