Auto_create_index randomly getting disabled

OpenSearch
,
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
v 2.5.0 on OpenSearch Operator, GKE

Describe the issue:
We have a self-managed OpenSearch cluster deployed through OpenSearch Operator in GKE, and we also use AWS-managed OpenSerach in our AWS environments (that’s v 2.13.0 but I don’t have relevant audit logs from AWS for now). Recently we have been noticing that auto_create_index in cluster settings randomly get disabled. So we turned on the security audit logging to troubleshoot.

From the audit logs it looks like it’s making a PutMappingRequest call related to auto_create_index. I would like to know if this is originated internally from within the OpenSerach as most likely none of us have made this request. Is it possible that this call could disable the cluster setting?

There’s also one more field audit_trace_resolved_indices which I am confused about. That is pointing to temporal index. What exactly does it indicate? The docs contain very little and vague information about this field, and Is that relevant to the issue we are facing?

Configuration:
N/A?

Relevant Logs or Screenshots:
security audit logs:

{
    "_index": "security-auditlog-2024.11.13",
    "_id": "JCsYJJMB080fu-NbxiXI",
    "_version": 1,
    "_score": null,
    "_source": {
      "audit_trace_task_parent_id": "0R4Ks7mZSTGfItYrsd2XIg:470190619",
      "audit_cluster_name": "os-cluster",
      "audit_transport_headers": {
        "_opendistro_security_remote_address_header": "...",
        "_opendistro_security_initial_action_class_header": "BulkShardRequest",
        "_opendistro_security_origin_header": "REST",
        "_opendistro_security_user_header": "...",
        "_opendistro_security_remotecn": "os-cluster"
      },
      "audit_node_name": "os-cluster-masters-1",
      "audit_trace_task_id": "yFDAc2uGRmm8Bz0Gb_IzAQ:96320951",
      "audit_transport_request_type": "PutMappingRequest",
      "audit_category": "INDEX_EVENT",
      "audit_request_origin": "REST",
      "audit_request_body": "{\"_doc\":{\"properties\":{\"log_processed\":{\"properties\":{\"persistent\":{\"properties\":{\"action\":{\"properties\":{\"auto_create_index\":{\"type\":\"text\",\"fields\":{\"keyword\":{\"type\":\"keyword\",\"ignore_above\":256}}}}}}},\"transient\":{\"type\":\"object\"}}}}}}",
      "audit_node_id": "yFDAc2uGRmm8Bz0Gb_IzAQ",
      "audit_request_layer": "TRANSPORT",
      "@timestamp": "2024-11-13T05:57:03.816+00:00",
      "audit_format_version": 4,
      "audit_request_remote_address": "35.191.1.224",
      "audit_request_privilege": "indices:admin/mapping/auto_put",
      "audit_node_host_address": "10.254.7.61",
      "audit_request_effective_user": "admin",
      "audit_trace_resolved_indices": [
        "temporal-2024.11.13"
      ],
      "audit_node_host_name": "os-cluster-masters-1"
    },
    "fields": {
      "@timestamp": [
        "2024-11-13T05:57:03.816Z"
      ]
    },
    "highlight": {
      "audit_request_body": [
        "{\"_doc\":{\"properties\":{\"log_processed\":{\"properties\":{\"persistent\":{\"properties\":{\"action\":{\"properties\":{\"@opensearch-dashboards-highlighted-field@auto_create_index@/opensearch-dashboards-highlighted-field@\":{\"type\":\"text\",\"fields\":{\"keyword\":{\"type\":\"keyword\",\"ignore_above\":256}}}}}}},\"transient\":{\"type\":\"object\"}}}}}}"
      ]
    },
    "sort": [
      1731477423816
    ]
  }
2

Hi @maulin.shah, this log rather indicates that a field called auto_create_index was added/updated. audit_trace_resolved_indices points to the index affected by this change, but the naming convention indicates that the index might have been created automatically.
Is auto_create_index disabled now if you check with GET /_cluster/settings?
If the setting was disabled you should be able to find it in the audit logs, but it also requires enabling a specific type.

3

@Patryk1 yes, the cluster settings was modified and auto_create_index was disabled

but it also requires enabling a specific type.

are you referring to log diffs in compliance settings here?

audit_trace_resolved_indices points to the index affected by this change

we found a temporal issue where it meddles with the cluster settings and disables it. But it is disabling a cluster level setting. So how is temporal is the only index “affected” by the change here?

4

@maulin.shah, thanks for additional information.

we found a temporal issue where it meddles with the cluster settings and disables it. But it is disabling a cluster level setting. So how is temporal is the only index “affected” by the change here?

I’m not sure what this log is related to, but it doesn’t look like it captured the cluster setting update.
This change is being recorded as GRANTED_PRIVILEGES (deactivated by default). I enabled it in a test cluster and it gave me the following output:

{
  "_index": "security-auditlog-2024.11.19",
  "_id": "eTC5QZMBEevWZXWwlSo0",
  "_score": null,
  "_source": {
    "audit_trace_task_parent_id": "e0RDbHLaT9-1TOzAgHlTOw:14240",
    "audit_cluster_name": "opensearch-test",
    "audit_transport_headers": {
      "_opendistro_security_remote_address_header": "...",
      "_opendistro_security_initial_action_class_header": "ClusterUpdateSettingsRequest",
      "_opendistro_security_origin_header": "REST",
      "_opendistro_security_user_header": "...",
      "_opendistro_security_remotecn": "opensearch-test"
    },
    "audit_node_name": "opensearch2",
    "audit_trace_task_id": "hoTuUxanTGaUXR1_Fhcj5w:21555",
    "audit_transport_request_type": "ClusterUpdateSettingsRequest",
    "audit_category": "GRANTED_PRIVILEGES",
    "audit_request_origin": "REST",
    "audit_request_body": """{"persistent_settings":{"action":{"auto_create_index":"true"}},"transient_settings":{"action":{"auto_create_index":"true"}}}""",
    "audit_node_id": "hoTuUxanTGaUXR1_Fhcj5w",
    "audit_request_layer": "TRANSPORT",
    "@timestamp": "2024-11-19T00:01:41.683+00:00",
    "audit_format_version": 4,
    "audit_request_remote_address": "172.19.0.2",
    "audit_request_privilege": "cluster:admin/settings/update",
    "audit_node_host_address": "172.19.0.3",
    "audit_request_effective_user": "admin",
    "audit_node_host_name": "172.19.0.3"
  },
  "sort": [
    1731974501683
  ]
},

Activating it can result in additional overhead though.

1 Like