Index.creation_date reset for old daily indices after major upgrade

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

- OpenSearch version: 3.5.0 (upgrade from 2.19.4)

- Deployment: Kubernetes (3 manager + 2 data nodes), manager and data PVCs preserved across upgrade

- Index Management (ISM) plugin: enabled

Describe the issue:

We upgraded a logging cluster from OpenSearch 2.19.4 to 3.5.0 with **PVCs preserved** (manager PVC for cluster state, data PVCs for shard data). ISM policy uses `min_index_age: 7d` for deletion.

After upgrade, ~87 of 94 daily log indices (e.g. `log-app-dbstorage-2026.04.10`) show:

index.creation_date= ~2026-06-23 13:50 UTC (same day for most affected indices)

Checks on a sample index `log-app-dbstorage-2026.04.10`:

- 1041 documents: `time` field = April 10, 2026 (original log event time)

- All documents: `date` field = `2026-06-23` (ingest/processing date)

- `creation_date` cannot be updated via settings API (`unknown setting [index.creation_date]`)

Configuration:

ISM Configuration:

{
  "policy_id": "internal-irm-default-index-retention-policy",
  "states": [
    {
      "name": "wait_for_aging",
      "transitions": [{
        "state_name": "wait_for_maintenance_window",
        "conditions": { "min_index_age": "7d" }
      }]
    },
    {
      "name": "wait_for_maintenance_window",
      "actions": [{ "cron": { "cron": { "expression": "0 */3 * * *", "timezone": "UTC" } } }],
      "transitions": [{ "state_name": "delete" }]
    },
    { "name": "delete", "actions": [{ "delete": {} }] }
  ]
}

Sample doc from a reset index -

GET log-app-dbstorage-2026.04.10/_search?size=1&_source=time,date

{
“_source”: {
“time”: “2026-04-10T07:38:41.324971Z”,
“date”: “2026-06-23”
}
}

Is this expected OpenSearch behaviour during upgrades, where the index is recreated instead of being reopened from the preserved cluster state?

If that’s expected behaviour then what is the recommended procedure after upgrade to avoid new creation_date on existing indices?

2

@shubtiwa
This isn’t expected behavior for a normal in-place upgrade. I reproduced your exact version path (2.19.4 → 3.5.0) with a preserved data volume: created an index on 2.19.4, recorded its creation_date, then swapped the image to 3.5.0 against the same volume and restarted. creation_date came back bit-for-bit identical (not touched). I also tested a worse case, wiping the manager’s cluster-state metadata entirely while leaving the index’s on-disk data intact, then letting 3.5.0 recover it. Even there, creation_date was recovered correctly from the per-index metadata. So a genuine restart/upgrade, even a degraded one, does not reset it.

The pattern you’re describing (most indices affected, all reset to the exact same upgrade timestamp) is the signature of those 87 indices having been recreated by something in your upgrade pipeline, rather than the cluster simply reattaching to existing storage on disk.

Can you check version.created_string on one of the affected indices:

GET log-app-dbstorage-2026.04.10/_settings?human

Look at settings.index.version. In a genuine in-place upgrade this looks like:

{ "created_string": "2.19.4", "upgraded_string": "3.5.0" }

created_string stays at whatever version the index was originally created on, and a separate upgraded_string field gets added once a newer-version node opens it.

If your affected indices instead show created_string: "3.5.0", that would indicate those indices were newly created during the upgrade window, not carried over, which would also fully explain the reset creation_date. At that point I’d look at whatever ran against your cluster around 2026-06-23 13:50 UTC, your K8s operator/Helm chart’s upgrade job, any pre-upgrade reindex step, or a snapshot/restore, for _reindex or index-recreate calls against those specific indices.

One thing you can rule out, there’s no automatic/silent reindexing anywhere in OpenSearch itself. The only version-compatibility check on upgrade throws a hard IllegalStateException and blocks cluster-manager startup if it finds an incompatible index, it doesn’t reindex anything quietly. So if this were a Lucene-version-compatibility issue, you’d have seen your cluster fail to come up with an explicit error, not a silent creation_date reset.

3

Thanks, Anthony.

We followed your guidance on created_string / upgraded_string and creation_date. On our cluster, none of those log-* indices showed a 2.19.4 → 3.5.0 upgrade signature; they all had created_string: 3.5.0 only. So the upgrade itself did not reset creation_date — the indices had been recreated on 3.5.0.

The failure chain was:

  1. ISM deleted ~86 daily log-* indices (Jun 23, 12:58 UTC) once they met min_index_age: 7d on their existing creation_date.
  2. ~52 minutes later, log shipper replayed buffered logs and auto-created the same historical index names (indices:admin/auto_create). That set a new creation_date (~Jun 23 13:50 UTC) while the index names still reflected old dates.
  3. ISM then treated those indices as new indices → stuck in wait_for_aging

In our case it was delete + ingest replay after upgrade, with ISM behaving as designed on creation_date.

Thanks again for the clear direction — it helped us close the investigation quickly.