Visualizations fail for Tenant Dashboards After Migration to .kibana_2 in OpenSearch Dashboards 2.15

[rahulkr]

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpenSearch-2.15.0

Describe the issue:
We upgraded OpenSearch Dashboards from 2.3.0 to 2.15.0, which triggered saved object migration and created a new global index: .kibana_2. .kibana_1 was created in 2.3.0.

Post-migration, the alias .kibana now points to .kibana_2. Some of the Dashboards in the ABC tenant began showing “Could not locate that visualization (id: <visualization_id>)” errors — even though:

• All referenced visualizations are present within the tenant index (.kibana_1_572569278_abctenant_1)
• These visualizations are not present in .kibana_2 or .kibana_1 global indices (as expected)

Strangely, when we revert the .kibana alias back to .kibana_1, the dashboards in the tenant work fine and do not show missing visualizations error.

Some Analysis:

• Dashboard object resides in the ABC tenant index
• Visualizations it references also reside in the ** ABC tenant index**
• Referenced visualizations are not present in .kibana_2
• When .kibana → .kibana_2, hydration fails in Dashboards 2.15
• When .kibana → .kibana_1, hydration works correctly

Is this a regression or behavioral change in saved object hydration logic in OpenSearch Dashboards 2.15?

• In 2.3.0, the system correctly resolves references from the tenant index only
• In 2.15.0, it seems to require referenced objects to also exist in the global index — or it fails silently with “Could not locate that visualization” errors

Expected Behavior

OpenSearch Dashboards should resolve references:

1. First from the current tenant index, then
2. Fall back to the global index only if not found

If the visualization object is present in the tenant (which it is), hydration should not fail. Is any one aware of this or help here ?

Configuration:

Relevant Logs or Screenshots:

• Clean migration logs (no failed migrations)
• Dashboards and referenced visualizations all have proper references[] pointing to objects within the tenant

[Leeroy]

Hi @rahulkr ,

So I have first tested creating visualizations via private tenant in 2.3.0. Then as expected I see similar Could not locate that visualization when I try view via the global tenant.

Then when I also create visualizations from the global tenant I see the same result when viewing from the private tenant.

However you may be aware this is by design and is the case for security purposes. Essentially when you query Opensearch for a visualization it query’s against the current tenant in use.

When I then upgrade to 2.15.0 I see the same behavior and both tenants work as before and as expected.

The easiest approach be to store all visualizations you want to be seen by all in the Global tenant, and then use the private or custom tenants to create unique visualizations you don’t want to be seen by the global tenant. Which from your issue I believe is what you’re doing.

One difference I do see is .kibana_1_572569278_abctenant_1 when I tested this on a fresh 2.3.0 and upgraded to 2.15.0 changes to the naming had already switched to .kibana_572569278_abctenant_1. Which would lead me to believe one of the the previous versions you used before upgrading to 2.3.0 might be the initial cause of the issue you’re seeing.

Which is why it looks to work fine when the alias is added to .kibana_1.

For me, post upgrade everything works fine, I see both .kibana_92668751_upgradeTestTenant_1 and .kibana_92668751_upgradeTestTenant_2 which shows visualizations created before the upgrade and can visualize them fine on 2.3.0. I also see .kibana , .kibana_1 and .kibana_2 which contain the global visualizations from before the upgrade.

I confirmed all the visualizations are working in the UI and queried the contents of each index using

GET /.someindex/_search
{
 "query": {
    "term": {
      "type": "visualization"
    }
  },
  "size": 1000 //update the size if needed
}

• If you know the version from before I could test and see if I get similar results.

Leeroy

[rahulkr]

Hello @Leeroy Thanks a lot for the writeup and your testing. To give more context on my current case.

We have been encountering a persistent issue with missing visualizations in OpenSearch Dashboards. Originally, the problematic dashboards and visualizations were created using version 2.3.0. After upgrading to 2.11, we began experiencing this issue. As a workaround, we reverted the .kibana alias back to .kibana_1, which had been migrated to .kibana_2 during the 2.11 upgrade. We also deleted the .kibana_2 index at that time.

Hoping the issue would be resolved, we upgraded further to 2.15. However, the problem persisted. We applied the same mitigation strategy again — reverting the alias to .kibana_1 and deleting .kibana_2. Unfortunately, this has become a recurring pain point. After every pod restart, the issue reappears and requires manual intervention.

To reproduce the issue in a development environment, I exported mappings, index objects from .kibana_1 as well as from the particular tenant index of the main(PROD) cluster to a dev cluster. I attempted an upgrade in the dev setup, but as you observed, we were unable to reproduce the missing visualizations.

Interestingly, we have another production cluster in a different region acting as a backup for the main cluster. This backup cluster is also running on OS version 2.15, with the global alias correctly pointing to .kibana_2, and it does not experience the missing visualization issue. Furthermore, the .kibana_2 index mappings in both the main and backup clusters are identical (as verified via GET .kibana_2/_mapping).

I also reviewed the verbose migration logs for the main (prod) cluster. They confirm that all visualizations were successfully migrated from .kibana_1 to .kibana_2 during the pod restart. However, we continue to manually revert the alias to .kibana_1 after each restart because of the missing visualizations issue. This cycle repeats with every pod restart, requiring the same manual mitigation.

One additional observation: in the main cluster, the tenant index (.kibana_1_572569278_abctenant_1, originally created in 2.3) does not migrate after a pod restart. In contrast, the backup cluster has the corresponding tenant index already migrated to .kibana_572569278_abctenant_2. It is created on 2.15. Note the difference in index naming — the main cluster uses .kibana_1_... while the backup uses .kibana_....

Analysis Summary:

I conducted a comparison of the impacted dashboard’s panelsJSON field between .kibana_1 and .kibana_2.

Observations:

• .kibana_1 Panels:

  • "version": "2.3.0-oracle.24"

  • dashboard.migrationVersion: 7.9.0

  • visualization.migrationVersion: 7.10.0

• .kibana_2 Panels:

  • "version": "2.15.0-oracle.19"

  • dashboard.migrationVersion: 7.9.0

  • visualization.migrationVersion: 7.10.0

Both indices (.kibana_1 and .kibana_2) show identical top-level migrationVersion values for the dashboard and visualizations. However, the individual panel versions differ.

Hypothesis:

Although the top-level dashboard.migrationVersion is set to 7.9.0, it appears that OpenSearch Dashboards 2.15 determines how to hydrate the saved object based on the version of the individual panels in panelsJSON.

• When the system detects panel versions ≥ 2.15.0, it likely triggers the newer (7.10+) hydration logic.

• However, since the dashboard-level metadata remains at an older version (7.9.0), the object may lack the normalized fields expected by the newer hydration logic — leading to partial or failed rendering (i.e., “Could not locate that visualization” errors).

Why reverting the alias to .kibana_1 works:

• .kibana_1 contains dashboards and panels created with 2.3.0.

• With both the dashboard and panel versions aligned to the pre-7.10 era, hydration logic defaults to the legacy safe path, handling references and structure in a backward-compatible way.

Possible Regression or Behavior Change:

• In version 2.3.0, OpenSearch Dashboards correctly resolved saved object references (e.g., visualizations) entirely from the tenant index.

• In version 2.15.0, it seems to expect referenced objects to also exist in the global index, and fails silently if they don’t — displaying generic missing visualization errors.

This could indicate either:

1. A regression in reference resolution logic, or

2. An intentional behavioral change that hasn’t been clearly documented.

3. I am missing some important piece in middle

Request for Guidance:

Could you please confirm:

• Whether this understanding aligns with known changes in object hydration or saved object migration logic between 2.3 and 2.15?

• If this behavior is expected, is there a recommended way to ensure reliable migration from older .kibana indices to .kibana_2 without breaking visualization references?

This issue is currently blocking us, as we must apply a manual alias rollback after every pod restart. Any insights or recommended next steps would be greatly appreciated. I can share additional logs, index exports, or config details if helpful.

[rahulkr]

@Leeroy and forum, any inputs/suggestions would be highly appreciated.

[Leeroy]

Hey @rahulkr ,

I have a couple of solutions you can try.

1. Export all visualizations. Start a fresh clean deploy of Dashboards/or backup and delete all kibana indices then restart dashboards and then re-import all objects back, at least this should resolve it going forward when a pod restarts. The fresh instance of 2.15 should have all objects and correct versions which should ideally prevent issues for future upgrades.
2. Another option is to manually reindex Reindex data - OpenSearch Documentation .

Although I think option 1 is best just to ensure everything is new and created by the correct versions. Plus giving you’re having an issue where the index reverts back I’d say this option could be problematic, but I’d test to see first if creating a fresh dashboards isn’t as feasible.

Next I would like to do some testing with your setup to reproduce and see what might be causing the issue.

Are you using helm, if so are you using any custom deploy?

Could you share the exports for the indices of issue, so I can reproduce the failure on my end. Also please share your configs with any sensitive information redacted so I can do some further testing.

Plus if you have any information on versions used before 2.3 that be great.

Leeroy.

[rahulkr]

Hi @Leeroy Thanks for the update. I had exporting earlier:

1. Export and re-import all visualizations: Exports failed because we have more than 23k dashboards. There is a limit of 10k exports from the dashboard as well as API

2. I will be trying reindexing and update you.

yes, we are using helm. The dashboards were created on 2.3 only. The issue started after upgrading to 2.11 and current version is 2.15.

Regarding sharing of these files, I am checking with the team for security violations as per the company policy. After that I will share them

[rahulkr]

@Leeroy A couple of clarifications around reindexing

• When to perform reindexing? (I assume after migration, but wanted to confirm)
• Reindex on which index? (global tenant’s index ( kibana_*) or the tenant’s index ) ?

We have more than 23k dashboards. I was getting error when I tried exporting earlier that opensearch does not allow to export more than 10k dashboards. But on further digging, I find this link which shows to download more. Though I think the configurable feature is from 2.17(current version is 2.15), but I will test this.

Since this is a prod cluster, deleting all indices might be risky. This becomes little tricky to operate as the issue is not reproducible locally, and we have not seen this issue in any other clusters. Any suggestion on this as well?

[Leeroy]

Hi @rahulkr ,

Did you confirm this cluster doesn’t have any issue in configuration when compared to the others which had no issue? If you could share your configs I can test and see on my end before we attempt to do anything further.

Did you manage to test exporting more than 10K objects? I would first backup your kibana index’s but I think giving it is prod I would much prefer you go the re-importing route.

If you can export the objects, spin up a new dashboards and populating all objects back in I think this will be the safest way forward. This will ensure you can test and validate it is working as expected before removing the old instance of dashboards.

Leeroy,