# Cross cluster search with secured clusters

**URL:** <https://forum.opensearch.org/t/cross-cluster-search-with-secured-clusters/5162>\
**Category:** Security\
**Created:** [March 1, 2021, 2:58pm UTC](https://forum.opensearch.org/t/cross-cluster-search-with-secured-clusters/5162 "2021-03-01T14:58:51Z")\
**Posts on this page:** 19\
**Page:** 1

<div class="post-metadata">

**Author:** ![robert](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/robert/32/1163_2.png) [@robert](https://forum.opensearch.org/u/robert)\
**Post date:** [March 1, 2021, 2:58pm UTC](https://forum.opensearch.org/t/cross-cluster-search-with-secured-clusters/5162/1 "2021-03-01T14:58:51Z")

</div>

Hi,

We’ve successfully connected our local cluster with the remote cluster and can do searches in the dev console. However, we can’t refresh the index pattern in Kibana (behind the hoods we see a 404 when trying this). This seems to also apply while creating the index pattern as Kibana claims this in step 2 in the UI:

```auto
Step 2 of 2: Configure settings
Specify settings for your \*:logs-\* index pattern.
The indices which match this index pattern don't contain any time fields.

```

The end result is that we can’t get the timestamp field, or any other field, set.

Our best guess is that we’re missing permissions or have something incorrectly configured but we have not managed to figure out what and thus reaching out in the hopes of someone being able to help us.

Please let me know which information I can provide to better help understand this issue.

Any insights into this are greatly appreciated.

---

<div class="post-metadata">

**Author:** ![robert](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/robert/32/1163_2.png) [@robert](https://forum.opensearch.org/u/robert)\
**Post date:** [March 5, 2021, 8:56am UTC](https://forum.opensearch.org/t/cross-cluster-search-with-secured-clusters/5162/2 "2021-03-05T08:56:10Z")

</div>

So, to share some more details in the hopes of someone picking this up:

The error we get (from within Kibana) when trying to create the index pattern is this call that returns a 404

```
https://redacted.host/api/index_patterns/_fields_for_wildcard?pattern=*%3Acluster-*&meta_fields=_source&meta_fields=_id&meta_fields=_type&meta_fields=_index&meta_fields=_score

```

Does anyone know which permissions might be missing from a user for the above mentioned call to succeed?

---

<div class="post-metadata">

**Author:** ![pablo](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/pablo/32/10363_2.png) [@pablo](https://forum.opensearch.org/u/pablo)\
**Post date:** [March 5, 2021, 11:54am UTC](https://forum.opensearch.org/t/cross-cluster-search-with-secured-clusters/5162/3 "2021-03-05T11:54:34Z")

</div>

What ODFE version are you on?

---

<div class="post-metadata">

**Author:** ![robert](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/robert/32/1163_2.png) [@robert](https://forum.opensearch.org/u/robert)\
**Post date:** [March 5, 2021, 12:41pm UTC](https://forum.opensearch.org/t/cross-cluster-search-with-secured-clusters/5162/4 "2021-03-05T12:41:56Z")

</div>

ODFE version is: 1.13.0

---

<div class="post-metadata">

**Author:** ![pablo](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/pablo/32/10363_2.png) [@pablo](https://forum.opensearch.org/u/pablo)\
**Post date:** [March 10, 2021, 12:28pm UTC](https://forum.opensearch.org/t/cross-cluster-search-with-secured-clusters/5162/5 "2021-03-10T12:28:58Z")

</div>

I’ve done some testing and repro your issue. I’m getting the same screen in Step 2 as your are, no matter if security plugin is installed or removed.

It looks like as ODFE limitation or bug not related to security.

I’ve checked in ELK 7.10.2 with security disabled and was able to select time filed and create the index pattern.

---

<div class="post-metadata">

**Author:** ![robert](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/robert/32/1163_2.png) [@robert](https://forum.opensearch.org/u/robert)\
**Post date:** [March 10, 2021, 12:42pm UTC](https://forum.opensearch.org/t/cross-cluster-search-with-secured-clusters/5162/6 "2021-03-10T12:42:56Z")

</div>

Thank you so much for investigating this, what would be the next step?

Also, in Elastic 7.11 this flow is changing as the cached mapping is removed. Is this something that potentially might make it into a future OpenDistro release? Or do I need to file a bug and/or feature request somewhere?

---

<div class="post-metadata">

**Author:** ![pablo](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/pablo/32/10363_2.png) [@pablo](https://forum.opensearch.org/u/pablo)\
**Post date:** [March 10, 2021, 12:47pm UTC](https://forum.opensearch.org/t/cross-cluster-search-with-secured-clusters/5162/7 "2021-03-10T12:47:55Z")

</div>

Since this is not security issue. I would file it either as a bug if it was in previous odfe version or feature request if was never implemented.

Can’t find anything in odfe documentation and can’t tell you anything about future release as I’m not odfe dev.

---

<div class="post-metadata">

**Author:** ![robert](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/robert/32/1163_2.png) [@robert](https://forum.opensearch.org/u/robert)\
**Post date:** [March 10, 2021, 12:49pm UTC](https://forum.opensearch.org/t/cross-cluster-search-with-secured-clusters/5162/8 "2021-03-10T12:49:22Z")

</div>

This has been working before around 1.10 or so. Did a detour on our setup and stopped using remote clusters for a while.  
But thanks, I’ll try and create a bug report for this.

---

<div class="post-metadata">

**Author:** ![robert](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/robert/32/1163_2.png) [@robert](https://forum.opensearch.org/u/robert)\
**Post date:** [March 10, 2021, 12:54pm UTC](https://forum.opensearch.org/t/cross-cluster-search-with-secured-clusters/5162/9 "2021-03-10T12:54:29Z")

</div>

Would you (or someone else) be able do advice on which repo the bug should be reported, especially as it’s not security related?

---

<div class="post-metadata">

**Author:** ![pablo](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/pablo/32/10363_2.png) [@pablo](https://forum.opensearch.org/u/pablo)\
**Post date:** [March 10, 2021, 1:26pm UTC](https://forum.opensearch.org/t/cross-cluster-search-with-secured-clusters/5162/10 "2021-03-10T13:26:40Z")

</div>

It is working on 1.9.0, 1.10.1, 1.11. Fails on 1.12.0, 1.13.0 and 1.13.1

---

<div class="post-metadata">

**Author:** ![orsifacundo](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/orsifacundo/32/1383_2.png) [@orsifacundo](https://forum.opensearch.org/u/orsifacundo)\
**Post date:** [March 11, 2021, 12:14am UTC](https://forum.opensearch.org/t/cross-cluster-search-with-secured-clusters/5162/11 "2021-03-11T00:14:04Z")

</div>

Hi! Is there any workaround for this in the meantime?

Regards.

---

<div class="post-metadata">

**Author:** ![pablo](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/pablo/32/10363_2.png) [@pablo](https://forum.opensearch.org/u/pablo)\
**Post date:** [March 11, 2021, 1:59pm UTC](https://forum.opensearch.org/t/cross-cluster-search-with-secured-clusters/5162/12 "2021-03-11T13:59:31Z")

</div>

Hi @orsifacundo. I’m not aware of any.

---

<div class="post-metadata">

**Author:** ![robert](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/robert/32/1163_2.png) [@robert](https://forum.opensearch.org/u/robert)\
**Post date:** [March 25, 2021, 3:08pm UTC](https://forum.opensearch.org/t/cross-cluster-search-with-secured-clusters/5162/13 "2021-03-25T15:08:50Z")

</div>

Hi @orsifacundo.

Here a work-around was suggested: [Cross cluster search with secured local and remote clusters - #2 by alexz00](https://forum.opensearch.org/t/cross-cluster-search-with-secured-local-and-remote-clusters/5109/2)

---

<div class="post-metadata">

**Author:** ![robert](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/robert/32/1163_2.png) [@robert](https://forum.opensearch.org/u/robert)\
**Post date:** [March 29, 2021, 11:00am UTC](https://forum.opensearch.org/t/cross-cluster-search-with-secured-clusters/5162/14 "2021-03-29T11:00:27Z")

</div>

Just want to update saying that the mentioned work-around does indeed work!

---

<div class="post-metadata">

**Author:** ![robert](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/robert/32/1163_2.png) [@robert](https://forum.opensearch.org/u/robert)\
**Post date:** [March 29, 2021, 1:33pm UTC](https://forum.opensearch.org/t/cross-cluster-search-with-secured-clusters/5162/15 "2021-03-29T13:33:32Z")

</div>

And, after some more testing: If the index pattern is created from code and you do supply a field mapping for the ‘@timestamp’ field then you don’t have to refresh it from Kibana.

I guess you should not rely on the automatic mapping anyways but it has been more convenient. So, instead of patching the Kibana index directly and updating the title you can also do something like this:

```
  {
    "attributes":
    {
      "title": "<my_index_pattern>",
      "timeFieldName": "@timestamp",
      "fields": "[{\"count\": 0,\"name\":\"@timestamp\",\"type\":\"date\",\"esTypes\":[\"date\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true}]"
    }
  }

```

And use the saved\_objects API in Kibana to create the index pattern with that JSON body, as long as the timeFieldName can be mapped to something in fields it will work.

---

<div class="post-metadata">

**Author:** ![robert](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/robert/32/1163_2.png) [@robert](https://forum.opensearch.org/u/robert)\
**Post date:** [April 6, 2021, 9:07am UTC](https://forum.opensearch.org/t/cross-cluster-search-with-secured-clusters/5162/16 "2021-04-06T09:07:49Z")

</div>

Referencing the open bug for this issue:

[https://github.com/opendistro-for-elasticsearch/security-kibana-plugin/issues/688](https://github.com/opendistro-for-elasticsearch/security-kibana-plugin/issues/688)

---

<div class="post-metadata">

**Author:** ![robert](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/robert/32/1163_2.png) [@robert](https://forum.opensearch.org/u/robert)\
**Post date:** [April 6, 2021, 12:04pm UTC](https://forum.opensearch.org/t/cross-cluster-search-with-secured-clusters/5162/17 "2021-04-06T12:04:13Z")

</div>

> [@Cannot create Index Patterns of remote clusters - Open Distro 1.12 version](https://forum.opensearch.org/t/cannot-create-index-patterns-of-remote-clusters-open-distro-1-12-version/4732/6):
>
> Any new here? having same problem when trying to create index patterns containing remote clusters. Indexes are being found but then fails to list fields in next step with same errors described above. User has full permissions on both cluster and index level. Can it be that some of these operations are being done by kibanaserver user? What exact permissions then needed to list fields? Also same user can make API calls and read desired indexes on remote cluster. Using OD 1.13.1 version

So, a fix is coming in 1.13.2!

---

<div class="post-metadata">

**Author:** ![orsifacundo](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/orsifacundo/32/1383_2.png) [@orsifacundo](https://forum.opensearch.org/u/orsifacundo)\
**Post date:** [April 15, 2021, 6:47pm UTC](https://forum.opensearch.org/t/cross-cluster-search-with-secured-clusters/5162/18 "2021-04-15T18:47:40Z")

</div>

Awesome. I’ll give it a try!.

I ended up loading the index pattern via API with the following call:

curl -sS -k -u \<ODFE\_USER\>:\<ODFE\_PASS\> -XPOST “https://\<KIBANA\_IP\>/api/saved\_objects/index-pattern/\<REMOTE\_CLUSTER\_NAME\>:\<INDEX\_PATTERN\>” -H ‘Content-Type: application/json’ -H ‘kbn-xsrf: true’ -d @new-pattern.json

Thanks for sharing the other solution.

Regards.

---

<div class="post-metadata">

**Author:** ![robert](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/robert/32/1163_2.png) [@robert](https://forum.opensearch.org/u/robert)\
**Post date:** [April 16, 2021, 6:31am UTC](https://forum.opensearch.org/t/cross-cluster-search-with-secured-clusters/5162/19 "2021-04-16T06:31:43Z")

</div>

Nice, this issue has also been fixed in ODFE 1.13.2!