Is there a way to reference one index from another as a "foreign key"?

Open Source Elasticsearch and Kibana
1

My team has two indexes: Document and Page. I would like to reference the Document metadata inside of the Page index. Is there a way to do this so that any changes to the Document index are viewable from inside the Page index? One member of our team tried to do this and was unsuccessful due to the index having security attributes. Is this something that is possible with OpenSearch? Also, is it possible refer to different entries within the same index where one Document references another Document?

Also, here is an example of the structure of our index:

{
“took” : 280,
“timed_out” : false,
“_shards” : {
“total” : 1,
“successful” : 1,
“skipped” : 0,
“failed” : 0
},
“hits” : {
“total” : {
“value” : 5,
“relation” : “eq”
},
“max_score” : 1.0,
“hits” : [
{
“_index” : “page_index”,
“_id” : “F32376F3B85C4B5D905E6200659DB2F2”,
“_score” : 1.0,
“_source” : {
“index_updated” : “YES”,
“page” : {
“descriptiveTitle” : “TEST PAGE ONE”,
“securityAttributes” : [
“AUTHORIZED_USERS”
],
“languages” : {
“descriptiveTitleLanguage” : “ENGLISH”,
“primaryLanguage” : “ENGLISH”
},
“dateCreated” : “2023-03-22T14:47:35.000Z”,
“securityLevel” : 0,
“entryMethod” : “TED_CRUD”,
“pageUUID” : “F32376F3B85C4B5D905E6200659DB2F2”,
“instId” : 29345631,
“trackingNumber” : “ELASTIC-2023-1”,
“status” : “S”,
“classifications” : {
“pageClassification” : “MYCLASS”,
“descriptiveTitleClassification” : “MYCLASS”
},
“createTStamp” : “2023-03-22T14:47:36.044Z”,
“items” : {
“projects” : [
{
“project” : “TEST DATA FOR ELASTIC”
}
]
},
“document” : {
“commentClassification” : “MYCLASS”,
“uploadNomination” : “UPLOAD TO SYSTEM”,
“classification” : “MYCLASS”,
“dateCaptured” : “2023-03-22T00:00:00.000Z”,
“uuid” : “0EB91169E11A437881F1BE400D5DC4B6”,
“securityLevel” : 0,
“comment” : “TEST”,
“status” : “S”,
“projectName” : “TEST DATA FOR ELASTIC”,
“createTStamp” : “2023-03-22T14:46:55.721Z”,
“isActive” : “1”,
“name” : “TEST DOCUMENT FOR ELASTIC”
},
“ownerAgency” : “MYCOMPANY”
},
“@version” : “1”,
“inst_id” : 29345631,
“@timestamp” : “2023-03-22T18:48:10.532287269Z”,
“page_uuid” : “F32376F3B85C4B5D905E6200659DB2F2”
}
},
{
“_index” : “page_index”,
“_id” : “09DD3CF781054055B327B02EF8414BEB”,
“_score” : 1.0,
“_source” : {
“index_updated” : “YES”,
“page” : {
“descriptiveTitle” : “TEST PAGE TWO”,
“securityAttributes” : [
“AUTHORIZED_USERS”
],
“languages” : {
“descriptiveTitleLanguage” : “ENGLISH”,
“primaryLanguage” : “ARABIC”
},
“dateCreated” : “2023-03-22T14:47:54.000Z”,
“securityLevel” : 0,
“entryMethod” : “TED_CRUD”,
“pageUUID” : “09DD3CF781054055B327B02EF8414BEB”,
“instId” : 29345632,
“trackingNumber” : “ELASTIC-2023-2”,
“status” : “S”,
“classifications” : {
“pageClassification” : “MYCLASS”,
“descriptiveTitleClassification” : “MYCLASS”
},
“createTStamp” : “2023-03-22T14:47:54.734Z”,
“items” : {
“projects” : [
{
“project” : “TEST DATA FOR ELASTIC”
}
]
},
“document” : {
“commentClassification” : “MYCLASS”,
“uploadNomination” : “UPLOAD TO SYSTEM”,
“classification” : “MYCLASS”,
“dateCaptured” : “2023-03-22T00:00:00.000Z”,
“uuid” : “0EB91169E11A437881F1BE400D5DC4B6”,
“securityLevel” : 0,
“comment” : “TEST”,
“status” : “S”,
“projectName” : “TEST DATA FOR ELASTIC”,
“createTStamp” : “2023-03-22T14:46:55.721Z”,
“isActive” : “1”,
“name” : “TEST DOCUMENT FOR ELASTIC”
},
“ownerAgency” : “MYCOMPANY”
},
“@version” : “1”,
“inst_id” : 29345632,
“@timestamp” : “2023-03-22T18:48:10.533636113Z”,
“page_uuid” : “09DD3CF781054055B327B02EF8414BEB”
}
},
{
“_index” : “page_index”,
“_id” : “A4D472AD04044A7192E6A8159E8E1476”,
“_score” : 1.0,
“_source” : {
“index_updated” : “YES”,
“page” : {
“descriptiveTitle” : “TEST PAGE THREE”,
“securityAttributes” : [
“AUTHORIZED_USERS”
],
“languages” : {
“descriptiveTitleLanguage” : “ENGLISH”,
“primaryLanguage” : “URDU”
},
“dateCreated” : “2023-03-22T14:48:12.000Z”,
“securityLevel” : 0,
“entryMethod” : “TED_CRUD”,
“pageUUID” : “A4D472AD04044A7192E6A8159E8E1476”,
“instId” : 29345633,
“trackingNumber” : “ELASTIC-2023-3”,
“status” : “S”,
“classifications” : {
“pageClassification” : “MYCLASS”,
“descriptiveTitleClassification” : “MYCLASS”
},
“createTStamp” : “2023-03-22T14:48:12.959Z”,
“items” : {
“projects” : [
{
“project” : “TEST DATA FOR ELASTIC”
}
]
},
“document” : {
“commentClassification” : “MYCLASS”,
“uploadNomination” : “UPLOAD TO SYSTEM”,
“classification” : “MYCLASS”,
“dateCaptured” : “2023-03-22T00:00:00.000Z”,
“uuid” : “0EB91169E11A437881F1BE400D5DC4B6”,
“securityLevel” : 0,
“comment” : “TEST”,
“status” : “S”,
“projectName” : “TEST DATA FOR ELASTIC”,
“createTStamp” : “2023-03-22T14:46:55.721Z”,
“isActive” : “1”,
“name” : “TEST DOCUMENT FOR ELASTIC”
},
“ownerAgency” : “MYCOMPANY”
},
“@version” : “1”,
“inst_id” : 29345633,
“@timestamp” : “2023-03-22T18:48:40.536360749Z”,
“page_uuid” : “A4D472AD04044A7192E6A8159E8E1476”
}
},
{
“_index” : “page_index”,
“_id” : “40519E68A8E543FD9E8F7084CB8CE579”,
“_score” : 1.0,
“_source” : {
“index_updated” : “YES”,
“page” : {
“descriptiveTitle” : “TEST PAGE FOUR”,
“securityAttributes” : [
“AUTHORIZED_USERS”
],
“languages” : {
“descriptiveTitleLanguage” : “ENGLISH”,
“primaryLanguage” : “ARAMAIC”
},
“dateCreated” : “2023-03-22T14:48:27.000Z”,
“securityLevel” : 0,
“entryMethod” : “TED_CRUD”,
“pageUUID” : “40519E68A8E543FD9E8F7084CB8CE579”,
“instId” : 29345634,
“trackingNumber” : “ELASTIC-2023-4”,
“status” : “S”,
“classifications” : {
“pageClassification” : “MYCLASS”,
“descriptiveTitleClassification” : “MYCLASS”
},
“createTStamp” : “2023-03-22T14:48:27.428Z”,
“items” : {
“projects” : [
{
“project” : “TEST DATA FOR ELASTIC”
}
]
},
“document” : {
“commentClassification” : “MYCLASS”,
“uploadNomination” : “UPLOAD TO SYSTEM”,
“classification” : “MYCLASS”,
“dateCaptured” : “2023-03-22T00:00:00.000Z”,
“uuid” : “0EB91169E11A437881F1BE400D5DC4B6”,
“securityLevel” : 0,
“comment” : “TEST”,
“status” : “S”,
“projectName” : “TEST DATA FOR ELASTIC”,
“createTStamp” : “2023-03-22T14:46:55.721Z”,
“isActive” : “1”,
“name” : “TEST DOCUMENT FOR ELASTIC”
},
“ownerAgency” : “MYCOMPANY”
},
“@version” : “1”,
“inst_id” : 29345634,
“@timestamp” : “2023-03-22T18:48:40.537500461Z”,
“page_uuid” : “40519E68A8E543FD9E8F7084CB8CE579”
}
},
{
“_index” : “page_index”,
“_id” : “F706F584D44E432592D6BE76EEE973EE”,
“_score” : 1.0,
“_source” : {
“index_updated” : “YES”,
“page” : {
“descriptiveTitle” : “TEST PAGE FIVE - You get the idea”,
“securityAttributes” : [
“AUTHORIZED_USERS”
],
“languages” : {
“descriptiveTitleLanguage” : “ENGLISH”,
“primaryLanguage” : “ARAMAIC”
},
“dateCreated” : “2023-03-22T14:48:45.000Z”,
“securityLevel” : 0,
“entryMethod” : “TED_CRUD”,
“pageUUID” : “F706F584D44E432592D6BE76EEE973EE”,
“instId” : 29345635,
“trackingNumber” : “ELASTIC-2023-5”,
“status” : “S”,
“classifications” : {
“pageClassification” : “MYCLASS”,
“descriptiveTitleClassification” : “MYCLASS”
},
“createTStamp” : “2023-03-22T14:48:45.590Z”,
“items” : {
“projects” : [
{
“project” : “TEST DATA FOR ELASTIC”
}
]
},
“document” : {
“commentClassification” : “MYCLASS”,
“uploadNomination” : “UPLOAD TO SYSTEM”,
“classification” : “MYCLASS”,
“dateCaptured” : “2023-03-22T00:00:00.000Z”,
“uuid” : “0EB91169E11A437881F1BE400D5DC4B6”,
“securityLevel” : 0,
“comment” : “TEST”,
“status” : “S”,
“projectName” : “TEST DATA FOR ELASTIC”,
“createTStamp” : “2023-03-22T14:46:55.721Z”,
“isActive” : “1”,
“name” : “TEST DOCUMENT FOR ELASTIC”
},
“ownerAgency” : “MYCOMPANY”
},
“@version” : “1”,
“inst_id” : 29345635,
“@timestamp” : “2023-03-22T18:49:10.521273629Z”,
“page_uuid” : “F706F584D44E432592D6BE76EEE973EE”
}
}
]
}
}

3

If you can be a bit flexible with the indexing you could do this all from the same index with the Join field type. It creates a parent/child relationship that is query-able.

The other option would be to change the model to using a graph DB. Something like YangDB would allow you to reference documents from other documents by representing them as a graph.

4

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.