Data not visible in web interface after migration on OpenSearch

kdr · August 13, 2024, 5:58pm

Hello.

I have the next situation.

I have 2 servers on rhel with:

Graylog 3.3 + ElasticSearch 6.8.5 + Mongo 3
Graylog 5.2.9 + OpenSearch 2.15.0 + Mongo 7

On both servers all programms work in docker.
I try to migrate data from ElasticSearch to OpenSearch. I’ve already updated ElasticSearch version to 7.0.0. Then made _reindex for necessary indexes:

docker exec graylog_elasticsearch_1 /bin/curl -X POST "127.0.0.1:9200/_reindex" -H 'Content-Type: application/json' -d'
{
  "source": {
    "index": "graylog_*"
  },
  "dest": {
    "index": "trans_graylog"
}
}
'

Created snapshot:

docker exec graylog_elasticsearch_1 /bin/curl -X PUT '127.0.0.1:9200/_snapshot/backup-trans/2024-07-23-icann?wait_for_completion=true' -H 'Content-Type: application/json' -d'
{ 
  "indices": "trans_icann",
  "ignore_unavailable": true,
  "include_global_state": true
}
'

And moved it on server with OpenSearch.

This snapshot was successfully restored:

docker exec graylog-opensearch-1 /bin/curl -X POST '127.0.0.1:9200/_snapshot/backup-new/2024-07-23-graylog/_restore' -H 'Content-Type: application/json' -d'
{
  "indices": "trans_graylog",
  "ignore_unavailable": true,
  "include_global_state": true,
  "rename_pattern": "trans_graylog",
  "rename_replacement": "trans_graylog_0"
}
'

And I could see my new index with data:

docker exec graylog-opensearch-1 /bin/curl -s -X GET '127.0.0.1:9200/_cat/indices?v'
health status index                     uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   trans_icann_0             6nIVruurT9yoR_d3sgj9eA   1   0   17215000            0       21gb           21gb

But I couldn’t see any data in web interface. I’ve tried to create the same index with same settings in web and restart all containers, but there is no result.

What could I check with this issue? I’m a newby with OpenSearch, so have no idea what to do.

Thanks in advance.

Gsmitt · August 14, 2024, 2:33am

hey @kdr

First thing I noticed was the migration from ES 7.x to OS 2.15, I believe OpenSearch 1.3 is compatible with ES 7.10.

maybe rotate you index set? In Graylog there should be a drop down to Index Rotation or manual Index Recalculation.

kdr · August 14, 2024, 3:44pm

@Gsmitt
Hello,

I believe OpenSearch 1.3 is compatible with ES 7.10.

For my OpenSearch container version 2.15 is required, so I can’t use previous. Although I could use version from 2.0.0 to 2.15.0. By link here I’ve found, that migration via snapshot could be performed from ES 7.X to OpenSearch 1.x or OpenSearch 2.x (I need 2.x).

Is there any sense in trying update ES 7.0.0 => ES 7.15.0 and than migrate data to OpenSearch 2.x?

maybe rotate you index set?

Tried to rotate and recalculate transferred indexes and on index page in Graylog I see the following line (screenshot)

But data from this index not show in Streams tab. Is it possible to display it in Streams?

Gsmitt · August 15, 2024, 4:12am

hey

I believe you need to go to 7.10.x then to 1.3 then 2.0. that what I did last year without issues. so I don’t know about going to 7.15

kdr · August 21, 2024, 4:06pm

@Gsmitt
I found table with application versions and their compatibility:
https://go2docs.graylog.org/1380099/planning_your_deployment/planning_your_upgrade_to_opensearch.htm

For example, I have index migrate_to_opensearch. Here what I’ve done:
1 - Updated ES version from 7.0.0 to 7.10.2, Graylog from 3.3 to 4.3 and Mongo from 3 to 3.6
2 - Created new index migrate_to_opensearch_0

docker exec graylog_elasticsearch_1 /bin/curl -s -X PUT '127.0.0.1:9200/migrate_to_opensearch_0' -H 'Content-Type: application/json' -d'
{
  "settings": {
    "index": {                      
      "number_of_shards": 6,
      "number_of_replicas": 0
}                                     
},
  "mappings": {
    "properties": {
      "timestamp": {
        "type": "keyword"
}
}
}
}
'

3 - Make _reindex for migrate_to_opensearch to update its version

docker exec graylog_elasticsearch_1 /bin/curl -X POST '127.0.0.1:9200/_reindex' -H 'Content-Type: application/json' -d' 
{
  "source": {
    "index": "migrate_to_opensearch"
},                          
  "dest": {
    "index": "migrate_to_opensearch_0"
} 
}              
'

4 - Check that its status is green:

docker exec graylog_elasticsearch_1 /bin/curl -s -X GET '127.0.0.1:9200/_cat/indices?v' | grep migrate_to_opensearch_0
green  open   migrate_to_opensearch_0 yHqirkbQRnerky9sGWq1sw   6   0    2267709            0      2.1gb          2.1gb

5 - Created in Graylog interface index with the same name and made “Recalculate index range” (sceenshot 1).
6 - On screenshot 1 you can see, that index has data
7 - Created new stream with new index migrate_to_opensearch
8 - But in Stream tab I couldn’t see my data (screenshot 2)
9 - Also I’ve checked Graylog and ES logs and found nothing.

What should I check now? May be I’ve missed something.

kdr · August 21, 2024, 4:06pm

kdr · August 21, 2024, 4:07pm

Gsmitt · August 21, 2024, 10:57pm

Hey @kdr

I had that happen also, For example my index set was graylog_2340 and in my logs it was scream it can find index graylog_2340, I end up create that index graylog_2340 and rotating my index set,
I would check you logs ( i.e., ES/OS and Graylog) look for any issues.

kdr · August 24, 2024, 5:27pm

@Gsmitt Greetings,

I have some good news.

According to your advice, I’ve updated ES from 7.0.0 to 7.10.2. Then I found this wonderful video:

And successfully migrated from ES 7.10.2 to OpenSearch 1.3.2. After this my versions were:

Graylog - 4.3
OpenSeasrch - 1.3.2
Mongo - 3.6

So I started to check the way to update to the following versions:

Graylog - 5.2.9
OpenSeasrch - 2.15.0
Mongo - 7

But I’d faced some issues with Mongo - 3.6. In order to solve them, firstly step by step I’ve updated Mongo version:

3.6 -> 4.0 -> 4.2 -> 4.4 -> 5.0 -> 6.0 -> 7.0

Now I have the following versions, which work perfectly:

Graylog - 4.3
Mongo - 7.0
OpenSearch - 1.3.2

So now I’d like to continue my works and upgrade:

Graylog 4.3 => 5.2.9 
OpenSearch 1.3.2 => 2.15.0

Tell me please, which intermediate versions shoud I use?
Also I’ve found that after each OpenSearch version update I have to make _reindex for my indexes. Is it true?

Gsmitt · August 26, 2024, 11:33pm

Hey @kdr

that’s awesome.

Graylog I would upgrade to the latest version of 4.x.x then upgrade to 5.x

I cant remember but I believe upgrading to the latest version of 1.3.x then to 2.0 → 2.15. But i think the new version of OpenSearch is 2.16. before you do that check with Graylog on capability version with OpenSearch. I know they had some issues.

kdr · August 29, 2024, 9:07am

@Gsmitt Hey,

I’ve migrated my data without updating Graylog from 4.x.x to 5.x and OpenSearch from 1.3.x to 2.0 on my new server via snapshots.

On destination server I restored snapshots and made one more _reindex to update OpenSearch version inside my indices. After this, I had some issues with inputs and node id. Namely in migrated indices was old node_id and input_id from previous server. That’s why these indices didn’t show in Streams.

How I solved it: I’ve checked id for my messages in transferred index:

docker exec -it graylog-opensearch-1 curl -X GET "http://localhost:9200/icann_19/_search?pretty=true&size=10" -H 'Content-Type: application/json' -d'
{
  "query": {
    "match_all": {}
  }
}'

And opened it in browser:
https://graylog-server.com/messages/icann_19/9cfe95e1-7849-11ee-a5d6-0242ac130004

There I saw error:

Fetching input 60c479f795a0380b84927ee1 failed with status: FetchError: There was an error fetching a resource: . Additional information: Input <60c479f795a0380b84927ee1> not found!

In order to solve it, I’ve disabled read-only status for my indices and use API request to update node_id and input_id. Here is commands:

disable read-only

curl -X PUT "http://localhost:9200/icann_19/_settings" -H 'Content-Type: application/json' -d '
{
  "index.blocks.write": false
}'

change ids in indices

curl -X POST "http://localhost:9200/icann_19/_update_by_query" -H 'Content-Type: application/json' -d '
{
  "script": {
    "source": "ctx._source.gl2_source_input = params.new_gl2_source_input; ctx._source.gl2_source_node = params.new_gl2_source_node; ctx._source.streams = params.new_streams;",
    "lang": "painless",
    "params": {
      "new_gl2_source_input": "669fae8644ef1901cd267bc1",
      "new_gl2_source_node": "e8d678b3-13be-4f9c-81d9-bbda85ec57dc",
      "new_streams": ["66cf288195694f34060d6782"]
    }
  },
  "query": {
    "match_all": {}
  }
}'

enable read-only

curl -X PUT "http://localhost:9200/icann_19/_settings" -H 'Content-Type: application/json' -d '
{
  "index.blocks.write": true
}'

Then in Graylog interface I made “Recalculate” and that’s it! I could see my data in Streams!

My suffering is finally over Much appreciated for your help. Peace to you

Gsmitt · August 30, 2024, 2:52am

thats awesome @kdr

Thank for show the results, I personal like how you adjust the id.

system · October 29, 2024, 2:52am

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Migration issue OpenSearch troubleshoot	1	373	September 10, 2022
Migrating from Elasticsearch to OpenSearch version 1.3.x OpenSearch upgrade	4	903	February 6, 2023
Remote cluster between es and opensearch OpenSearch troubleshoot	10	555	April 6, 2023
Update 2.4 to 2.9 OpenSearch	4	241	September 27, 2023
Opensearch Migration Open Source Elasticsearch and Kibana	2	460	February 10, 2023

Data not visible in web interface after migration on OpenSearch

Related topics