Data corruption(checksum failed) on Block Storage

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpenSearch 2.4 / CentOS

Describe the issue:
Hi,
I’m investigating whether there have been cases of data conflicts when building OpenSearch nodes using block storage such as AWS EBS.
I know that in various cases, Lucene-based services on NFS can cause errors. But I would like to know if this could cause similar problems in block storage like EBS.
(I may be wrong, even block storage is thought to be more stable, it also uses the network, which may cause performance delay issues.)

Does anybody know of any cases related to this, or any reasons why Lucene causes data corruption in network-based storage? And can we experience a similar experience with NFS in block storage?

Configuration:

Relevant Logs or Screenshots:

2

AWS EBS is quite standard and may even bring better performance than local disk. the fact it’s offered by means of networking doesn’t really matter, since it’s a block device and that’s on amazon’s side.

NFS on the other hand is much higher in the (storage) layers. Awful performance is to be expected, obviously, and even worse, like you mentioned.

1 Like
3

Hi, @elge
Thanks for reply.
Yes, you are right. In fact, I’m operating a ceph-based block storage, and I’ve not confirmed whether the issue is reproduced on AWS EBS. The problem was caused by the disk cache option of OpenStack Nova compromising data integrity when connected to Block Storage. The detailed information was shared at the OpenSearch Conference: