Illegal base64 character 22

OpenSearch
, ,
1

We recently upgraded a cluster to Opensearch 2.4.0. Since the upgrade, all of our snapshots have failed with the error:

{
“error” : {
“root_cause” : [
{
“type” : “repository_verification_exception”,
“reason” : “[data_snapshots] path is not accessible on cluster-manager node”
}
],
“type” : “repository_verification_exception”,
“reason” : “[data_snapshots] path is not accessible on cluster-manager node”,
“caused_by” : {
“type” : “illegal_argument_exception”,
“reason” : “Illegal base64 character 22”,
“suppressed” : [
{
“type” : “exception”,
“reason” : “#block terminated with an error”
}
]
}
},
“status” : 500
}

We are using the repository-azure plugin and the repo is a container in an Azure storage account. At this point, we can’t even list snapshots in the repo, take new snapshots, etc as we always get this error. Does anyone have any suggestions?

2

Just had the exact same problem going from 1.3.7 to 2.4.0, also using repository-azure to store snapshot data. Tracked it down to something wrong within the keystore file that defines the Azure storage account and key. Java keytool does not seem to be able to read keystore files created by opensearch-keystore, so wasn’t able to troubleshoot further.

Did not touch the keystore file during the upgrade, so not sure how this broke. Manually rebuilt the keystore on each node, did a rolling restart of all cluster nodes, then snapshot operations worked again.

Commands relevant to the repository-azure plugin (rpm install):

echo "<redacted>" | /usr/share/opensearch/bin/opensearch-keystore add -xf azure.client.default.account
echo "<redacted>" | /usr/share/opensearch/bin/opensearch-keystore add -xf azure.client.default.key
1 Like