Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Hi All,
I am using Localstack to set up AWS environment in my Test project (Java-maven) and creating an S3 bucket. Also I have installed repository-s3 on the localstack container as well.
String repositoryName = “newsflo-opensearch-snapshot”;
PutRepositoryRequest request = new PutRepositoryRequest(repositoryName);
request.type(“s3”);
request.settings(Settings.builder()
.put(“bucket”, bucketName)
.build());
AcknowledgedResponse response = null;
try {
response = openSearchClient.snapshot().createRepository(request, RequestOptions.DEFAULT);
} catch (IOException e) {
throw new RuntimeException(e);
}
I get the error as below
OpenSearchStatusException[OpenSearch exception [type=repository_exception, reason=[newsflo-opensearch-snapshot] repository type [s3] does not exist]]
Could anyone please help me to create repository of type s3 ? Am i missing anything please help
Describe the issue :
Configuration :
Relevant Logs or Screenshots :
at org.opensearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:207)
at org.opensearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2228)
at org.opensearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2205)
at org.opensearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1924)
at org.opensearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1877)
at org.opensearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1845)
at org.opensearch.client.SnapshotClient.createRepository(SnapshotClient.java:123)
at com.elsevier.rdp.newsflo.bdd.steps.StepsDefinitions.createSBucketToStoreTheOpenSearchIndexSnapshot(StepsDefinitions.java:108)```
You can check whether repository-s3 plugin is installed or not:
GET _cat/plugins
, note that the plugin should be installed on every node.
Hi @gaobinlong thanks for your suggestion. I have installed the plugins and restarted the container and checked whether repository-s3 plugin is available or not. Yes it is available. However when i
PutRepositoryRequest request = new PutRepositoryRequest(repositoryName);
request.type("s3");
request.settings(Settings.builder()
.put("bucket", bucketName)
.put("region", LOCALSTACK.getRegion())
.put("client.access_key", LOCALSTACK.getAccessKey())
.put("client.secret_key", LOCALSTACK.getSecretKey())
.put("base_path", "/")
.put("endpoint", "http://host.docker.internal:" + LOCALSTACK.getEndpointOverride(LocalStackContainer.Service.S3).getPort())
.build());
try {
openSearchClient.snapshot().createRepository(request, RequestOptions.DEFAULT);
} catch (Exception e) {
log.warn("{}", e.getMessage());
}
I get below exception . Any suggestions please, I followed the documentation
2024-02-26 10:28:36 org.opensearch.repositories.RepositoryVerificationException: [newsflo-housekeeper-lambda-snapshot-repo] path is not accessible on cluster-manager node
I installed the plugin and restarted OpenSearch docker container and I can see repository-s3 plugin is available. However, when I create a repository using below code
PutRepositoryRequest request = new PutRepositoryRequest(repositoryName);http://host.docker.internal :” + LOCALSTACK.getEndpointOverride(LocalStackContainer.Service.S3).getPort())
I get an error as below
OpenSearchStatusException[OpenSearch exception [type=repository_verification_exception, reason=[newsflo-housekeeper-lambda-snapshot-repo] path is not accessible on cluster-manager node]]; nested: OpenSearchException[OpenSearch exception [type=i_o_exception, reason=Unable to upload object [tests-jGV-KdnHSLemDqapJ7I5AQ/master.dat] using a single upload]]; nested: OpenSearchException[OpenSearch exception [type=sdk_client_exception, reason=Failed to load credentials from IMDS.]]; nested: OpenSearchException[OpenSearch exception [type=unchecked_i_o_exception, reason=java.net.SocketTimeoutException: Connect timed out]]; nested: OpenSearchException[OpenSearch exception [type=socket_timeout_exception, reason=Connect timed out]];
I would like to explain the context of the above error. I am trying to run my test suite with the following pre-requisite,s where I need an S3 bucket and Opensearch cluster so that I can create a Repository of type S3 (mention the S3 bucket while creating the repo) and then I need to create a few indices and then I need to take the snapshot to the repository of type S3. Also, I wanted to run this test on my Jenkins pipeline. So my plan is to bring up LocalStack docker container to create S3 and Opensearch docker container to bring up the Opensearch Cluster. Also using RestHighLevelClient I will connect with the cluster for different operations like create repo, take snapshot etc. SO the catch here is the S3 I am referring to is the S3 bucket created in the LocalStack not the actual AWS S3. Can anyone please review this and confirm whether this is possible ?
system
April 28, 2024, 9:10pm
6
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.
