Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpenSearch 2.13 (latest)
Describe the issue:
I have an AWS Opensearch Domain to which I am trying to Ingest Json codec files from GCS bucket using Logstash. I can see a log stating that my logstash has connected to Opensearch domain and a few other logs stating it is trying to fetch logs from GCS bucket.
This is my logstash configuration.
input {
google_cloud_storage {
bucket_id => "test-bucket"
json_key_file => "/etc/logstash/credentials.json"
codec => "json_lines"
}
}
filter {
}
output {
opensearch {
hosts => "https://<name>.us-east-1.es.amazonaws.com:443"
user => "admin"
password => "admin"
index => "logstash-test-1"
ssl_certificate_verification => true
}
}
I am running this config on my GKE. I created a custom image out of the official Elastic Search Logstash image and installed opensearch output plugin on the same. These are my logs.
Logs to find out that Opensearch connection has been successful:
[2024-06-18T21:26:20,336][INFO ][logstash.javapipeline ] Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
[2024-06-18T21:26:20,353][INFO ][logstash.outputs.opensearch][main] New OpenSearch output {:class=>"LogStash::Outputs::OpenSearch", :hosts=>["https://search-<domain>.us-east-1.es.amazonaws.com:443"]}
[2024-06-18T21:26:20,371][INFO ][logstash.outputs.opensearch][main] OpenSearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://admin:xxxxxx@search-<domain>.us-east-1.es.amazonaws.com:443/]}}
[2024-06-18T21:26:20,639][WARN ][logstash.outputs.opensearch][main] Restored connection to OpenSearch instance {:url=>"https://admin:xxxxxx@search-<domain>.us-east-1.es.amazonaws.com:443/"}
[2024-06-18T21:26:20,696][INFO ][logstash.outputs.opensearch][main] Cluster version determined (2.13.0) {:version=>2}
[2024-06-18T21:26:20,735][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/usr/share/logstash/pipeline/logstash.conf"], :thread=>"#<Thread:0x701b2684 /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}
This log is continously appearing but no signs of new logs found/pushed to domain:
Fetching blobs from test-bucket
Also, these logs appear very often, but I am assuming it is due to missing elastic search live cluster. Let me know if these are good to be ignored
[2024-06-18T21:27:43,039][ERROR][logstash.licensechecker.licensereader] Unable to retrieve Elasticsearch cluster info. {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError}
[2024-06-18T21:27:43,041][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"No Available connections"}
[2024-06-18T21:27:43,099][INFO ][logstash.licensechecker.licensereader] Failed to perform request {:message=>"elasticsearch: Name or service not known", :exception=>Manticore::ResolutionFailure, :cause=>#<Java::JavaNet::UnknownHostException: elasticsearch: Name or service not known>}
[2024-06-18T21:27:43,101][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch: Name or service not known"}
Cheers