Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
opensearch 7.10.2
Describe the issue:
trying to write spark RDD to opensearch
Configuration:
conf.set("opensearch.nodes.wan.only", "true")
conf.set("opensearch.nodes" , "my url")
conf.set("opensearch.net.ssl" , "true")
val sc = new SparkContext(conf)
// Read data from Elasticsearch indices
val numbers = Map("one" -> 1, "two" -> 2, "three" -> 3)
val airports = Map("OTP" -> "Otopeni", "SFO" -> "San Fran")
sc.makeRDD(Seq(numbers, airports)).saveToOpenSearch("spark/docs")
Relevant Logs or Screenshots:
getting error
Exception in thread “main” org.opensearch.hadoop.OpenSearchHadoopIllegalArgumentException: Cannot detect OpenSearch version - typically this happens if the network/OpenSearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting ‘opensearch.nodes.wan.only’
at org.opensearch.hadoop.rest.InitializationUtils.discoverClusterInfo(InitializationUtils.java:423)
at org.opensearch.spark.rdd.OpenSearchSpark$.doSaveToOpenSearch(OpenSearchSpark.scala:114)
at org.opensearch.spark.rdd.OpenSearchSpark$.saveToOpenSearch(OpenSearchSpark.scala:89)
at org.opensearch.spark.rdd.OpenSearchSpark$.saveToOpenSearch(OpenSearchSpark.scala:84)