Versions OpenSearch: 2.3.0
Databricks: Runtime 10.4 LTS
Describe the issue:
Unable to delete child documents in opensearch from databricks. Using elasticsearch-spark-30_2.12-7.15.2.jar
The job runs successfully, but the child record remains in the index after refresh. No issues in deleting the parent record.
Configuration:
import boto3
import requests
import json
from pyspark.sql.types import *
import pyspark.sql.functions as F
data_df.write
.mode(append)
.format(“org.elasticsearch.spark.sql”)
.option(“es.nodes”, ‘https://’ + [“HOST”])
.option(“es.port”, [“PORT”])
.option(“es.net.ssl”, “true”)
.option(“es.net.ssl.cert.allow.self.signed”, “true”)
.option(“es.net.http.auth.user”, “USER”)
.option(“es.net.http.auth.pass”, “PASSWORD”)
.option(“es.nodes.wan.only”, “true”)
.option(‘es.nodes.discovery’, ‘false’)
.option(“spark.es.scheme”, “https”)
.option(‘es.resource’, index_name)
.option(‘es.mapping.id’, <_id field of the record to be delete>)
.option(‘es.mapping.join’, )
.option(‘es.index.auto.create’, ‘false’)
.option(‘es.write.operation’, delete)
.option(‘es.http.timeout’, timeout_hrs)
.option(‘es.http.retries’, retry_num)
.save()
Relevant Logs or Screenshots: