Update a field to null in OpenSearch index

OpenSearch
,
1

Version OpenSearch: 2.3.0
Databricks: Runtime 10.4 LTS

Describe the issue:
Hi All,
I have a field called submission_code. In the current opensearch index it has a value assigned to it, Type 2. I’ve updated it to be null in the databricks side and when I run an upsert job, I see that submission_code is still “Type 2” instead of null.

Is there a reason submission_code is not being updated to null and is there a way to update it without updating the schema mapping?

Thanks,
Lyn

Configuration:

Databricks :
response = opensearch_helper.ingest_OS(‘append’,‘temp-index’,‘id’,‘upsert’,‘1h’,‘3’,updateQuery_df,‘document_join’)

Relevant Logs or Screenshots:

2

I cant seem to find any documentation from databricks on this. Do you have some that you are following that you could link? Also, what is the data type of the field? For example, if it is of type string maybe you could try sending a blank string? “”

3

Hi dtaivpp,
Yep, so for the configuration to push data from databricks to opensearch we are using the following
https://www.elastic.co/guide/en/elasticsearch/hadoop/current/configuration.html

Here is a sample of what our configuration looks like
data_df.write
.mode(append)
.format(“org.elasticsearch.spark.sql”)
.option(“es.nodes”, ‘https://’ + [“HOST”])
.option(“es.port”, [“PORT”])
.option(“es.net.ssl”, “true”)
.option(“es.net.ssl.cert.allow.self.signed”, “true”)
.option(“es.net.http.auth.user”, “USER”)
.option(“es.net.http.auth.pass”, “PASSWORD”)
.option(“es.nodes.wan.only”, “true”)
.option(‘es.nodes.discovery’, ‘false’)
.option(“spark.es.scheme”, “https”)
.option(‘es.resource’, index_name)
.option(‘es.mapping.id’, <_id field of the record to be delete>)
.option(‘es.mapping.join’, )
.option(‘es.index.auto.create’, ‘false’)
.option(‘es.write.operation’, delete)
.option(‘es.http.timeout’, timeout_hrs)
.option(‘es.http.retries’, retry_num)
.save()

The original data type is a string. We could try sending a blank string but the source system populates a lot of data with nulls and applying blank string to about 100 pieces of metadata if it is null is a lot.