Default Index mapping _doc change to doc

OpenSearch
1

Hello there,

I would like to ask for help with change of Mapping type _doc to doc.

My new position includes taking care of OpenSearch that runs in cluster on 7 nodes.

Somehow, we managed to delete security index and it took out our access way into dashboard.

After dealing with it our setup is gone and I can’t figure out how it was previous setup.

Logstash sends data from File Beat to OpenSearch but can’t update Index that was somehow created with Mapping type _doc but Logstash wants to update it and expects type = doc.

[2021-12-08T11:59:32,761][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>[“index”, {:_id=>nil, :_index=>“wfm-test-debug-2021.12.08”, :_routing=>nil, :_type=>“doc”}, #LogStash::Event:0x7c935a4d], :response=>{“index”=>{“_index”=>“wfm-test-debug-2021.12.08”, “_type”=>“doc”, “_id”=>“”, “status”=>400, “error”=>{“type”=>“illegal_argument_exception”, “reason”=>“Rejecting mapping update to [wfm-test-debug-2021.12.08] as the final mapping would have more than 1 type: [_doc, doc]”}}}}

Default template for index that logstash has in default.json is

“order”: 0,
“version”: 50003,
“index_patterns”: [
“*”
],
“settings”: {
“index”: {
“codec”: “best_compression”,
“routing”: {
“allocation”: {
“require”: {
“box_type”: “hot”
}
}
},
“analysis”: {
“analyzer”: {
“whitespace”: {
“type”: “keyword”
}
}
},
“number_of_shards”: “2”
}
},
“mappings”: {},
“aliases”: {}
}

Forgive mine ignorance and Any advice is appreciated

Thank you in advanced Krejcar S.

2

document_type: _doc is the only document_type that can be used in elasticsearch 7.x and up, including OpenSearch. The ability to set different document_types, to support different mapping in a single index, is a legacy feature which has been deprecated and no longer supported.
I’m not sure what details have been left out of this scenario you’re describing that would explain how you got into this mess. I suspect you are running an old logstash, which is fine, but might require you to explicit set document_type in your elasticsearch{} output filter; example below:

output {
        elasticsearch {
            id => "elasticsearch-blahblah"
            hosts => [ "192.168.0.1:9200", "192.168.0.2:9200" ]
            document_type => "_doc"  #This is forward-compatible with 7.x
            index => "current-wfm-test-debug"
        }
}

If the scenario you’re describing came about as a result as a major version upgrade from before 7.x, you likely need to do some re-indexing.

3

Hello Mhoydis,

Thank you for reply.

This problem was caused by accidentaly deleting Security index in Opensearch 1.0.This caused that we had to create a new one and old setting was gone. After this was resolved logstash wanted to use document_type doc which was used previously but opensearch now started creating indexis with document_type _doc.

We resolved this by setting in the logstash template document_type to _doc and restarted logstash itself. This worked.

Now I have still no idea why this happened but I started in this company not too long ago a this system that replaced Kibana was implemented by someone who is no longer here nor he left any useable documentation…

Still thank you for reply !

4

Solution in this case:

I edited the in the logstash config=>conf.d=>output file by edditing document_type

output {
if ![@metadata][document_id] {
elasticsearch {
hosts => [“https://localhost:port”]
ssl => true
ssl_certificate_verification => true
cacert => “/home/kibana/logstash/config/cert.pem”
index => “%{index_replace}-%{+YYYY.MM.dd}”
document_type => “_doc”
template_name => “default”
template => “/.logstash/config/templates/default.json”
user => admin
password => admin
}
}
else {
elasticsearch {
hosts => [“https://localhost:port”]
ssl => true
ssl_certificate_verification => true
cacert => “/home/kibana/logstash/config/cert.pem”
index => “%{index_replace}-%{+YYYY.MM.dd}”
document_id => “%{[@metadata][document_id]}”
document_type => “_doc”
template_name => “default”
template => “/.config/templates/default.json”
user => admin
password => admin
}
}
}

1 Like