Indexing with an IP type caused error

OpenDistro
1

I have been using ELK + OD for almost 2 years. I also used template to map my data into right type. One of the columns is in IP type and it starts getting error after upgrading to OpenSearch. The error is:

[2021-06-22T09:58:28,388][WARN ][logstash.outputs.elasticsearch][csv2es][4b0c260d0d4cc87a2e2573c6915e7d31d43c8e67fba73a496b973901f6797306] Could not index event to Elasticsearch. {:status=>400, :action=>[“index”, {:_id=>nil, :_index=>“index20210622”, :_routing=>nil, :_type=>“doc”}, #LogStash::Event:0x5966a733], :response=>{“index”=>{“_index”=>“index20210622”, “_type”=>“doc”, “_id”=>“hsxwMXoBD8uHKzxEr5KM”, “status”=>400, “error”=>{“type”=>“illegal_argument_exception”, “reason”=>“mapper [client_ip] cannot be changed from type [ip] to [text]”}}}}

My template looks like this:

"mappings": {
	"properties": {
		"@timestamp": {
			"type": "date"
		},
		"client_ip": {
			"type": "text"
		}
	}
}

Is the type IP not supported in OpenSearch?

Thanks,

2

It seems like the document about index template at https://docs-beta.opensearch.org/opensearch/index-templates/ is inaccurate or incomplete.

For example, using the command below resulted an error:
GET _index_template/daily_logs

instead, using the one below is correct:
GET _template/daily_logs

I think I need the latest complete document about the template syntax in order to resolve this issue.

Thanks,

3

I don’t know of any changes like this.

What version of Open Distro were you using? Also, have you changed your version of Logstash or anything else in your setup?

4

I’m using OpenSearch 1.0.0-rc1 with Logstash 7.11.1. Before upgrading to OpenSearch I was using OD 1.13.1 with Logstash 7.11.1 with no such an issue, I meant, the template works fine with no error in Logstash log.

5

This sounds similar to: [BUG] Logstash cannot submit events to OpenSearch since version change · Issue #667 · opensearch-project/OpenSearch · GitHub

If the issue is the same, you might try setting document_type => "_doc" in the elasticsearch output config as a temporary workaround. A version compatibility flag for opensearch is in the works, but isn’t available yet.

6

This suggestion did eliminate the errors, however, it also bypass the template and creates the index with column names of its default data type interpretation. I do need the template to build the columns with data type I need. I’ll wait for next release and hope it will be addressed. Thanks.