Unable to send data from packetbeat OSS through logstash

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
opensearch version is 2.6 and OS is Ubuntu

Describe the issue:
I am trying to ingest a data from packetbeat to opensearch through logstash and its failing. Any idea if any other settings needs to be activated?

Configuration:

Relevant Logs or Screenshots:

[2023-03-14T20:57:14,498][WARN ][logstash.outputs.opensearch] Could not index event to OpenSearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"dnsamtrap-2023.03.14", :routing=>nil}, {"network"=>{"type"=>"ipv4", "bytes"=>197, "community_id"=>"1:FDgcbt/sozQ4ptt/kNRUWvOR6S4=", "transport"=>"udp", "direction"=>"egress", "protocol"=>"dns"}, "client"=>{"ip"=>"192.168.5.112", "port"=>37687, "bytes"=>70}, "dns"=>{"op_code"=>"QUERY", "flags"=>{"recursion_available"=>true, "authentic_data"=>false, "truncated_response"=>false, "authoritative"=>false, "checking_disabled"=>true, "recursion_desired"=>true}, "authorities_count"=>0, "response_code"=>"NOERROR", "resolved_ip"=>["184.105.176.47"], "opt"=>{"do"=>true, "version"=>"0", "udp_size"=>4096, "ext_rcode"=>"NOERROR"}, "header_flags"=>["RD", "RA", "CD", "DO"], "type"=>"answer", "additionals_count"=>0, "id"=>45192, "answers_count"=>2, "question"=>{"type"=>"A", "top_level_domain"=>"net", "name"=>"community.emergingthreats.net", "subdomain"=>"community", "class"=>"IN", "etld_plus_one"=>"emergingthreats.net", "registered_domain"=>"emergingthreats.net"}, "answers"=>[{"data"=>"emergingthreats.hosted-by-discourse.com", "ttl"=>"300", "name"=>"community.emergingthreats.net", "type"=>"CNAME", "class"=>"IN"}, {"data"=>"184.105.176.47", "ttl"=>"600", "name"=>"emergingthreats.hosted-by-discourse.com", "type"=>"A", "class"=>"IN"}]}, "@version"=>"1", "related"=>{"ip"=>["192.168.5.112", "208.67.222.222", "184.105.176.47"]}, "event"=>{"start"=>"2023-03-14T15:27:12.954Z", "end"=>"2023-03-14T15:27:13.027Z", "type"=>["connection", "protocol"], "category"=>["network_traffic", "network"], "dataset"=>"dns", "kind"=>"event", "duration"=>73138000}, "server"=>{"ip"=>"208.67.222.222", "port"=>53, "bytes"=>127}, "resource"=>"community.emergingthreats.net", "destination"=>{"ip"=>"208.67.222.222", "port"=>53, "bytes"=>127}, "method"=>"QUERY", "type"=>"dns", "ecs"=>{"version"=>"1.8.0"}, "@timestamp"=>2023-03-14T15:27:12.954Z, "host"=>{"name"=>"dns-am-trap"}, "tags"=>["beats_input_raw_event", "_grokparsefailure"], "status"=>"OK", "source"=>{"ip"=>"192.168.5.112", "port"=>37687, "bytes"=>70}, "query"=>"class IN, type A, community.emergingthreats.net", "agent"=>{"version"=>"7.12.1", "type"=>"packetbeat", "id"=>"13453dea-bcdd-4f7f-93ae-ac66eb2d1017", "ephemeral_id"=>"809cbf7c-17e9-4679-a04d-8c05a025ebeb", "name"=>"dns-am-trap", "hostname"=>"dns-am-trap"}}], :response=>{"index"=>{"_index"=>"dnsamtrap-2023.03.14", "_id"=>"fMm74IYBFnhqGyylkvM-", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [host] of type [text] in document with id 'fMm74IYBFnhqGyylkvM-'. Preview of field's value: '{name=dns-am-trap}'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:1646"}}}}}
2

This is my logstash config

input {
  beats {
    port => 5044
  }
}
output {
  opensearch {
    hosts => ["https://localhost:9200"]
    user => admin
    password => admin
    ssl => true
    ssl_certificate_verification => false
    index => "%{[@metadata][beat]}-%{[@metadata][version]}"
    action => "create"
  }
}
3

Hello @blason,

I think a similar problem was discussed here, with a link to the explanation here.

What type does “host” currently have in your index (dnsamtrap-2023.03.14)? It seems like it is text, when it should be “object” i think.

Take a look a the threads I linked and tell me if you have the same problem.

BR,
Andreas

4

Thanks - Let me review that.

5

Now here is the confusion - I have two output files created. Will that cause issue? Becasue I am looking my packetbeat should send a data to beat-* indices where rest of the data where I wrote parsers should be sent to dnsamtrap-*

6

With “output files” do you mean indices?

I will not cause any problems in that case if you are ok with that. If you want to only index those events a single time maybe add a fitting tag to the documents you parsed. Maybe something like “dnsamtrap”, then at the output section add an if-else statement for that tag and send the logs to either index.

BR,
Andreas

7

This is correct - I have two output here are those.

more 02-network.conf

input {
  beats {
    port => 5044
  }
}
output {
  opensearch {
    hosts => ["https://127.0.0.1:9200"]
    user => admin
    password => admin
    ssl => true
    ssl_certificate_verification => false
    index => "%{[@metadata][beat]}-%{[@metadata][version]}"
    action => "create"
  }
}

And here is one more

more 200-rpzoutput.conf
output {
        opensearch {
    hosts => ["https://127.0.0.1:9200"]
    user => admin
    password => admin
    ssl => true
    ssl_certificate_verification => false
 #   template => "/etc/logstash/elasticsearch-template-es7x.json"
    index => "dnsamtrap-%{+YYYY.MM.dd}"
  }
}
8

Also is it not possible for packetbeat tp send a events directly to opensearch??

9

Its still not sending any logs - I made the changes

[2023-03-14T23:29:57,885][WARN ][logstash.outputs.opensearch] Could not index event to OpenSearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"dnsamtrap-2023.03.14", :routing=>nil}, {"@version"=>"1", "@timestamp"=>2023-03-14T17:59:56.738Z, "destination"=>{"ip"=>"192.168.5.112", "port"=>53, "bytes"=>52}, "type"=>"dns", "resource"=>"csp.withgoogle.com", "dns"=>{"header_flags"=>["RD", "RA"], "id"=>18081, "type"=>"answer", "additionals_count"=>0, "answers_count"=>1, "resolved_ip"=>["142.250.199.145"], "question"=>{"class"=>"IN", "name"=>"csp.withgoogle.com", "top_level_domain"=>"withgoogle.com", "etld_plus_one"=>"csp.withgoogle.com", "type"=>"A", "registered_domain"=>"csp.withgoogle.com"}, "op_code"=>"QUERY", "flags"=>{"recursion_available"=>true, "authentic_data"=>false, "authoritative"=>false, "checking_disabled"=>false, "truncated_response"=>false, "recursion_desired"=>true}, "authorities_count"=>0, "response_code"=>"NOERROR", "answers"=>[{"data"=>"142.250.199.145", "ttl"=>"300", "class"=>"IN", "name"=>"csp.withgoogle.com", "type"=>"A"}]}, "tags"=>["beats_input_raw_event", "_grokparsefailure"], "query"=>"class IN, type A, csp.withgoogle.com", "client"=>{"ip"=>"192.168.5.77", "port"=>55690, "bytes"=>36}, "related"=>{"ip"=>["192.168.5.77", "192.168.5.112", "142.250.199.145"]}, "network"=>{"direction"=>"ingress", "bytes"=>88, "type"=>"ipv4", "transport"=>"udp", "community_id"=>"1:1yNRX9fDg62ijFA0dhQYAIHiKvU=", "protocol"=>"dns"}, "ecs"=>{"version"=>"1.8.0"}, "host"=>{"name"=>"dns-am-trap"}, "method"=>"QUERY", "source"=>{"ip"=>"192.168.5.77", "port"=>55690, "bytes"=>36}, "server"=>{"ip"=>"192.168.5.112", "port"=>53, "bytes"=>52}, "event"=>{"end"=>"2023-03-14T17:59:56.807Z", "dataset"=>"dns", "type"=>["connection", "protocol"], "kind"=>"event", "start"=>"2023-03-14T17:59:56.738Z", "duration"=>68958000, "category"=>["network_traffic", "network"]}, "status"=>"OK", "agent"=>{"ephemeral_id"=>"809cbf7c-17e9-4679-a04d-8c05a025ebeb", "name"=>"dns-am-trap", "id"=>"13453dea-bcdd-4f7f-93ae-ac66eb2d1017", "type"=>"packetbeat", "version"=>"7.12.1", "hostname"=>"dns-am-trap"}}], :response=>{"index"=>{"_index"=>"dnsamtrap-2023.03.14", "_id"=>"pMpH4YYBFnhqGyylZAm3", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [host] of type [text] in document with id 'pMpH4YYBFnhqGyylZAm3'. Preview of field's value: '{name=dns-am-trap}'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:1202"}}}}}

And here is my output file now.

output {
        if "queries" or "rpz" or "deception" {
        opensearch {
        hosts => ["https://127.0.0.1:9200"]
        user => admin
        password => admin
        ssl => true
        ssl_certificate_verification => false
        #template => "/etc/logstash/elasticsearch-template-es7x.json"
        index => "dnsamtrap-%{+YYYY.MM.dd}"
        }
  }     else {
        opensearch {
        hosts => ["https://127.0.0.1:9200"]
        user => admin
        password => admin
        ssl => true
        ssl_certificate_verification => false
        index => "%{[@metadata][beat]}-%{[@metadata][version]}"
                }
        }

}