Can't see IIS windows logs on the OpenSearch Dashboards

OpenDistro
1

My setup is the following:
Opensearch with Opensearch Dashboards and Logstash.
The last versions of everything.

Logstash collect the logs from dockers and from Windows servers and forwarding the logs to OpenSearch Dashboards.
I can see all the docker logs in Dashboards.

The problem is i cannot see IIS Windows logs on OpenSearch Dashboards.
The weird part that i can see them coming inside the Logstash but for some reason does not forwarding to Dashboards.
Cannot find any errors related to that in the opensearch logs.

Here is the windows logs that forwarding to Dashboards:

2022-02-03 13:02:58,501 [11820] DEBUG Start LoginControllerIndex  userName:fack27 returnUrl:
2022-02-03 13:02:58,501 [11820] DEBUG Get login index!!
2022-02-03 13:02:58,517 [11260] INFO  End SearchApiController SearchByCompanyName term  
2022-02-03 13:02:58,548 [8028] INFO  End SearchApiController SearchByCompanyName term  456767
2022-02-03 13:02:58,579 [5876] INFO  Start CompanyDetailsController Index  dcros: 645456 , regis:

Hers is the logs thats NOT forwarding to Dashboards:

#Software: Microsoft Internet Information Services 10.0
#Version: 1.0
#Date: 2022-02-02 22:00:02
#Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) cs(Referer) sc-status sc-substatus sc-win32-status time-taken x-forwarded-for
2022-02-02 21:59:44 10.10.10.10 GET /Content/images/shared/someimage.jpg - 443 - 10.10.10.10  Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/97.0.4692.99+Safari/537.36 https://mywebsiteContent/css?v=ZaqwcPGHUWOJWEWEEWjePv41 200 0 0 0 192.168.0.0
2022-02-02 21:59:44 10.10.10.10. GET /home/Touch _=15784653 443 - 10.10.10.10  Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/97.0.4692.99+Safari/537.36 https://mywebsite/Home/Opportunities 200 0 0 31 192.168.0.0

Here is how i see the input of not forwarding to OpenSearch logs in Logstash:

Screen Shot 2022-02-03 at 14.13.25
Screen Shot 2022-02-03 at 14.13.251316×315 65.9 KB

Logstash.conf:

input {                                                                                                                                                                             
 gelf {                                                                                                                                                                             
    port => 12201                                                                                                                                                                   
#    type => docker                                                                                                                                                                 
    use_tcp => true                                                                                                                                                                 
  }                                                                                                                                                                                 
} 

output {                                                                                                                                                                            
  stdout {}                                                                                                                                                                         
  opensearch {                                                                                                                                                                      
    hosts => ["https://opensearch-node:9200"]                                                                                                                                      
    index => "opensearch-index"                                                                                                                                           
    user => "admin"                                                                                                                                                                 
    password => "1234"                                                                                                                                                             
    ssl => true                                                                                                                                                                     
    ssl_certificate_verification => false                                                                                                                                           
  }                                                                                                                                                                                 
}

Thanks for any advice.

1 Like
2

Ok, i figured it out.
I should use type => IISLog in input of logstash.conf and stdout { codec => rubydebug } in output.

input {
	file {
		type => "IISLog"
		path => "C:/inetpub/logs/LogFiles/W3SVC*/*.log"
		start_position => "beginning"
	}
}

filter {

	# ignore log comments
	if [message] =~ "^#" {
		drop {}
	}
 
 	# check that fields match your IIS log settings
	grok {
        match => ["message", "%{TIMESTAMP_ISO8601:log_timestamp} %{IPORHOST:site} %{WORD:method} %{URIPATH:page} %{NOTSPACE:querystring} %{NUMBER:port} %{NOTSPACE:username} %{IPORHOST:clienthost} %{NOTSPACE:useragent} (%{URI:referer})? %{NUMBER:response} %{NUMBER:subresponse} %{NUMBER:scstatus} %{NUMBER:time_taken}"]
	}
  
	# set the event timestamp from the log
	# https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html
	date {
		match => [ "log_timestamp", "YYYY-MM-dd HH:mm:ss" ]
		timezone => "Etc/UCT"
	}
	
	# matches the big, long nasty useragent string to the actual browser name, version, etc
	# https://www.elastic.co/guide/en/logstash/current/plugins-filters-useragent.html
	useragent {
		source=> "useragent"
		prefix=> "browser_"
	}
	
	mutate {
		remove_field => [ "log_timestamp"]
	}
}

# output logs to console and to elasticsearch
output {
    stdout { codec => rubydebug }
	elasticsearch { hosts => ["localhost:9200"] }
}
2 Likes