The time mapping to date type is not working in Opensearch Index

OpenSearch
, , ,
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Opensearch/OpenSearch Dashboard: v 2.11.1

Describe the issue:
I am able to to send apache logs to OpenSearch Dashboard through fluentbit and dataprepper pipeline to OpenSearch Dashboard without mapping time filed for index. But if I create a new index mapping for new index, the logs are not sent to Dashboard. I am not getting any errors from Dataprepper. I am not sure what is exact issue. Here is my config details:

Configuration:

[0] site_logs: [[1703689296.000000000, {}], {"host"=>"192.168.5.10", "user"=>"-", "time"=>"27/Dec/2023:15:01:36 +0000", "method"=>"GET", "path"=>"/assets/vendor/swiper/swiper-bundle.min.js", "code"=>"304", "size"=>"-", "referer"=>"http://example.com/", "agent"=>"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0"}]

[SERVICE]
    Flush        5
    Daemon       Off
    Log_Level    info
    Parsers_File /home/dataprepper/parsers.conf
    Log_File      /home/dataprepper/error.log

[INPUT]
    name             tail
    refresh_interval 5
    path             /home/dataprepper/access-log.txt
    tag              site_logs
    read_from_head true

[FILTER]
    Name parser
    Match site_logs*
    Key_Name log
    Parser apache
   

[OUTPUT]
    name http
    match *
    host logp.example.com
    port 2021
    uri /log/ingest
    format json
    #json_date_key time
    #json_date_format %d/%b/%Y:%H:%M:%S %z

Data Prepper Config:

log-pipeline:
  source:
    http:
      ssl: false

  sink:
    - opensearch:
        hosts: [ "https://os1.example.com:9200" ]
        username: admin
        password: Password
        index: rt-logs

Index Mapping:

PUT rt-logs
{
  "mappings": {
    "properties": {
      "time": {
        "type": "date",
        "format": "dd/MMM/yyyy:HH:mm:ss Z"
      }
      
    }
  }
}

Any suggestion/advice will be highly appreciated. Thanks. :slight_smile:

2

You can check the logs of OpenSearch to see if there are some error logs, it maybe caused by that the date format you specify in the mapping is not consistent with the actual format when indexing.

1 Like
3

@gaobinlong , Thank you so much for your suggestion. The issue has been fixed now.