I want to automate log ingestion similar similar to what is in the playground

Hello everyone, I want to create fake logs same as what is in the playground with the same indexes.
Does anyone have an idea how I can do that ? I assume I should also make changes to my log pipeline which looks as the following :

log-pipeline:
source:
http:
ssl: false
processor:
- grok:
match:
log: [ “%{COMMONAPACHELOG}” ]
sink:
- opensearch:
hosts: [ “https://opensearch:9200” ]
insecure: true
username: admin
password: admin
index: apache_logs

any help will be appreciated

Hey @zorghost

what kind of log shipper do you have?

I am using Fluentbit, so it goes like this : data-prepper → fluent-bit → opensearch → dashboards

Hey @zorghost

Fluentbit, create a file called test.log, throw some message in there. When you restart fluentbit it should read from the top of the file.

[INPUT]
  name                  tail
  refresh_interval      5
  path                  /var/log/test.log
  read_from_head        false --> true
  tag test

The Read_from_Head For new discovered files on start (without a database offset/position), read the content from the head of the file, not tail.
Default is false.

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.