I want to automate log ingestion similar similar to what is in the playground

DevOps
1

Hello everyone, I want to create fake logs same as what is in the playground with the same indexes.
Does anyone have an idea how I can do that ? I assume I should also make changes to my log pipeline which looks as the following :

log-pipeline:
source:
http:
ssl: false
processor:
- grok:
match:
log: [ “%{COMMONAPACHELOG}” ]
sink:
- opensearch:
hosts: [ “https://opensearch:9200” ]
insecure: true
username: admin
password: admin
index: apache_logs

any help will be appreciated

2

Hey @zorghost

what kind of log shipper do you have?

3

I am using Fluentbit, so it goes like this : data-prepper → fluent-bit → opensearch → dashboards

4

Hey @zorghost

Fluentbit, create a file called test.log, throw some message in there. When you restart fluentbit it should read from the top of the file.

[INPUT]
  name                  tail
  refresh_interval      5
  path                  /var/log/test.log
  read_from_head        false --> true
  tag test

The Read_from_Head For new discovered files on start (without a database offset/position), read the content from the head of the file, not tail.
Default is false.