Data-prepper example log-ingestion not working

Data Prepper
1

Versions:

Latest version

Describe the issue:

I tried the example of log-ingestion, downloaded all yaml files, startet the first docker (docker-compose.yaml), logged in to http://127.0.0.1:5601/ and all looks fine.

Next step is loading an Apache log file as “test.log” by starting a second docker (docker-compose-dataprepper.yaml).

It starts an the last line is “… Started http source on port 2021…”.

No errors so far, but it doesn’t start ingesting the test.log file.

At the Dashboard I can create an index pattern and see “apache_logs” for selection.

After creating an index pattern and going to Discover there is only an empty index. Nothing has been ingested from test.log.

Any idea why?

Regards Bernd

Configuration:

Relevant Logs or Screenshots:

2

Hey @bernd ,

please share all your config files so we can get a better picture to what might be going wrong.

Leeroy.

3

docker-compose.yaml

version: '3'
services:
  fluent-bit:
    container_name: fluent-bit
    image: docker.io/fluent/fluent-bit
    volumes:
      - ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf
      - ./test.log:/var/log/test.log
    networks:
      - opensearch-net
  opensearch:
    container_name: opensearch
    image: docker.io/opensearchproject/opensearch:latest
    environment:
      - discovery.type=single-node
      - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
      - "OPENSEARCH_INITIAL_ADMIN_PASSWORD=Developer@123"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems
        hard: 65536
    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer
    networks:
      - opensearch-net
  dashboards:
    container_name: opensearch-dashboards
    image: docker.io/opensearchproject/opensearch-dashboards:latest
    ports:
      - 5601:5601
    expose:
      - "5601"
    environment:
      OPENSEARCH_HOSTS: '["https://opensearch:9200"]'
    depends_on:
      - opensearch
    networks:
      - opensearch-net

networks:
  opensearch-net:

docker-compose-dataprepper.yaml

version: '3.7'
services:
  data-prepper:
    container_name: data-prepper
    image: opensearchproject/data-prepper:latest
    volumes:
      - ./log_pipeline.yaml:/usr/share/data-prepper/pipelines/log_pipeline.yaml
      - ./data-prepper-config.yaml:/usr/share/data-prepper/config/data-prepper-config.yaml
    ports:
      - 2021:2021
    networks:
      - opensearch-net

networks:
  opensearch-net:
    driver: bridge

log_pipeline.yaml

log-pipeline:
  source:
    http:
      ssl: false
  processor:
    - grok:
        match:
          log: [ "%{COMBINEDAPACHELOG}" ]
  sink:
    - opensearch:
        hosts: [ "https://opensearch:9200" ]
        insecure: true
        username: admin
        password: Developer@123
        index: apache_logs

data-prepper-config.yaml

ssl: false
serverPort: 4900
authentication:
  http_basic:
    username: admin
    password: Developer@123

fluent-bit.conf

[INPUT]
  name                  tail
  refresh_interval      5
  path                  /var/log/test.log
  read_from_head        true

[OUTPUT]
  Name http
  Match *
  Host data-prepper
  Port 2021
  URI /log/ingest
  Format json
  HTTP_User admin
  HTTP_Passwd Developer@123

First Docker:

docker compose --project-name data-prepper -f docker-compose.yaml up

Second Docker:

docker compose --project-name data-prepper -f docker-compose-dataprepper.yaml up
4

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.