attaching docker compose for fluentbit, opensearch & opensearch dashboard
version: ‘3’
services:
fluent-bit:
container_name: fluent-bit
image: fluent/fluent-bit
volumes:
- ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf
- ./test.log:/var/log/test.log
networks:
- dscnet
opensearch:
container_name: opensearch
image: opensearchproject/opensearch:latest
environment:
- discovery.type=single-node
- bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
- “OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m” # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
- OPENSEARCH_INITIAL_ADMIN_PASSWORD=Developer@123
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems
hard: 65536
ports:
- 9200:9200
- 9600:9600 # required for Performance Analyzer
networks:
- dscnet
dashboards:
image: opensearchproject/opensearch-dashboards:latest
container_name: opensearch-dashboards
ports:
- 5601:5601
expose:
- “5601”
environment:
OPENSEARCH_HOSTS: ‘[“https://10.177.164.51:9200”]’
depends_on:
- opensearch
networks:
- dscnet
networks:
dscnet:
external: true
driver: overlay
name: test-net
fluent-bit.conf
[INPUT]
name tail
refresh_interval 5
path var/log/test.log
read_from_head true
[OUTPUT]
Name http
Match *
Host data-prepper
Port 2021
URI /log/ingest
Format json
docker compose for data prepper
version: ‘3.7’
services:
data-prepper:
image: opensearchproject/data-prepper:2.0.0
container_name: data-prepper
volumes:
- ./log_pipeline.yaml:/usr/share/data-prepper/pipelines/log_pipeline.yaml
ports:
- 2021:2021
networks:
- test-net
networks:
test-net:
external: true
[if i use depends upon i get error]
log-pipeline:
source:
http:
ssl: false
processor:
- grok:
match:
log: [ “%{COMMONAPACHELOG}” ]
sink:
- opensearch:
hosts: [“https://10.177.164.51:9200”]
insecure: true
username: admin
password: Developer@123
index: test_logs
test.log file has necessary permissions.
Fluent-bit
Opensearch
Opensearch-dashboards
dataprepper
all are running without any error but fluent-bit is not able to read the test.log file