Logstash conf Converter to Data Prepper

Data Prepper
, ,
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Hello, I have downloaded on my RHEL 8.6:
opensearchproject/data-prepper:2.0.1

Describe the issue:
I used logstash before this and want to migrate do Data Prepper. I tried to follow:

but it isnot working for me.

Configuration:

I want to migrate this logstash.conf configuration.

input {
  tcp {
    mode => "server"
    host => "IP"
    port => "6379"
    ssl_enable => "true"
    ssl_cert => "/usr/share/logstash/config/server.crt"
    ssl_key => "/usr/share/logstash/config/privateKey.key"
    ssl_key_passphrase => "PW"
    ssl_verify => "false"
    ssl_cipher_suites => ['TLS_AES_256_GCM_SHA384', 'TLS_AES_128_GCM_SHA256', 'TLS_CHACHA20_POLY1305_SHA256', 'TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384', 'TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384', 'TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256', 'TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256', 'TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256', 'TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256', 'TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384', 'TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384', 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256', 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256']
    ssl_supported_protocols => ['TLSv1.2', 'TLSv1.3']
    codec => "json_lines"
    tags => "ssl_TCPinput"
  }
}

filter {
if [LogType] == "TrxPersist" {
        mutate { add_tag => "trx_log" }
}
else if [LogType] == "TrxPostProc" {
  mutate { add_tag => "trx_time" }
 }

#filtr-indexy
if [appname] == "INT_EDDIE" {
mutate { add_field => { "[@metadata][target_index]" => "eddie-int" } }
}
}



output {
if [enviroment] == "integration" {
if [appname] == "INT_EDDIE" {
opensearch {
hosts => ["IP:9200"]
ssl => true
ssl_certificate_verification => false
user => "admin"
password => "admin"
index => "%{[@metadata][target_index]}-temporary-%{+YYYY-MM-dd}"
manage_template => false
}
}

else {
opensearch {
hosts => ["IP:9200"]
ssl => true
ssl_certificate_verification => false
user => "admin"
password => "admin"
index => "trash-int-%{+YYYY.MM.dd}"
manage_template => false
}
}
}

else {
opensearch {
hosts => ["IP:9200"]
ssl => true
ssl_certificate_verification => false
user => "admin"
password => "admin"
index => "trash"
manage_template => false
}
}
}

Relevant Logs or Screenshots:

I tried:
docker run --name data-prepper -p 4900:4900 -v ./logstash.conf:/usr/share/data-prepper/pipelines/pipeline.yaml opensearchproject/data-prepper:2.0.1 pipelines.yaml
docker run --name data-prepper -p 4900:4900 -v ./logstash.conf:/usr/share/data-prepper/pipelines/pipelines.yaml opensearchproject/data-prepper:2.0.1 pipelines.yaml
docker run --name data-prepper -p 4900:4900 -v ./logstash.conf:/usr/share/data-prepper/pipelines/pipelines.yaml opensearchproject/data-prepper:2.0.1 pipelines.conf
docker run --name data-prepper -p 4900:4900 -v ./logstash.conf:/usr/share/data-prepper/pipelines/pipelines.conf opensearchproject/data-prepper:2.0.1 pipelines.conf
docker run --name data-prepper -p 4900:4900 -v ./logstash.conf:/usr/share/data-prepper/pipelines.conf opensearchproject/data-prepper:2.0.1 pipelines.conf

I feel like I tried everything but I get error:

Error: runc: runc create failed: unable to start container process: exec: "pipelines.conf": executable file not found in $PATH: OCI runtime attempted to invoke a command that was not found

Please help or convert it for me…
Thanks

2

Okay so the last command should be the one to run.

docker run --name data-prepper -p 4900:4900 -v ./logstash.conf:/usr/share/data-prepper/pipelines.conf opensearchproject/data-prepper:2.0.1 pipelines.conf

It seems like the logstash.conf isn’t actually getting mapped in which could be a permissions issue (does your docker group have permissions to this file?) or a directory issue.

Have your tried providing a fully qualified path for logstash.conf? Something like /Users/<username>/home/logstash.conf (this is just a sample you would need the right path depending on your system.

3

Thanks for reply,
I made these changes.
chmod to 777, chown to 1000.1000
now my file looks like:
-rwxrwxrwx 1 1000 1000 1902 Jan 11 16:44 logstash.conf
I tried to run:
docker run --name data-prepper -p 4900:4900 -v /etc/opensearch/dataprepper/logstash.conf:/usr/share/data-prepper/pipelines.conf opensearchproject/data-prepper:2.0.1 pipelines.conf
But go to an error:
Error: runc: runc create failed: unable to start container process: exec: "pipelines.conf": executable file not found in $PATH: OCI runtime attempted to invoke a command that was not found

Looks like I miss some exec file in $PATH which is /usr/share/data-prepper/ I think. But I pulled latest version and didnt make any changes to build.

4

@dlv Could you take a look at this? This seems like there may be a race condition here with the logstash formatting on data-prepper.

5

@vnovotny98,

Thank you for your interest. Your Logstash configuration has conditionals in it. At the moment, the Logstash converter does not support Logstash’s conditional statements.

Also, Data Prepper does not have a TCP source plugin. Is this something you would be interested in having?

Feel free to create a GitHub issue if you are interested in either of these features in Data Prepper.

6

@dlv
thanks, I started issue TCP source plugin · Issue #2162 · opensearch-project/data-prepper · GitHub
I would love to use it, if you will handle this. I dont know another way how to collect app logs :slight_smile:

7

interesting topic… just wondering … what’s the difference between logstash and data-prepper, why would i want to use one above the other?

8

We used logstash for a long time. But for every audit we had a security problem that it was Debian or Ubuntu based and we need to have every app RedHat or UBI-based. I know it is harder to build your own logstash than data-prepper build, because Opensearch makes great how-to-build and give you dockerfiles and everything you need.

1 Like
9

Yeah I think as well data-prepper is providing better support for open telemetry traces which is its big selling point.