Hi,
What is the best way to get json records from MQTT clients into an Index ?
I am running a Mosquitto MQTT server that I could subscribe to from some Opensearch tool that could put it into an index.
Hi, @olavk :
Currently Data Prepper only supports HTTP and Opentelemetry Protobuf. You might want to post this question in the opensearch client libraries to see if there is any solution.
I am running a Mosquitto MQTT server that I could subscribe to from some Opensearch tool that could put it into an index.
Could you elaborate more on your use case?
Hi thanks for the repsone,
I am collecting data from clients with MQTT to a MQTT broker called Mosquitto.
I was hoping to transfer these data to Opensearch indexes for analysis.
There is a mosquitto_sub command that will subcscibe and return on stdout available messages. These are in JSON-format. Perhaps there is a hook for a command for input somewhhere ?
As @qchea hinted at, Data Prepper doesn’t support MQTT as a remote source. Data Prepper does support STDIN and files as local sources through some common plugins: file and stdin. However, there are some limitations.
The stdin plugin will put the entire message in a field with the key “message”. This may not work well as your messages are JSON and may not result in correct data structure in OpenSearch. Data Prepper does not support a JSON processor at this time but the feature is in our backlog. The file plugin supports JSON messages however it reads the whole file and does not tail.
These are probably less than ideal for your use case. What gaps addressed or additional features would you need to meet your use case? I would encourage you to create an issue or indicate your need on the data prepper repo.