Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
AWS OpenSearch - not sure of version, I am checking on that
FluentBit 1.8.x
Describe the issue:
JSON log data does not get parsed/rendered correctly in OpenSearch UI, I see it as a single text field, rather than the individual internal JSON fields.
Configuration:
Multiple components written in Go, Python, Elixer, Ruby deployed in Kubernetes logging to STDOUT/STDERR. Some components log as text, some log as JSON. FluentBit forwarding all logs. All log data for all components goes to an index field called “log” which has Type “string” in the Index Pattern.
Full details here: fluent bit - How to configure FluentBit & OpenSearch so that json and non-json logs are handled correctly - Stack Overflow
Relevant Logs or Screenshots:
See Stackoverflow link above
I am happy to feedback about how to structure my end-to-end logging from FluentBit to OpenSearch or even just feedback on the following specific question.
If I have a field in an OpenSearch index called “log” and the field current is configured with type “text” and when I look at that field in OpenSearch UI I see for example:
Notice a few things here:
1 - Both log lines start with:
- timestamp
- source - stdout/stderr
- F - I don’t know why
2 - First log line then has a JSON-formatted string
3 - Second log line then has a text formatted string
How should I configure the properties of the ‘log’ field in my index so the JSON is parsed/rendered correctly, or is this not possible to “fix/handle” in OpenSearch, do I need to go back upstream to FluentBit and make a change there?
Looking over what you posted it seems you are trying to put a lot of different types of logs into the same field. The issue with this approach is if they are in the same index that field can only be one type which in this case is string.
2 paths going forward as far as I can tell.
-
(What I am recommending) Separate your logs into different indexes based off of their type. You can do this with patterns and matching in fluentbit. This way your Kubernetes logs can be parsed and you can leverage the right data type for all of the subfields. Fluentbit has a really great guide for Kubernetes logging here: Kubernetes - Fluent Bit: Official Manual
-
Parsing json into different fields. You mentioned this in your stackoverflow post and this is not a bad idea. I would strongly encourage you to not go with this method. While it may work now it becomes much more difficult to maintain in the long run.
Thanks for your input @dtaivpp.
When I read through the link you supplied I came across the section on the CRI handling. I notice that the regex used by the CRI parser will handle the structure of logs that I am seeing, so I think I must be using the CRI in my K8 env. I tested the CRI regex:
^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
using rubular and it does parse the logs I am seeing correctly.
Here is an example output that I am seeing in OpenSearch Dashboard under the log field with my current setup (before adding CRI Parser):
2023-01-09T23:41:56.279212506Z stdout F {"level":"WARN","ts":1673307716278.9448,"caller":"internal/internal_task_pollers.go:348","message":"Failed to process workflow task.","Namespace":"ai-platform-dev.59ee7","TaskQueue":"WORKFLOW_QUEUE","WorkerID":"1@workflow-worker-ai-workflow-worker-6c445f59f7-pgn6v@","WorkflowType":"NotesProWorkflow","WorkflowID":"workflow_1169649613530771459_1664751006481316721","RunID":"1ae58130-62d6-4f6a-a6db-8789be13d567","Attempt":12530,"Error":"lookup failed for scheduledEventID to activityID: scheduleEventID: 36, activityID: 36"}
I will confirm that CRI is used in the K8 env, then try the CRI parser instead of my own custom regex that I was considering.
