Would you like to present? Tag @kris@dtaivpp@nateynate and we’ll work to get you added to the agenda!
Feel free to comment on this agenda before the meeting if you want to add an item or have a question.
After the meeting, we will post the chat log and any meeting notes. We welcome you to keep the conversation going here on the forum.
======== By joining the OpenSearch Community Meeting, you grant OpenSearch, and our affiliates the right to record, film, photograph, and capture your voice and image during the OpenSearch Community Meeting (the “Recordings”). You grant to us an irrevocable, nonexclusive, perpetual, worldwide, royalty-free right and license to use, reproduce, modify, distribute, and translate, for any purpose, all or any part of the Recordings and Your Materials. For example, we may distribute Recordings or snippets of Recordings via our social media outlets.
07:03:11 From David Tippett to Everyone:
Morning everyone!
07:03:20 From D Swager to Everyone:
good morning!
07:03:34 From Eli Fisher to Everyone:
Hi everyone!
07:03:46 From Kris Freedain to Everyone:
Good morning!
07:04:51 From Kris Freedain to Everyone:
Community Meeting survey: Community Meeting Survey
07:05:44 From Kris Freedain to Everyone:
OpenSearchCon: OpenSearchCon 2022 · OpenSearch
07:06:35 From Kris Freedain to Everyone:
Ingestion survey: Survey: OpenSearch Ingestion Survey
07:16:42 From Lukáš Vlček (@Aiven.io) to Everyone:
Very nice!
07:17:41 From Joshua Bright to Everyone:
Woot!
07:17:56 From Anurag Gupta to Everyone:
(Fluent Bit demo space) Instruqt
07:17:57 From Rahul Chordiya to Everyone:
thanks Anurag
07:18:10 From Kris Freedain to Everyone:
Open Observability Day: Open Observability Day North America | Linux Foundation Events
07:22:35 From Kris Freedain to Everyone:
thanks again Anurag - fantastic demo
07:23:32 From Alejandro Guida to Everyone:
Anurag, about for example Fluentbit supporting particular parsings of data of different sources, like ECS, CEF, etc out of the box. Is that out of the scope for Fluentbit, or maybe we can see that in the future?
07:24:38 From Nate Boot to Everyone:
Most of the AWS Elemental Media services can write their logs to S3.
07:25:17 From Anurag Gupta to Everyone:
Hi @Alejandro, On the roadmap! We have parsings of popular applications today that don’t have any particular schema like nginx, apache2, syslog, and others.
07:26:15 From Anurag Gupta to Everyone:
We are looking at ECS in context of OpenTelemetry and have some initial implementations there as well
07:26:42 From Alejandro Guida to Everyone:
Perfect, thanks Anurag
07:27:10 From Anurag Gupta to Everyone:
For CEF would be great to understand use cases, I’ve seen folks in the community write a few regex parsers for it (E.g. Cisco ASA Firewalls)
07:30:29 From Nate Boot to Everyone:
Cool!
07:32:06 From Ryan Paras to Everyone:
A question for after the presentations - Both fluentbit and data prepper (and other projects) lack the ability to handle individual messages that fail to push during bulk pushes - ie mapping failures, security failures, etc… this is something that logstash configs sometimes handled via a “DLQ” usage pattern. Are either projects looking at this from a client percpective? would it make sense for opensearch to develop a method, so that during ingest, there are options to divert messages with errors to different indexes?
07:37:46 From Nate Boot to Everyone:
Perfect
07:39:41 From Kris Freedain to Everyone: https://github.com/opensearch-project/documentation-website/issues/810
07:39:59 From Kris Freedain to Everyone:
we’d appreciate your feedback on this
07:40:09 From David Venable to Everyone: @Ryan Paras, If I understand correctly, you are asking about having a DLQ type of feature. From a Data Prepper perspective, we are looking into having a mechanism for saving failed writes to some form of DLQ to retry later. It sounds like you are perhaps suggesting having a “failure" index in OpenSearch? Where Data Prepper sends failed messages to a simpler index perhaps with simpler mappings and security?
07:40:16 From Nate Boot to Everyone:
Kris++ for droppin’ them links.
07:40:58 From Lukáš Vlček (@Aiven.io) to Everyone:
This is cool. Thanks for building the docs team.
07:42:32 From David Venable to Everyone:
Links for Data Prepper
Documentation for S3 source: Configuration reference - OpenSearch documentation
README documentation: https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/s3-source/README.md
Forum Category for Data Prepper: Data Prepper - OpenSearch
Data Prepper roadmap: https://github.com/opensearch-project/data-prepper/projects/1
07:48:41 From Charlotte Henkle to Everyone:
Or feel free to open an issue in the OpenSerach repo
07:48:54 From Kris Freedain to Everyone:
07:52:50 From Anurag Gupta to Everyone:
One note from Fluent Bit side, we currently have a debug log for 400 errors to Splunk that we’ve been looking to make more generic for all outputs
07:53:21 From Anurag Gupta to Everyone:
while not the exact feature, it could at least help with re-routing 400 errors to another output
07:53:34 From Ryan Paras to Everyone: @anurag - would be happy to test
07:53:58 From Kris Freedain to Everyone:
thank you everyone!
07:54:01 From Lukáš Vlček (@Aiven.io) to Everyone:
Thanks, have a good summer.
07:54:23 From William Beckler, OpenSearch (he/him) to Everyone:
Thank you everyone!!
07:54:28 From Anurag Gupta to Everyone:
Thanks all!
07:54:29 From Ryan Paras to Everyone:
thanks
07:54:33 From Cole to Everyone:
Thanks!
07:54:39 From Terry Q to Everyone:
Thanks
07:54:39 From William Beckler, OpenSearch (he/him) to Everyone:
Thank you Nate for hosting!!
