Using security analytics is hard mostly because of a lack of documentation

Plugins Security Analytics
21

Guys? :slight_smile:

22

Also posted a reminder to the Slack channel.

23

Hi @neographikal -

Thanks for reaching out in slack. Sometimes the different ways we gather information can get a bit muddled. Iā€™ve forwarded your message to our documentation manager to see if we can get some movement forward.

Letā€™s not forget that if you end up learning something that doesnā€™t appear in the documentation, we would love for you to submit your work as a PR.

Right now, I think the best thing to do is to file very specific issues in the documentation repo indicating what information is missing and/or what doesnā€™t make sense or is incorrect.

Iā€™ll keep tapping @hdhalter on the shoulder and see if thereā€™s a plan to make the feature better.

Nate

1 Like
24

Thanks for the nudge, @nateynate, and thanks for your persistence, @neographikal! We have been a bit underwater lately, but have some bandwidth to work on the updates to the security analytics documentation starting this week. The information provided in this post and the issue is very helpful in pointing out specific areas that need to be updated, and we really appreciate the time it took to write out the details. The tech writer, @Naarcha, will be in touch if there are any questions. Again, thanks for your patience and if there is anything else you need from the doc team, please reach out in Slack, GitHub, or in the forum.

25

Iā€™m not shy when it comes to contributing to the docs, Iā€™ve done it in the past. The problem is: I donā€™t know where to begin. I truly canā€™t make heads or tails from the log sources and mapping currently present in security analytics and Iā€™ve tried a fair amount of log forwarders. Gathering from the messages on Slack and here Iā€™m not alone.

The problem isnā€™t that it isnā€™t one thing that needs to be fixed. Itā€™s applies to every log source. Thanks for all the effort!

26

Thanks! Weā€™ll do our best to sort it out and make it understandable from a documentation standpoint.

27

Great thanks!

28

Any news? :slight_smile:

29

Hi @neographikal - the content is in progress. Would you like to review it once we get the PR up?

30

Yes maā€™am!

31

@neographikal: Hereā€™s a draft of the update log types documentation for your review: Add log types section to Security Analytics by Naarcha-AWS Ā· Pull Request #6235 Ā· opensearch-project/documentation-website Ā· GitHub.

I split each log type into its own page so it would be easier to find the mappings for each. Let me know your feedback. :slight_smile:

32

Thanks for all the work! Itā€™s kind of difficult because Iā€™m looking at the diffs and the raw markup, is there a rendered version somewhere I could review?

Maybe I missed it, but has the ingest of the data been documented? How do you get the data in opensearch so that the security analytics module can parse it and generate alerts?

33

Hi @neographikal, let me make a few more edits and send you the screenshots of each page.

As far as the ingestion piece, thatā€™s something I might need a little more assistant on. Would you be interested in meeting on a Zoom call so we can discuss it more?

34

Great thanks, sure no problem. You could send me an email at sander.vandegeijn@wur.nl so we can schedule a call.

35

Is there any news on this?

36

Afraid I donā€™t have any updates, tried to schedule an appointment to review de documentation, but it hasnā€™t been planned yet :slight_smile:

37

Hi all! Recently, we added a new log types section which includes the data mappings for most standard log types. You can find that here: Supported log types - OpenSearch Documentation

Still would love to find the time to meet so considering starting a consistent office hours. If thatā€™s something you would be interested in, let me know.

In the meantime, the best way to bring things to our attention is to post an issue in the documentation website repo.

38

Hi Naarcha,

Thank you for the effort, this gives some insight! But whatā€™s missing is an effective guide on how to get the data in opensearch in these formats, which log ingesters / pipelines do you need to use? I tried to plan a meeting with Nate, but that hasnā€™t worked out.

39

Having the supported log types and their ECS mapping is great, but as @neographikal pointed out, what is missing is the actual source of the log data.

I understand youprobably would not want to see implementations limited to single sources, but apparently the log ingestion world is VERY diverse and ECS is not always the same ECS.
Therefore it would help to see what the dev team tested against.

E.g. in the case of Windows logs - Winlogbeat is probably one of the most popular log senders:
Windows - OpenSearch Documentation uses winlog.event_data.Destination

There is no such field in Winlogbeat fields | Winlogbeat Reference [7.17] | Elastic (in fact it gets renamed to process.executable in the ingestion pipeline)

1 Like
40

Hello, would like to ask any update on documenting the data ingestion methodology for SA?

Iā€™m not sure weather I have to keep use elastic products like winlogbeat for Windows and AD, logstash with syslog plugin for Network, WAF, and logstash with netflow plugin for netflow?

Any other ideas for this kind of ingestion method? For example, by using data prepper? However, it seems do not support syslog or netflow. Or install opentelementry collector in Windows to collect Windows metric?

Any successful ingestion case here?