In OpenSearch 2.4, we are introducing the first set of Security Analytics capabilities as an experimental feature. This is an open source solution that will help users detect common threat patterns using over 2000 pre-packaged rules, while providing the flexibility to customize them to meet your requirements.Security analytics will include support for eight log sources including windows logs, AWS CloudTrail logs, S3 access logs etc. Users can create detectors using pre-packaged rule sets, to automatically generate security findings that help identify potential threats. Users can also create customized alerts from security findings and trigger workflows such as sending customized notifications on slack, email or a webhook.
To get started, check out the Github repo and documentation.The OpenSearch team is actively looking for your feedback and ideas to enhance the security analytics capabilities (including UI/UX workflows). Please include a description of your use cases to help understand the context of your feedback.
2 Likes
First off, this looks promising. Glad you are working on it.
First thing I get when I go to create a detector is the following error message:
[security_analytics_exception] Custom rule index doesnt exist. Please create custom rules first.
Also, when selecting the data source, you are presented with a long list of individual indexes which does not appear to be searchable (or at lest the list is not limited to items that match what you are typing). Also, you do not appear to be able to specify an index pattern which seem to make the whole thing entirely useless as you would need to create a new detector each day if using the standard per day index naming scheme.
If I go to create a detector for the Windows logs data type using a winlogbeat data source I get a required field mapping page which I have no idea how to fill out. The docs seem to indicate that this would be populated initially and one would only need to modify it if desired.
I am running into this same issue. @opoplawski - based on your follow up post “maybe” you have overcome some of these issues? Any steps you can share to proceed? For what it is worth my installation is an upgrade to 2.4 rather than a fresh install. Not sure if that matters.
FWIW - I haven’t made any progress on any of these issues.
@opoplawski , The mapping relates to the schema on the Sigma rule (for Windows logs data) and the actual log data. We plan to automatically populate the mapping if the log conforms to ECS schema (will be released in 2.6 (sooner if we can make it in 2.5). Looking at the log field names, they appear to be related to x509 certificate data on the windows server, however I was not able to find a corresponding Sigma rule for x509. Let me research and get back to you.
Appreciate your patience.
@opoplawski Thanks for the feedback, and we are prioritizing the support for index patterns in 2.5 release and hoping to be able to remove the experimental tag as well. For index mapping, currently we expect the users to map the log fields to conforms to ECS schema via the mapping interface screen. This allows Rule engine to identify the relevant fields even if they originally had a different name. As @praveensameneni mentioned above we are looking forward to automate this mapping in future, to reduce user setup interaction.
Guys, I’m very eager to experiment with this feature, the general direction is very promising although the current implementation is still a bit rough. What I’m missing from the docs is how to ingest i.e. the Windows event logs (which logcollector to be used, schema/mappings to be used, etc). The overall architecture is not clear from the docs.
Can someone put me in the right direction?
Hi @neographikal We are conforming to ECS schema for all the log types, including the Windows event logs. Even if you are sending data in a different schema format, the detector creation workflow will provide interface for mapping the fields correctly.
The overall architecture is present on the RFC at github : [RFC] Security Analytics initial offering in OpenSearch · Issue #2 · opensearch-project/security-analytics · GitHub
Cool thanks for the info! Would be nice to include more of these details in the documentation 
