# Has anyone ever created a detector based on a custom log type? **URL:** \ **Category:** Security Analytics\ **Created:** [April 3, 2024, 7:07am UTC](https://forum.opensearch.org/t/has-anyone-ever-created-a-detector-based-on-a-custom-log-type/18683 "2024-04-03T07:07:33Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![OpenAndreas](https://avatars.discourse-cdn.com/v4/letter/o/49beb7/32.png) [@OpenAndreas](https://forum.opensearch.org/u/OpenAndreas)\ **Post date:** [April 3, 2024, 7:07am UTC](https://forum.opensearch.org/t/has-anyone-ever-created-a-detector-based-on-a-custom-log-type/18683/1 "2024-04-03T07:07:33Z")
Hello, I am really wondering if it is possible to use custom log sources with detectors. I tried configure just the minimal setup but fail. Here is a draft of what I do: - Have logs sent to OpenSeach. - Those logs end up in an index ‘app-2024-04-03’ and so forth with an index pattern ‘app-\*’ - I have (and need) an index template to define fields and set a default ingest pipeline for processing. - I created a log type ‘custom-app’ (without further details regarding the log; maybe this is wrong, are there requirements?) - I created one rule for that log type. When I now try to create one detector using that log type, that rule and the required mapping of the field the rule works on, it fails with “Invalid field mapping” and the debug reveals that the detector logic tries to create an index template conflicting with mine. I see no way of working without my own index template since it defines fields and parses the logs through several ingest pipelines. So, the crucial detail is having an index template. Without this, there is no issue. I filed a bug report with further details of this behaviour: > > > \*\*What is the bug?\*\* > It is not possible to create a detector. Creation aborts w…ith "Invalid field mappings" due to template overlaps. > > \*\*How can one reproduce the bug?\*\* > Steps to reproduce the behavior: > 1. Go to Security Analytics \> Detectors \> Create new > 2. Fill in name > 3. Specify Data Source, use an index alias > 4. Choose a Log Type (I want a custom log type but also tried with builtin log types) > 5. Depending on the log type, field mappings are preconfigured or need to be explicitly configured, I tried both ways. > 6. Got to next page > 7. Remove Trigger (just to exclude issues with the trigger setup and remove complexity) > 8. Try to finish > 9. See error message "Invalid field mapping" and errors in log. > > \*\*What is the expected behavior?\*\* > No errors, alternatively more concise statements about the nature of the problem, ideally hints to correct the issue. > > \*\*What is your host/environment?\*\* > - OS: Ubuntu 22.04 > - Version 2.12 (upgraded from 2.11) > - Plugins Security Analytics Plugin > > \*\*Do you have any screenshots?\*\* > > !\[Screenshot 2024-03-28 091521\](https://github.com/opensearch-project/security-analytics/assets/124690157/88e266ec-ee01-4af9-8155-8d3b4294851e) > > > \*\*Do you have any additional context?\*\* > \` > "Caused by: java.lang.Exception: java.lang.IllegalStateException: Found conflicting templates: \[project-systems, project-kubernetes, project-app\]", > \` > I read the report of bug #830 which also deals with detector creation problems. Besides the different error message I do have the in the other case missing index ".opensearch-sap-custom-app-detectors-queries", "custom-app" is my log type of choice but I encounter the same problem when choosing the builtin Linux Log type instead. Just to exclude issues with custom log types. > > The conflicting index template have these index patterns: \`project-\*-systems-\*, project-\*-kubernetes-\* and project-\*-app-\*.\` > The legacy index template which the detector creation logic seems to be trying to create is: project-app\* > > As can be confirmed by the DEBUG output of org.opensearch.cluster.metadata: > > \`\[2024-03-28T07:24:11,495\]\[DEBUG\]\[o.o.c.m.MetadataIndexTemplateService\] \[siem.example.com\] legacy template .opensearch-sap-alias-mappings-index-template-project-app and composable template project-systems would overlap: \[project-app\*\] \<=\> \[project-\*-systems-\*\]\` > > The same message is repeated for the other two indexes. What I am now interested in is if anybody is able to succeed with a scenario like this? Just some proof-of-concept that it is possible would help. Best regards Andreas ---
**Author:** ![system](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/3/33547ea01a5b12dcca2958411d3edd97ae2ea8c1.png) [@system](https://forum.opensearch.org/u/system)\ **Post date:** [June 2, 2024, 7:07am UTC](https://forum.opensearch.org/t/has-anyone-ever-created-a-detector-based-on-a-custom-log-type/18683/2 "2024-06-02T07:07:47Z")
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.