Obfuscating data in the ingestion pipeline

Data Prepper
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser): OpenSearch 2.11

Describe the issue:

I need to obfuscate PII data and I need to use the obfuscate_when parameter, because I want to obfuscate everything that is EMAIL except what comes with the app_name tag: email_mail. I must apply this filter in the ingestion pipeline.

Configuration:

version: "2"
logpipeline:
  source:
    http:
      path: "/v1/ec2"
  processor:
    - date:
        from_time_received: true
        destination: "@timestamp"
    - obfuscate:
        source: "log"
        patterns:
          - '%{EMAIL_ADDRESS}'
        action:
          mask:
            mask_character: "#"
            mask_character_length: 6
  sink:
    - opensearch:
        hosts: [ "htttp:xxxxxxxx" ]
        index: "ec2-%{yyyy.MM.dd}"
        max_retries: 20
        dlq:
          s3:
            bucket: "opensearch-rdc-zv-production-sink"
            key_path_prefix: "dlq-logs/%{yyyy}-%{MM}-%{dd}/"
            region: "sa-east-1"
            sts_role_arn: "xxxxxxxxxxxx"
        aws:
          region: "sa-east-1"
          sts_role_arn: "XXXXXXXXX"
    - s3:
        bucket: "xxxxxxx-production-sink"
        object_key:
          path_prefix: "ec2/%{yyyy}/%{MM}/%{dd}"
        compression: gzip
        threshold:
          maximum_size: 50mb
          event_collect_timeout: 60s
        codec:
          ndjson:
        aws:
          region: "sa-east-1"
          sts_role_arn: "XXXXX"

Relevant Logs or Screenshots:

2

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.