Versions (relevant - OpenSearch/Dashboard/Server OS/Browser): OpenSearch 2.11
Describe the issue:
I need to obfuscate PII data and I need to use the obfuscate_when parameter, because I want to obfuscate everything that is EMAIL except what comes with the app_name tag: email_mail. I must apply this filter in the ingestion pipeline.
Configuration:
version: "2"
logpipeline:
source:
http:
path: "/v1/ec2"
processor:
- date:
from_time_received: true
destination: "@timestamp"
- obfuscate:
source: "log"
patterns:
- '%{EMAIL_ADDRESS}'
action:
mask:
mask_character: "#"
mask_character_length: 6
sink:
- opensearch:
hosts: [ "htttp:xxxxxxxx" ]
index: "ec2-%{yyyy.MM.dd}"
max_retries: 20
dlq:
s3:
bucket: "opensearch-rdc-zv-production-sink"
key_path_prefix: "dlq-logs/%{yyyy}-%{MM}-%{dd}/"
region: "sa-east-1"
sts_role_arn: "xxxxxxxxxxxx"
aws:
region: "sa-east-1"
sts_role_arn: "XXXXXXXXX"
- s3:
bucket: "xxxxxxx-production-sink"
object_key:
path_prefix: "ec2/%{yyyy}/%{MM}/%{dd}"
compression: gzip
threshold:
maximum_size: 50mb
event_collect_timeout: 60s
codec:
ndjson:
aws:
region: "sa-east-1"
sts_role_arn: "XXXXX"
Relevant Logs or Screenshots: