Data prepper - Workers did not terminate in PT30S, forcing termination of sink workers

Data Prepper
1

Getting error:

2024-11-28T17:06:52,041 [log-pipeline-sink-worker-2-thread-1] WARN  org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [log-pipeline] - Workers did not terminate in PT30S, forcing termination of sink workers.

2024-11-28T17:06:52,042 [log-pipeline-sink-worker-2-thread-1] INFO  org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [log-pipeline] - Pipeline fully shutdown.

2024-11-28T17:06:52,048 [log-pipeline-sink-worker-2-thread-1] INFO  org.opensearch.dataprepper.pipeline.server.DataPrepperServer - Data Prepper server stopped

However the pod running data-prepper container is intact and not terminated.

Data Prepper ConfigMap:

---
apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app: data-prepper-app-logs
  name: data-prepper-config-app-logs
data:
  pipeline.yaml: |
    log-pipeline:
      workers: 2 # Number of workers
      delay: 200 # in milliseconds, The number of milliseconds that workers wait between buffer read attempts. Default is 3000.
      buffer:
        bounded_blocking:
          buffer_size: 200000
          batch_size: 1000
      source:
        http:
          ssl: false
      processor:
        - grok:
            match:
              log: [ '${GROK_EXPRESSION_KEY_VALUE}' ]
        - date:
            from_time_received: true
            destination: "@timestamp"
      sink:
        - opensearch: 
            hosts: [ "${OPENSEARCH_ENDPOINT}" ]
            insecure: false
            username: "${ES_USERNAME}"
            password: "${ES_PASSWORD}"
            index: "${OPENSEARCH_INDEX_NAME}"
        - stdout:
  config.yaml: |
    ssl: false

Data Prepper Deployment:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: data-prepper-app-logs
  namespace: ${EKS_NAMESPACE}
  labels:
    app: data-prepper-app-logs
spec:
  replicas: 1
  selector:
    matchLabels:
      app: data-prepper-app-logs
  template:
    metadata:
      labels:
        app: data-prepper-app-logs
    spec:
      nodeSelector:
        group: app
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app: data-prepper-app-logs
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: kubernetes.io/arch
                    operator: In
                    values:
                      - amd64
                      - arm64
      containers:
        - name: data-prepper-app-logs
          image: opensearchproject/data-prepper:2
          args:
            - /usr/share/data-prepper/bin/data-prepper
            - /etc/data-prepper/pipeline.yaml
            - /etc/data-prepper/config.yaml
          ports:
            - containerPort: 2021
          resources:
            limits:
              cpu: 1000m
              memory: 2048Mi
            requests:
              cpu: 1000m
              memory: 2048Mi
          volumeMounts:
            - mountPath: /etc/data-prepper
              name: prepper-configmap-claim0
      volumes:
        - name: prepper-configmap-claim0
          configMap:
            name: data-prepper-config-app-logs

Appreciate any leads, thank you!