Creating a transform for ingestion statistics

General Feedback
1

I’m trying to create a aggregation of the number of inserted documents pr agent.hostname pr ten minutes, and have created the below transform job. The numbers that comes out of it looks kinda odd though, am I on the right track or am I overlooking something obvious?

Ideally I’d like for the transform job to create more detailed aggregation data for the fields event.datatype and host.hostname (the source indices are for apache weblogs), so that I can query those from grafana to show throughput over time without taxing the opensearch cluster that much by querying tons of documents directly, but my idea was to start by just getting the doccount and then expand from there.

image
image990×1013 61.6 KB

Am I on the right track here, or totally looking at it the wrong way?

PUT _plugins/_transform/count_docs_pr_hostname_10minutes
{
  "transform": {
    "enabled": true,
    "schedule": {
      "interval": {
        "period": 1,
        "unit": "Minutes",
        "start_time": 1708336546
      }
    },
    "description": "Count documents per agent.hostname per 10 minutes",
    "source_index": "someindex*",
    "target_index": "index-doccount",
    "page_size": 10000,
    "groups": [
      {
        "terms": {
          "source_field": "agent.hostname",
          "target_field": "agent_hostname"
        }
      },
      {
        "date_histogram": {
          "source_field": "@timestamp",
          "target_field": "timestamp_minute",
          "fixed_interval": "10m"
        }
      }
    ],
    "aggregations": {
      "documents_count": {
        "value_count": {
          "field": "agent.hostname"
        }
      }
    }
  }
}