Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OS:
NAME=“Oracle Linux Server”
VERSION=“8.8”
“version”: {
“distribution”: “opensearch”,
“number”: “2.9.0”,
“lucene_version”: “9.7.0”,
“minimum_wire_compatibility_version”: “7.10.0”,
“minimum_index_compatibility_version”: “7.0.0”
}
Describe the issue:
I have defined a transform job to run every 10 minutes. However, the initial execution is taking forever. Could you please support to diagnose and speed up my job.
What I am intending to do is to calculate the number of transactions per second. My index is about 15GB of data. I am filtering on a term then aggregate and calculate the doc count per second.
Configuration:
Transform job definition:
{
“_id”: “aida_tps”,
“_version”: 4,
“_seq_no”: 166173,
“_primary_term”: 1,
“transform”: {
“transform_id”: “aida_tps”,
“schema_version”: 18,
“schedule”: {
“interval”: {
“start_time”: 1703484768,
“period”: 10,
“unit”: “Minutes”
}
},
“metadata_id”: “iENEPT8gWitGerPfeuwHpA”,
“updated_at”: 1703484768879,
“enabled”: true,
“enabled_at”: 1703481783298,
“description”: “AIDA TPS index”,
“source_index”: “ptm_counters”,
“data_selection_query”: {
“bool”: {
“filter”: [
{
“term”: {
“Stream”: {
“value”: “AIDA”,
“boost”: 1
}
}
}
],
“adjust_pure_negative”: true,
“boost”: 1
}
},
“target_index”: “aida_tps”,
“page_size”: 1000,
“groups”: [
{
“date_histogram”: {
“fixed_interval”: “1s”,
“source_field”: “Date”,
“target_field”: “Date”,
“timezone”: “Asia/Kabul”,
“format”: null
}
}
],
“aggregations”: {},
“continuous”: true
}
}
Relevant Logs or Screenshots:
Transform job explain logs:
{
“aida_tps”: {
“metadata_id”: “iENEPT8gWitGerPfeuwHpA”,
“transform_metadata”: {
“transform_id”: “aida_tps”,
“after_key”: {
“Date”: 1702900165000
},
“last_updated_at”: 1703486004106,
“status”: “started”,
“failure_reason”: null,
“stats”: {
“pages_processed”: 3623,
“documents_processed”: 145,
“documents_indexed”: 64,
“index_time_in_millis”: 1395,
“search_time_in_millis”: 3277420
},
“continuous_stats”: {
“documents_behind”: {
“ptm_counters”: 146212395
}
}
}
}
}