You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
12 lines
3.3 KiB
12 lines
3.3 KiB
{
|
|
"uc_ref": "splk_queues_filling",
|
|
"uc_vendor": "Splunk",
|
|
"uc_description": "Monitors the Splunk indexing queues filling, triggers when a certain queue filling goes beyond accepted thresholds, which indicates potential infrastructure scaling issues",
|
|
"uc_category": "splunk_infrastructure",
|
|
"uc_earliest": "-5m",
|
|
"uc_latest": "now",
|
|
"uc_cron": "*/5 * * * *",
|
|
"uc_replacements": "tenant_id:mytenant,group:mygroup",
|
|
"uc_metrics": "queue.avg_pct_fill_queue,queue_max_pipelines,queue_max_size_kb,queue_current_size,queue_largest_size,queue_smallest_size,queue_max_queue_fill_percent,queue_pct_fill_queuequeue_latest_fill_percent",
|
|
"uc_search": "index=_internal source=*metrics.log sourcetype=splunkd host=* host!=c0m1*.splunkcloud.com host!=sh*.splunkcloud.com NOT HttpInputDataHandler NOT \"splunk-system-user\" component=Metrics group=queue \n| eval max=if(isnotnull(max_size_kb),max_size_kb,max_size) \n| eval curr=if(isnotnull(current_size_kb),current_size_kb,current_size) \n| eval fill_perc=round((curr/max)*100,2) \n| stats max(ingest_pipe) as max_pipelines max(max_size_kb) as max_size_kb max(current_size) as current_size max(largest_size) as largest_size max(smallest_size) as smallest_size max(fill_perc) as max_queue_fill_percent avg(fill_perc) AS avg_pct_fill_queue latest(fill_perc) as latest_queue_fill_percent by name host\n| eval avg_pct_fill_queue=round(avg_pct_fill_queue,2)\n\n```\nmax_pipelines will no exist if there is a single concurrent pipeline```\n| eval max_pipelines = if(isnum(max_pipelines), max_pipelines, 1)\n\n``` ensures every metric has a value ```\n| foreach max_size_kb, current_size, largest_size, smallest_size, max_queue_fill_percent, avg_pct_fill_queue, latest_queue_fill_percent [ eval <<FIELD>> = if(isnum('<<FIELD>>'), '<<FIELD>>', 0) ]\n\n``` set status and status_description ```\n| eval status=case(avg_pct_fill_queue>80 AND avg_pct_fill_queue<90, 3, avg_pct_fill_queue>=90, 2, 1=1, 1)\n| eval status_description=case(status=1, \"The queue is healthy, Avg pct filling: \" . round(avg_pct_fill_queue, 2), status=2, \"The queue is in alert with filling>90%, Avg pct filling: \" . round(avg_pct_fill_queue, 2), status=3, \"The queue is in warning with filling>80% and <90%, Avg pct filling: \" . round(avg_pct_fill_queue, 2))\n\n``` set group ```\n| eval group=\"infrastructure:splk_queue_filling\"\n\n``` set object, object_description and alias ```\n| eval object = host . \":\" . name, alias=object\n| eval object_description = \"Splunk Queue: \" . name . \" for host: \" . host . \", max_pipelines: \" . max_pipelines . \", max_size_kb: \" . max_size_kb \n\n``` set metrics ```\n| eval metrics = \"{'queue.avg_pct_fill_queue': \" . avg_pct_fill_queue . \", 'queue_max_pipelines': \" . max_pipelines .\", 'queue_max_size_kb': \" . max_size_kb .\", 'queue_current_size': \" . current_size . \", 'queue_largest_size': \" . largest_size . \", 'queue_smallest_size': \" . smallest_size . \", 'queue_max_queue_fill_percent': \" . max_queue_fill_percent . \", 'queue_pct_fill_queue': \" . avg_pct_fill_queue . \", 'queue_latest_fill_percent': \" . latest_queue_fill_percent . \"}\"\n\n``` set default metric ```\n| eval default_metric=\"queue.avg_pct_fill_queue\"\n\n``` queues can be inactive and not generating any results, do not alert if so ```\n| eval max_sec_inactive=0"
|
|
} |