<form version="1.1">
  <label>Dashboard - Splunk forwarder output tuning</label>
  <description>Splunk forwarder to indexer output tuning</description>
  <fieldset submitButton="false">
    <input type="time" token="time">
      <label>time</label>
      <default>
        <earliest>-60m@m</earliest>
        <latest>now</latest>
      </default>
    </input>
    <input type="text" token="host">
      <label>host</label>
      <default>`heavyforwarderhosts`</default>
    </input>
    <input type="dropdown" token="output_group">
      <label>Output Group</label>
      <fieldForLabel>output_name</fieldForLabel>
      <fieldForValue>output_name</fieldForValue>
      <search>
        <query>index=_internal $host$ sourcetype=splunkd `splunkadmins_metrics_source` TERM(group=tcpout_connections)
| rex field=name "(?P&lt;output_name&gt;[^:]+)"
| stats count by output_name
| fields output_name</query>
        <earliest>-60m@m</earliest>
        <latest>now</latest>
      </search>
    </input>
    <input type="dropdown" token="split_by">
      <label>Split by host?</label>
      <choice value="host">Yes</choice>
      <choice value="&quot;&quot;">No</choice>
    </input>
  </fieldset>
  <row>
    <panel>
      <title>Data output per-second</title>
      <chart>
        <search>
          <query>index=_internal $host$ sourcetype=splunkd `splunkadmins_metrics_source` TERM(group=tcpout_connections) name=$output_group$*
| rex field=name "(?&lt;output_name&gt;[^:]+)"
| search output_name=$output_group$
| fillnull ingest_pipe
| eval combined = output_name . "_" . ingest_pipe
| bin _time span=1m
| stats sum(kb) AS totalkb by combined, host, _time
| eval totalkb=totalkb/60
| eval combined = $split_by$ . combined
| timechart limit=99 avg(totalkb) AS avgkb, perc95(totalkb) AS perc95kb, min(totalkb) AS minkb by combined</query>
          <earliest>$time.earliest$</earliest>
          <latest>$time.latest$</latest>
        </search>
        <option name="charting.chart">line</option>
        <option name="charting.drilldown">none</option>
        <option name="refresh.display">progressbar</option>
      </chart>
    </panel>
  </row>
  <row>
    <panel>
      <title>Destination count</title>
      <table>
        <search>
          <query>index=_internal $host$ sourcetype=splunkd `splunkadmins_metrics_source` group=tcpout_connections name=$output_group$*
| rex field=name "(?&lt;output_name&gt;[^:]+)"
| search output_name=$output_group$
| bin _time span=5m
| stats dc(destIp) AS destination_count by output_name, host, _time
| stats min(destination_count) AS min_destination_count, avg(destination_count) AS avg_destination_count by output_name</query>
          <earliest>-24h@h</earliest>
          <latest>now</latest>
        </search>
        <option name="drilldown">none</option>
        <option name="refresh.display">progressbar</option>
      </table>
    </panel>
  </row>
  <row>
    <panel>
      <title>Data output std deviation</title>
      <chart>
        <search>
          <query>```Credit to Brett Adams``` index=_internal $host$ sourcetype=splunkd `splunkadmins_metrics_source` component=Metrics TERM(group=tcpout_connections) name=$output_group$*
| rex field=name "(?P&lt;destination&gt;[^:]+)"
| search destination=$output_group$*
| timechart span=1m sum(kb) by destIp limit=50
| fillnull value=0
| untable _time destIp kb 
| stats avg(kb) as avg stdev(kb) as stdev by _time
| eval dev_perc = stdev/avg*100
| table _time dev_perc</query>
          <earliest>$time.earliest$</earliest>
          <latest>$time.latest$</latest>
        </search>
        <option name="charting.chart">line</option>
        <option name="charting.drilldown">none</option>
        <option name="refresh.display">progressbar</option>
      </chart>
    </panel>
  </row>
  <row>
    <panel>
      <title>Dashboard info</title>
      <html>
        <body>
          <p>Purpose of destination count table? metrics.log only records the tcpout data *if* the connection is open at the time the metrics.log writes, so the count is to sanity-check that the numbers of connections matches the number of forwarders on the backend (this will happen with the below outputs.conf settings combined with regular data flow)</p>
          <br/>
          <p><a href="https://docs.splunk.com/Documentation/SVA/current/Architectures/Intermediaterouting#Asynchronous_load_balancing"> Asynchronous load balancing (docs.splunk.com) </a></p>
<p><a href="https://www.linkedin.com/pulse/splunk-asynchronous-forwarding-lightning-fast-data-ingestor-rawat"> Splunk Asynchronous Forwarding (Lightning-fast data ingestor)</a></p>
    <p>Purpose of the data output per-second timechart? The current goal is to get close to switching indexers every second for an output group (per-pipeline), note that this will result in more open connections to indexers so only really works if this is deployed to a moderate number of intermediate forwarders (HF's or similar). Note that you want to do this with autoLBVolume, if you lower autoLBFrequency to a very short time period you may result in un-even data balance due to switching frequently when forwarding smaller volumes of data. In my testing so far it would appear that aiming above the average kb/s for the autoLBVolume appears to work well, going too low doesn't work well in my testing so far</p>
    <p>Please read the linked article for information on these settings, note that when using async forwarding the open file descriptor usage is higher than without async forwarding as the connections are held open by forwarders. So this works great on an intermediate forwarding tier, this may not work so well with a very large number of forwarders</p>
    <p>Also note that the maxQueueSize should not be below 10MB (10MB minimium size)</p>
    <p>If you are using an AWS NLB, you may wish to refer to this newer post <a href="https://www.linkedin.com/posts/harendra-rawat-b10b41_asynchronous-forwarding-with-nlb-activity-7112204069363933185-SYRv"> Asynchronous forwarding with NLB</a></p>
    <p>Finally while this also works on UF's, there are some reasons why you may want to consider HF's if you are running an intermediate tier, answers post <a href="https://community.splunk.com/t5/Getting-Data-In/Wrongly-merged-Events-permanently-blocked-tcpout-queue-with/m-p/508743">Wrongly merged Events/permanently blocked tcpout queue with Intermediate Universal Forwarder</a></p>
          <br/><p>Finally you may want to refer to <a href="https://community.splunk.com/t5/Knowledge-Management/Slow-indexer-receiver-detection-capability/m-p/683768">Slow indexer/receiver detection capability</a></p>
    <p>What config  is used to achieve the above?</p>
    <p>outputs.conf file based on 1MB/s
    </p><p><code>maxQueueSize = 10MB</code>
          </p>
    <p>
            <code>#autoLBVolume is set below 1/5 of the maxQueueSize due to changes post 7.3.6 which will hopefully be documented in the very near future, minimum 10MB queue</code>
          </p>
    <p>
            <code>autoLBVolume = 1024000</code>
          </p>
    <p>
            <code>autoLBFrequency = 10</code>
          </p>
    <p>
            <code>connectionTTL = 300</code>
          </p>
        </body>
      </html>
    </panel>
  </row>
</form>
