<form stylesheet="howto.css,ui_simple.css" script="modal.js" hideEdit="True" isVisible="true" version="1.1" theme="dark">
	<label>HOWTO TOP: Generate stats and charts with Splunk Search Processing Language (SPL)</label>
	<description>This collection of request samples will help you generating your own reports, dashboards and alerts for Nmon Performance data using the native Search Processing Language</description>

	<row>
		<panel>

			<html>

				<div class="imgheader custom-modal">
					<img src="../../static/app/metricator-for-nmon/icons/dark_theme/process.png" alt="Process"/>
					<h2><a data-modal-name="Help_modal">Process Resources Usage Statistics (click here for more information)</a></h2>
				</div>

                <!-- Modal -->
                <div class="modal custom-modal-60 fade" id="Help_modal" tabindex="-1" role="dialog" aria-labelledby="Help_modal_Label">
                    <div class="modal-dialog" role="document">
                        <div class="modal-content">
                            <div class="modal-header">
                                <button type="button" class="close" data-dismiss="modal" aria-label="Close"><span aria-hidden="true"></span></button>
                                <div style="text-align: left;">
                                    <h4 class="modal-title" id="Help_modal_Label">Integrated Navigation:</h4>
                                </div>
                            </div>
                            <div class="modal-body">

                                <div style="text-align: center;">
                                    <img src="../../static/app/metricator-for-nmon/icons/dark_theme/process.png" alt="CPU"/>
                                    <h1>Processes statistics (metric_name=os.unix.nmon.processes.top.*)
                                    </h1>
                                </div>

                                <div>
                                    The TOP monitor captures processes statistics for system resources metrics such as CPU, memory and more.
                                    <br />
                                    This section is available in any supported operating systems, but some metrics are specifics for some systems.
                                    <br />
                                    The <b>UARG monitor</b> is associated with the TOP monitor as it will capture the full list of processes arguments.
                                    <br />
                                    <br />
                                    Processes statistics are generated at the PID level for a given "Command" invocation, as such, calculation will aggregate statistics of all PIDs for that given Command.
                                </div>

                                <div style="text-align: left;">

                                    <h1>raw metrics and dimensions:</h1>

                                    <div class="list">
                                        <lu>
                                            <li><b>dimension_Command:</b> The name of the global command that generated the process</li>
                                            <li><b>dimension_PID:</b> The PID number of the process</li>
                                            <li><b>pct_CPU:</b> (%) average amount of CPU used by this process</li>
                                            <li><b>pct_Usr:</b> (%) average amount of user-mode CPU used by this process</li>
                                            <li><b>pct_Sys:</b> (%) average amount of kernel-mode CPU used by this process</li>
                                        </lu>
                                    </div>

                                    <div class="list">
                                        <lu>
                                            <li><b>Size:</b> The size of the pages in kilobytes.</li>
                                            <li><b>ResSet:</b> The sum of real-memory data used by the process</li>
                                            <li><b>ResText:</b> The real-memory text size of the process.</li>
                                            <li><b>ResData:</b> The real-memory data size of the process.</li>
                                            <br />
                                        </lu>
                                    </div>

                                </div>

                                <div style="text-align:left;">
                                    <h1>Related links:</h1>
                                </div>

                                <div style="text-align: center;" class="cat_title">

                                    <a target="_blank" href="search?q=%7C%20mcatalog%20values(metric_name)%20as%20metrics%20where%20%60nmon_metrics_index%60%20metric_name%3D&quot;os.unix.nmon.processes.top.*&quot;%20host%3D&quot;*&quot;" class="tryitbtnxl">mcatalog »
                                    </a>

                                    <a target="_blank" href="search?q=%7C%20mstats%20avg(_value)%20as%20value%20where%20%60nmon_metrics_index%60%20metric_name%3Dos.unix.nmon.processes.top.*%20host%3D*%20dimension_Command%3D*%20dimension_PID%3D*%20by%20metric_name%2C%20host%20span%3D1m" class="tryitbtnxl">mstats raw query»
                                    </a>

                                    <a target="_blank" href="search?q=%7C%20mstats%20sum(_value)%20as%20pct_CPU%20where%20%60nmon_metrics_index%60%20metric_name%3Dos.unix.nmon.processes.top.pct_CPU%20host%3D*%20dimension_Command%3D*%20by%20dimension_Command%20span%3D1m%0A%7C%20eval%20usage_per_core%3D(pct_CPU%2F100)%0A%7C%20timechart%20%60nmon_span%60%20useother%3Df%20limit%3D50%20max(usage_per_core)%20as%20usage_per_core%20by%20dimension_Command" class="tryitbtnxl">mstats pre-built query »
                                    </a>

                                    <a target="_blank" href="Howto_TOP_spl" class="tryitbtnxl">HOWTO Interface »
                                    </a>

                                    <a target="_blank" href="UI_data_dictionary?&amp;form.itemfilter=PROCESSES%20STATISTICS" class="tryitbtnxl">Processes Data Dictionary »
                                    </a>

                                    <a target="_blank" href="Dashboard_Bubblechart_top_processes?form.osfilter=Linux" class="tryitbtnxl">Bubblechart dashboard »
                                    </a>

                                </div>

                                <div style="text-align:left;">
                                    <h1>Related metrics:</h1>
                                </div>

                                <div style="text-align: center;" class="cat_title">

                                    <a target="_blank" href="search?q=%7C%20mstats%20avg(_value)%20as%20value%20where%20%60nmon_metrics_index%60%20metric_name%3Dos.unix.nmon.cpu.cpu_all.*%20host%3D*%20by%20metric_name%2C%20host%20span%3D1m" class="tryitbtnxl">os.unix.nmon.cpu.cpu_all.* »
                                    </a>

                                    <a target="_blank" href="search?q=%7C%20mstats%20avg(_value)%20as%20value%20where%20%60nmon_metrics_index%60%20metric_name%3Dos.unix.nmon.cpu.cpunn.*%20host%3D*%20by%20metric_name%2C%20host%20span%3D1m" class="tryitbtnxl">os.unix.nmon.cpu.cpunn.* »
                                    </a>

                                    <a target="_blank" href="search?q=search%20eventtype%3Dnmon%3Aevents%20type%3DUARG" class="tryitbtnxl">UARG events »
                                    </a>

                                </div>

                            </div>

                            <div class="modal-footer">
                                <button type="button" class="btn btn-default" data-dismiss="modal">Close</button>
                            </div>

                        </div>
                    </div>
                </div>

			</html>

		</panel>

		<panel>
			<title>Optionally preset host, Command and press Enter:</title>

			<input type="text" token="host" searchWhenChanged="true">
				<label>Host:</label>
				<default>*</default>
			</input>

			<input type="text" token="Command" searchWhenChanged="true">
				<label>Command:</label>
				<default>*</default>
			</input>

		</panel>

	</row>

	<row>
		<panel>
			<title>Time oriented and statistics oriented charts:</title>

			<html>

				<div>
					<h2>Time Oriented Charts:</h2>
					<br />
				</div>

				<div style="text-align: left;">
					<br />
					<img src="../../static/app/metricator-for-nmon/various/mini/minichart.png" alt="minichart"/>
					<br />
					<br />
				</div>

				<div>
					<h2>By Server / Program Analysis: Core CPU usage of main processes</h2>
					<pre>| mstats max(_value) as value where `nmon_metrics_index` metric_name=os.unix.nmon.processes.top.pct_CPU host=$host$ (dimension_Command=$Command$) (dimension_PID=*) by host dimension_Command dimension_PID span=1m
| stats sum(value) as pct_CPU by _time, host, dimension_Command
| eval usage_per_core=(pct_CPU/100)
| where (usage_per_core>0)
| eval key=host . ":" . dimension_Command
| timechart `nmon_span` limit=50 useother=f max(usage_per_core) as cpu_usage_per_core by key</pre>
				</div>

				<div>
					<div class="cat_title">
						<a target="_blank" href="search?q=%7C%20mstats%20max(_value)%20as%20value%20where%20%60nmon_metrics_index%60%20metric_name%3Dos.unix.nmon.processes.top.pct_CPU%20host%3D$host$%20(dimension_Command%3D$Command$)%20(dimension_PID%3D*)%20by%20host%20dimension_Command%20dimension_PID%20span%3D1m%0A%7C%20stats%20sum(value)%20as%20pct_CPU%20by%20_time%2C%20host%2C%20dimension_Command%0A%7C%20eval%20usage_per_core%3D(pct_CPU%2F100)%0A%7C%20where%20(usage_per_core>0)%0A%7C%20eval%20key%3Dhost%20.%20&quot;%3A&quot;%20.%20dimension_Command%0A%7C%20timechart%20%60nmon_span%60%20limit%3D50%20useother%3Df%20max(usage_per_core)%20as%20cpu_usage_per_core%20by%20key&amp;display.page.search.tab=visualizations&amp;display.general.type=visualizations&amp;display.visualizations.charting.axisY.minimumNumber=0&amp;display.visualizations.charting.chart=area&amp;display.visualizations.charting.axisY.maximumNumber=&amp;display.visualizations.charting.legend.placement=top&amp;earliest=-24h&amp;latest=now&amp;display.visualizations.charting.chart.stackMode=stacked&amp;display.visualizations.charting.axisTitleY.text=Logical%20Core%20CPU%20usage" class="tryitbtn-blue">Open in search »
						</a>
					</div>
					<br />
				</div>

				<div>
					<h2>By Server / Program Analysis: Core memory usage of main processes</h2>
					<pre>| mstats max(_value) as value where `nmon_metrics_index` metric_name=os.unix.nmon.processes.top.ResSet OR metric_name=os.unix.nmon.processes.top.ResText OR metric_name=os.unix.nmon.processes.top.ResData host=$host$ dimension_Command=$Command$ by OStype, host, metric_name, dimension_Command, dimension_PID span=1m
| stats sum(value) as value by _time, OStype, host, metric_name, dimension_Command
| `extract_metrics`
| eval {metric}=value
| stats values(ResSet) as ResSet, values(ResText) as ResText, values(ResData) as ResData by _time, OStype, host, dimension_Command
| fillnull value=0 ResSet ResText ResData
| eval mem_usage_mb=case(match(OStype, "Linux"), ((ResSet)/1024), match(OStype, "AIX"), ((ResData+ResText)/1024), match(OStype, "Solaris"), (ResSize/1024) )
| timechart `nmon_span` limit=10 useother=f max(mem_usage_mb) AS mem_usage_mb by dimension_Command</pre>
				</div>

				<div>
					<div class="cat_title">
						<a target="_blank" href="search?q=%7C%20mstats%20max(_value)%20as%20value%20where%20%60nmon_metrics_index%60%20metric_name%3Dos.unix.nmon.processes.top.ResSet%20OR%20metric_name%3Dos.unix.nmon.processes.top.ResText%20OR%20metric_name%3Dos.unix.nmon.processes.top.ResData%20host%3D$host$%20dimension_Command%3D$Command$%20by%20OStype%2C%20host%2C%20metric_name%2C%20dimension_Command%2C%20dimension_PID%20span%3D1m%0A%7C%20stats%20sum(value)%20as%20value%20by%20_time%2C%20OStype%2C%20host%2C%20metric_name%2C%20dimension_Command%0A%7C%20%60extract_metrics%60%0A%7C%20eval%20%7Bmetric%7D%3Dvalue%0A%7C%20stats%20values(ResSet)%20as%20ResSet%2C%20values(ResText)%20as%20ResText%2C%20values(ResData)%20as%20ResData%20by%20_time%2C%20OStype%2C%20host%2C%20dimension_Command%0A%7C%20fillnull%20value%3D0%20ResSet%20ResText%20ResData%0A%7C%20eval%20mem_usage_mb%3Dcase(match(OStype%2C%20&quot;Linux&quot;)%2C%20((ResSet)%2F1024)%2C%20match(OStype%2C%20&quot;AIX&quot;)%2C%20((ResData%2BResText)%2F1024)%2C%20match(OStype%2C%20&quot;Solaris&quot;)%2C%20(ResSize%2F1024)%20)%0A%7C%20timechart%20%60nmon_span%60%20limit%3D10%20useother%3Df%20max(mem_usage_mb)%20AS%20mem_usage_mb%20by%20dimension_Command&amp;display.page.search.tab=visualizations&amp;display.general.type=visualizations&amp;display.visualizations.charting.axisY.minimumNumber=0&amp;display.visualizations.charting.chart=area&amp;display.visualizations.charting.axisY.maximumNumber=&amp;display.visualizations.charting.legend.placement=top&amp;earliest=-24h&amp;latest=now&amp;display.visualizations.charting.chart.stackMode=stacked&amp;display.visualizations.charting.axisTitleY.text=Logical%20Core%20CPU%20usage" class="tryitbtn-blue">Open in search »
						</a>
					</div>
					<br />
				</div>

				<div style="text-align: left;">
					<img src="../../static/app/metricator-for-nmon/various/mini/minicolumn.png" alt="minicolumn"/>
					<br />
					<br />
				</div>

				<div>
					<h2>Column chart of TOP 20 average CPU usage in percentage over last 24 hours</h2>
					<pre>| mstats max(_value) as value where `nmon_metrics_index` metric_name=os.unix.nmon.processes.top.pct_CPU host=$host$ (dimension_Command=$Command$) (dimension_PID=*) by host dimension_Command dimension_PID span=1m
| stats sum(value) as pct_CPU by _time, dimension_Command
| eval usage_per_core=(pct_CPU/100)
| eval xlabel=""
| chart useother=f limit=20 avg(usage_per_core) as usage_per_core by xlabel, dimension_Command | rename xlabel AS "TOP 10 CPU % usage"</pre>
				</div>

				<div>
					<div class="cat_title">
						<a target="_blank" href="search?q=%7C%20mstats%20max(_value)%20as%20value%20where%20%60nmon_metrics_index%60%20metric_name%3Dos.unix.nmon.processes.top.pct_CPU%20host%3D$host$%20(dimension_Command%3D$Command$)%20(dimension_PID%3D*)%20by%20host%20dimension_Command%20dimension_PID%20span%3D1m%0A%7C%20stats%20sum(value)%20as%20pct_CPU%20by%20_time%2C%20dimension_Command%0A%7C%20eval%20usage_per_core%3D(pct_CPU%2F100)%0A%7C%20eval%20xlabel%3D&quot;&quot;%0A%7C%20chart%20useother%3Df%20limit%3D20%20avg(usage_per_core)%20as%20usage_per_core%20by%20xlabel%2C%20dimension_Command%20%7C%20rename%20xlabel%20AS%20&quot;TOP%2010%20CPU%20%25%20usage&quot;&amp;earliest=-24h&amp;latest=now&amp;display.page.search.tab=visualizations&amp;display.general.type=visualizations" class="tryitbtn-blue">Open in search »
						</a>
					</div>
					<br />
				</div>

				<div>
					<h2>Column chart of TOP 20 average memory usage MB over last 24 hours</h2>
					<pre>| mstats max(_value) as value where `nmon_metrics_index` metric_name=os.unix.nmon.processes.top.ResSet OR metric_name=os.unix.nmon.processes.top.ResText OR metric_name=os.unix.nmon.processes.top.ResData host=$host$ dimension_Command=$Command$ by OStype, host, metric_name, dimension_Command, dimension_PID span=1m
| stats sum(value) as value by _time, metric_name, OStype, host, dimension_Command
| `extract_metrics`
| eval {metric}=value
| stats values(ResSet) as ResSet, values(ResText) as ResText, values(ResData) as ResData by _time, OStype, host, dimension_Command
| fillnull value=0 ResSet ResText ResData
| eval mem_usage_mb=case(match(OStype, "Linux"), ((ResSet)/1024), match(OStype, "AIX"), ((ResData+ResText)/1024), match(OStype, "Solaris"), (ResSize/1024) )
| eval xlabel=""
| chart useother=f limit=0 avg(mem_usage_mb) as mem_usage_mb by xlabel,dimension_Command | rename xlabel AS "TOP 10 memory MB usage"</pre>
				</div>

				<div>
					<div class="cat_title">
						<a target="_blank" href="search?q=%7C%20mstats%20max(_value)%20as%20value%20where%20%60nmon_metrics_index%60%20metric_name%3Dos.unix.nmon.processes.top.ResSet%20OR%20metric_name%3Dos.unix.nmon.processes.top.ResText%20OR%20metric_name%3Dos.unix.nmon.processes.top.ResData%20host%3D$host$%20dimension_Command%3D$Command$%20by%20OStype%2C%20host%2C%20metric_name%2C%20dimension_Command%2C%20dimension_PID%20span%3D1m%0A%7C%20stats%20sum(value)%20as%20value%20by%20_time%2C%20metric_name%2C%20OStype%2C%20host%2C%20dimension_Command%0A%7C%20%60extract_metrics%60%0A%7C%20eval%20%7Bmetric%7D%3Dvalue%0A%7C%20stats%20values(ResSet)%20as%20ResSet%2C%20values(ResText)%20as%20ResText%2C%20values(ResData)%20as%20ResData%20by%20_time%2C%20OStype%2C%20host%2C%20dimension_Command%0A%7C%20fillnull%20value%3D0%20ResSet%20ResText%20ResData%0A%7C%20eval%20mem_usage_mb%3Dcase(match(OStype%2C%20&quot;Linux&quot;)%2C%20((ResSet)%2F1024)%2C%20match(OStype%2C%20&quot;AIX&quot;)%2C%20((ResData%2BResText)%2F1024)%2C%20match(OStype%2C%20&quot;Solaris&quot;)%2C%20(ResSize%2F1024)%20)%0A%7C%20eval%20xlabel%3D&quot;&quot;%0A%7C%20chart%20useother%3Df%20limit%3D0%20avg(mem_usage_mb)%20as%20mem_usage_mb%20by%20xlabel%2Cdimension_Command%20%7C%20rename%20xlabel%20AS%20&quot;TOP%2010%20memory%20MB%20usage&quot;&amp;earliest=-24h&amp;latest=now&amp;display.page.search.tab=visualizations&amp;display.general.type=visualizations" class="tryitbtn-blue">Open in search »
						</a>
					</div>
					<br />
				</div>

			</html>

		</panel>
	</row>

	<row>
		<panel>
			<title>Table Statistics:</title>

			<html>

				<div style="text-align: left;">
					<br />
					<img src="../../static/app/metricator-for-nmon/various/mini/minitable.png" alt="minitable"/>
					<br />
					<br />
				</div>

				<div>
					<h2>Max / Average CPU Usage in Logical Cores and Global % usage over last 24 hours ordered and sorted by Max Core Usage, frame ID, host</h2>
					<pre>| mstats max(_value) as value where `nmon_metrics_index` metric_name=os.unix.nmon.processes.top.pct_CPU host=$host$ (dimension_Command=$Command$) (dimension_PID=*) by host dimension_Command dimension_PID span=1m
| stats sum(value) as pct_CPU by _time, host, dimension_Command
| eval usage_per_core=(pct_CPU/100)
| where (usage_per_core>0)
| stats max(usage_per_core) AS max_usage_per_core, avg(usage_per_core) AS avg_usage_per_core,
median(usage_per_core) AS median_usage_per_core, min(usage_per_core) AS min_usage_per_core, sparkline(avg(usage_per_core)) As sparkline by host,dimension_Command
| sort - avg_usage_per_core,host
| foreach max_* avg_* median_* min_* [ eval &lt;&lt;FIELD&gt;&gt;=round(&lt;&lt;FIELD&gt;&gt;, 2) ]</pre>
				</div>

				<div>
					<div class="cat_title">
						<a target="_blank" href="search?q=%7C%20mstats%20max(_value)%20as%20value%20where%20%60nmon_metrics_index%60%20metric_name%3Dos.unix.nmon.processes.top.pct_CPU%20host%3D$host$%20(dimension_Command%3D$Command$)%20(dimension_PID%3D*)%20by%20host%20dimension_Command%20dimension_PID%20span%3D1m%0A%7C%20stats%20sum(value)%20as%20pct_CPU%20by%20_time%2C%20host%2C%20dimension_Command%0A%7C%20eval%20usage_per_core%3D(pct_CPU%2F100)%0A%7C%20where%20(usage_per_core&gt;0)%0A%7C%20stats%20max(usage_per_core)%20AS%20max_usage_per_core%2C%20avg(usage_per_core)%20AS%20avg_usage_per_core%2C%0Amedian(usage_per_core)%20AS%20median_usage_per_core%2C%20min(usage_per_core)%20AS%20min_usage_per_core%2C%20sparkline(avg(usage_per_core))%20As%20sparkline%20by%20host%2Cdimension_Command%0A%7C%20sort%20-%20avg_usage_per_core%2Chost%0A%7C%20foreach%20max_*%20avg_*%20median_*%20min_*%20%5B%20eval%20&lt;&lt;FIELD&gt;&gt;%3Dround(&lt;&lt;FIELD&gt;&gt;%2C%202)%20%5D&amp;earliest=-24h&amp;latest=now&amp;display.page.search.tab=statistics&amp;display.general.type=statistics" class="tryitbtn-blue">Open in search »
						</a>
					</div>
					<br />
				</div>

			</html>

		</panel>
	</row>

</form>
