Core Dumps Disabled Transparent Huge Pages is enabled and should not be ulimit on Splunk enterprise servers is below 8192 MonitoringConsole - Check OS ulimits via REST KVStore Process Terminated Unable to dispatch searches due to disk space Low disk space Splunkd Crash Logs Have Appeared in Production AllSplunkLevel - Unexpected termination of a Splunk process unix AllSplunkLevel - Unexpected termination of a Splunk process windows Splunk Servers with resource starvation Time skew on Splunk Servers Detect LDAP groups that no longer exist File integrity check failure Non-existent roles are assigned to users TCP or SSL Config Issue WARN iniFile Configuration Issues error in stdout.log Application Installation Failures From Deployment Manager DeploymentServer Application Installation Error Email Sending Failures sendmodalert errors Splunk Servers throwing runScript errors execprocessor errors Data Loss on shutdown AllSplunkLevel - TailReader Ignoring Path AllSplunkLevel - No recent metrics.log data Splunk Scheduler excessive delays in executing search Splunk Scheduler skipped searches and the reason Replication Failures Unable To Distribute to Peer Data Loss on shutdown Losing Contact With Master Node AllSplunkLevel - No recent metrics.log data Splunkd Log Messages Admins Only Per index status Excess buckets on master ClusterMaster Advising SearchOrRep Factor Not Met Splunkd Log Messages Admins Only Losing Contact With Master Node Application Not Found On Deployment Server btool validation failures occurring on deployment server Forwarder has changed properties on phone home Unsupported attribute within DS config Application Installation Failures From Deployment Manager Splunkd Log Messages Admins Only Error Found On Deployment Server Forwarders in restart loop Splunk Forwarder Down Splunk forwarders failing due to disk space issues Splunk Universal Forwarders that are time shifting Splunk universal forwarders with ulimit issues Splunk Universal Forwarders Exceeding the File Descriptor Cache MonitoringConsole - Check OS ulimits via REST (useful for HF's only) crcSalt or initCrcLength change may be required File Too Small to checkCRC occurring multiple times Splunk Insufficient Permissions to Read Files Splunk forwarders that are not talking to the deployment server Bandwidth Throttling Occurring Read operation timed out expecting ACK Splunk forwarders are having issues with sending data to indexers Splunk Heavy logging sources TCP Output Processor has paused the data flow Splunk HTTP Listener Overwhelmed SplunkStream Errors SSL Errors In Logs (Potential Universal Forwarder and LicenseIssue) Unusual number of duplication alerts Splunk HEC issues AllSplunkLevel - No recent metrics.log data Stopping all listening ports Splunkd Log Messages Admins Only Channel churn issues Buckets have being frozen due to index sizing Buckets have being frozen due to index sizing SmartStore Buckets rolling more frequently than expected These Indexes Are Approaching The warmDBCount limit strings_metadata triggering bucket rolling Data parsing error IndexConfig Warnings from Splunk indexers Index not defined ForwarderLevel - Stopping all listening ports IndexerLevel - replicationdatareceiverthread close to 100% utilisation Failures To Parse Timestamp Correctly (excluding breaking issues) Future Dated Events that appeared in the last week Large multiline events using SHOULD_LINEMERGE setting Old data appearing in Splunk indexes Time format has changed multiple log types in one sourcetype Timestamp parsing issues combined alert Too many events with the same timestamp Valid Timestamp Invalid Parsed Time Weekly Broken Events Report Weekly Truncated Logs Report S2SFileReceiver Error Unclean Shutdown - Fsck AllSplunkEnterpriseLevel - Losing Contact With Master Node IndexerLevel - SmartStore - Bucket cache errors audit logs AllSplunkLevel - No recent metrics.log data Connection errors to SmartStore Splunkd Log Messages Admins Only Indexer Queues May Have Issues Indexer replication queue issues to some peers Slow peer from remote searches ForwarderLevel - Channel churn issues IndexerLevel - replicationdatareceiverthread close to 100% utilisation Indexer not accepting TCP Connections Uneven Indexed Data Across The Indexers ForwarderLevel - Stopping all listening ports Peer will not return results due to outdated generation Search Failures Cold data location approaching size limits Volume (Cold) Has Been Exceeded Indexer Out Of Disk Space Rolling Hot Bucket Failure Duplicated License Situation datamodel errors in splunkd Detect MongoDB errors Indexer Peer Connection Failures KVStore Or Conf Replication Issues Are Occurring Long filenames may be causing issues Script failures in the last day SHCluster Artifact Replication Issues SHC Captain unable to establish common bundle splunk_search_messages dispatch dispatch metadata files may need removal Dashboards invalid character in splunkd savedsearches invalid character in splunkd datamodel errors in splunkd IndexerLevel - SmartStore - Bucket cache errors audit logs AllSplunkLevel - No recent metrics.log data Detect bundle pushes no longer occurring Peer timeouts or authentication issues Splunkd Log Messages Admins Only Search Messages user level Search Messages admins only Realtime Scheduled Searches are in use Realtime Search Queries in dashboards Accelerated DataModels with All Time Searching Enabled Accelerated DataModels with wildcard or no index specified User - Dashboards searching all indexes macro version User - Dashboards searching all indexes SearchHeadLevel - Dashboards with all time searches set Scheduled searches not specifying an index macro version Scheduled searches not specifying an index Scheduled Searches without a configured earliest and latest time Splunk alert actions exceeding the max_action_results limit Splunk Scheduler logs have not appeared in the last SearchHeadLevel - summary indexing searches not using durable search SearchHeadLevel - Excessive REST API usage Captain Switchover Occurring Disabled modular inputs are running Long Running Searches Found SearchHeadLevel - SHC Captain unable to establish common bundle Slow peer from remote searches SearchHeadLevel - Excessive REST API usage LDAP users have been disabled or left the company cleanup required Saved Searches with privileged owners and excessive write perms Scheduled Searches Configured with incorrect sharing Splunk login attempts from users that do not have any LDAP roles SearchHeadLevel - authorize.conf settings will prevent some users from appearing in the UI SearchHeadLevel - summary indexing searches not using durable search Splunk Max Historic Search Limits Reached Splunk Users Violating the Search Quota Users exceeding the disk quota WLM aborted searches Scheduled searches failing in cluster with 404 error Scheduled Searches That Cannot Run savedsearches invalid character in splunkd The cluster_health_tools git repository contains very useful dashboards for various indexer related performance stats Extended Search Reporting (and others) Search Scheduler Tuning searches Sideview UI (User Activity details) Admins Little Helper for Splunk (btool, bundle utils and similar) TrackMe (Data Ingestion) Getting Smarter about Splunk SmartStore (including HEC dashboards) Maximizing Splunk Core: Analyzing Splunk Searches Using Audittrail and Native Splunk Telemetry Core dumps have appeared on the filesystem Crash logs have appeared on the filesystem one or more servers require configuration one or more servers require configuration automated