On the Scheduling Activity dashboard of the Scheduling Assistant element, navigate down to the Concurrency Investigation Header. There will be two visuals that will help identify the source of concurrency issues.
Using the Cron Schedule Details Table
This report enables users to sort by Concurrent Scheduling per Year or Average Concurrency to identify searches that run next to many other searches or on average run in congested times. Factor in the number of times the search is ran in the Scheduled Runs per Year column to see the effect improving this search will have on the environment.
Scheduled Runs per Year:
- This field shows how many times this search will run on the environment in a 365-day period.
Concurrent Schedulings per Year:
- This field counts how many searches run at the same time as the identified search in a 365-day period.
Average Concurrency:
- This field contains the average of the number of concurrent searches that fire off when the identified search runs.
A Scheduled Search called ExampleSearch with 10 Scheduled Runs Per Year, 200 Concurrent Schedulings per Year, and an Average Concurrency of 20 means:
- EampleSearch runs 10 times a year, and during the timeslot it runs, 200 other searches also start, with an average of around 20 other searches for each time it runs.
It is recommended to find a Scheduled Search with all three of these numbers being high, and then ensure that its App isn’t Enterprise Security or other premium app’s unless one operates the premium app as well. After identifying a search, continue with improving the search’s scheduling on Scheduling Assistant.
Using the Historical Searches Scheduled In Time Range Time Chart
Under the Concurrency Investigation header & past the Concurrency Investigation Table, there is a Time Chart that offers a visual analysis of Concurrency spikes. Select ‘minutes’ as the Time Unit to get a clear picture.
These bars represent concurrency spikes in the system. The two limits offer insights into how environments can handle multiple searches and crossing the bottom threshold of Max Historical Scheduled Searches is tracked as a concurrency warning in the Health Score. You can identify moments when the bars representing stacked searches crosses the threshold. Splunk’s zoom ability can be used to focus on a specific point.
After zooming, we have identified a time period where searches cross the limit. Select the bar to view the searches.
This group of searches are stacked so much on top of each other it poses a threat to the Splunk environment, and if not properly managed, can lead to more skipped searches and issues for users. To continue with improving the search’s scheduling, identify a search that should be changed and press the ‘Go’ on the ‘Assistant’ column (for more info, click here).