Identify Untriggered Alerts in Splunk
In this blog we ll get to know whole about alerts where they are saved and how to list those alerts which are triggered and which are not triggered?
Why is need to track the alerts which are triggered or not??
In our splunk environment we created many apps and work on many ongoing projects, we have to create many alerts with respective of each app where we keep track on whats going on in our projects . Imagine we have more apps and we have alerts in each of them so we wont have that much spare time to go and check each and every alerts in their apps which are triggered or not
Which we ll come to know which of the query needs to rewrite and arises problem in them.
So to get rid of that time consuming work for checking each and every alert check we have created a this blog which will let you know alerts created in all apps in splunk environment and to identify which of the alerts is not triggered so we can identify them and work on them.
How Alert work flow:-
1 — Set a search which return events
2 — How often we want that search to run to get specific events
3 — Set an alert condition, when condition is met alert will get triggered
Alerts when created are saved in specific app locations it can also be shared in all apps too
Location is :- /opt/splunk/etc/apps/[app-name]/local/savedsearches.conf
In our saved searches we have reports and alerts saved
How to identify Alerts and Reports in savedsearches.conf file??
alert.track = 1 is for saved search is configured as Alert alert.track = 0 is for saved search is configured as Report alert.severity values are 1 — debug, 2 — info, 3 — warn, 5 — severe, 6 — fatal
When set alerts are triggered they can be seen in Activity → Triggered Alerts
Here is an alert set in Search & Reporting app
Here is an alert set in Demo app
Alerts are triggered or not are set in Alerts column that how many times it is been triggered
If Alerts has 0 so that specific alert is not triggered or specific condition is not met for alert to get triggered
In above example for Search & Reporting App Alerts are triggered 144 times and for
Demo App created Alert count is 0 so that alert is not triggered yet
By selecting All App we can get results of all alerts which are configured in Splunk Environment
Query :-
|rest/servicesNS/-/search/saved/searches
| search alert.track=1
| fields title description search disabled triggered_alert_count actions action.script.filename alert.severity cron_schedule
| table title search disabled triggered_alert_count alert.severity cron_schedule
| rename title as “Alert Name”, search as “Alert query”, triggered_alert_count as “Triggered
Count”, cron_schedule as “Scheduling Time”, alert.severity as “Mode”
Above query is to get alerts which are scheduled in our environment it is for Search & Reporting app
For getting schedules alerts which are set in different app
|rest/servicesNS/-/[app — name]/saved/searches
| search alert.track=1
| fields title description search disabled triggered_alert_count actions action.script.filename alert.severity cron_schedule
| table title search disabled triggered_alert_count alert.severity cron_schedule
| rename title as “Alert Name”, search as “Alert query”, triggered_alert_count as “Triggered
Count”, cron_schedule as “Scheduling Time”, alert.severity as “Mode”
For getting scheduled alerts for all apps
|rest/servicesNS/-/-/saved/searches
| search alert.track=1
| fields title description search disabled triggered_alert_count actions action.script.filename alert.severity cron_schedule
| table title search disabled triggered_alert_count alert.severity cron_schedule
| rename title as “Alert Name”, search as “Alert query”, triggered_alert_count as “Triggered
Count”, cron_schedule as “Scheduling Time”, alert.severity as “Mode”
Listing Alerts which are Triggered
Query :-
|rest/servicesNS/-/-/saved/searches
| search alert.track=1
| fields title description search disabled triggered_alert_count actions action.script.filename alert.severity cron_schedule | fillnull value=0 triggered_alert_count
| where disabled=0 AND triggered_alert_count>0
| table title search disabled triggered_alert_count alert.severity cron_schedule
| rename title as “Alert Name”, search as “Alert query”, triggered_alert_count as “Triggered
Count”, cron_schedule as “Scheduling Time”, alert.severity as “Alert Severity”
Listing Alerts which are not Triggered in our Environment
Query :-
|rest/servicesNS/-/-/saved/searches
| search alert.track=1
| fields title description search disabled triggered_alert_count actions action.script.filename alert.severity cron_schedule
| fillnull value=0 triggered_alert_count
| where disabled=0 AND triggered_alert_count=0
| table title search disabled triggered_alert_count alert.severity cron_schedule
| rename title as “Alert Name”, search as “Alert query”, triggered_alert_count as “Triggered
Count”, cron_schedule as “Scheduling Time”, alert.severity as “Alert Severity”
Alerts are triggered or not can be identified by their Triggered Count if trigger count = 0 it is not triggered yet.
I hope you like this blog if you are still facing issue regarding this topic feel free to ask doubts in the comment section below and don’t forget to follow our more Splunk blogs on Avotrix.
happy Splunking >
