Alerts and Notification > Programmable Alerts Conditions and Use Cases
1. Getting Started with vuSmartMaps™
3. Console
5. Configuration
6. Data Management
9. Monitoring and Managing vuSmartMaps™
vuSmartMaps lets you use Python scripts to create programmable alerts. Using an evaluation script, you can generate alerts for breaching any business logic. Below is a typical alert engine execution workflow and where the evaluation script is used.
The evolution script runs after metrics are checked and thresholds are applied, allowing you to customize alert behavior. Apart from implementing business logic to generate the alert, you can also tweak alert notification content and channel settings like who gets notified.
vuSmartMaps makes various parameters like a list of metrics, grouping values, and metadata in the evaluation script for you to use in your programming logic. The following table lists the input parameters available to the evaluation script
Parameter | Type | Description | How to Access |
R<n> | Boolean | True or False indicating whether Rule-n thresholds matched or not | The result of Rule-1 is available as variable R1, Rule-2 as R2, etc |
D | List | The Data Model results list | D[0] contains Data Model values for Rule-1, D[1] for Rule-2 etc. Please note the array like indexing. |
grouping_values | List | Grouping values associated with the notification. This corresponds to buckets configured in Data Model. | grouping_value[0] for first bucketing value, grouping_value[1] for second etc. |
META_DATA | Dictionary | Metadata associated with this alarm | |
META_DATA[‘duration’] | Seconds | Duration for which this alarm has been active | META_DATA.get(‘duration, 0) |
META_DATA[‘history’] | Dictionary | History of this alarm |
The following table lists the output parameters available to the evaluation script for controlling the behavior of the alarm.
Parameter | Type | Description | How to Access |
RESULT | Boolean | Setting this value to True results mean alert should be generated. while setting this to False will not generate the alerts. state. | Eg: RESULT = True or RESULT = False |
R<n> | Boolean | True or False indicating whether Rule-n thresholds matched or not | The result of Rule-1 is available as variable R1, Rule-2 as R2, etc |
D | List | The Data Model results list | D[0] contains Data Model values for Rule-1, D[1] for Rule-2 etc |
META_DATA | Dictionary | Metadata associated with this alarm | |
META_DATA[‘force_update’] | Boolean | True or False indicating whether to send an update notification for this alert or not |
By default, vuSmartMaps generates an alert only when all conditions in an alert rule are true. So if there are two rules R1 and R2, an alert will be generated if both R1 and R2 are True.
In the evaluation script, each rule’s result is represented by variables like R1, R2, etc., where True is represented as “R1 = True.” You can use logical operators in the evaluation script. So the above two rules evaluation can be represented in the evaluation script as:
if R1 and R2:
return True;
You can use “and” and “or” logic to create complex conditions, like “R1 and (R2 or R3) and not R4.” So for instance, the evaluation script will look like the following if either R1 or R2 condition is to be met:
if R1 or R2:
return True;
This gives you control over when alerts are generated based on your specific criteria and combinations of conditions.
In the evaluation script, you can access the current values of the metrics used in your alert conditions. Each metric can be accessed through a multi-level Python data structure.
To simplify this process, an accessor function is available to help the evaluation script retrieve the values of specific metric columns from the data model used in your alert rule. This allows you to make informed decisions in your script about whether or not to trigger an alert.
D – Rule result dictionary rule_identifier – Identifier of the rule (1, 2 etc) metric_column – Metric column Example: get_vumetric_value(D, 1, ‘success_rate’) Internally, the accessor function validates the presence of the metric and returns the value from the multi-level dictionary hierarchy.def |
current_success = get_vumetric_value(D, 1, ‘success_rate’) daily_avg = get_vumetric_value(D, 1, ‘daily_average’) RESULT = True if current_success is not None and daily_avg is not None: Ratio = current_success/daily_avg If ratio < .75: RESULT = False |
The metric values can be used to decide on whether to generate an alert or not.
If we want to generate a warning alert if the value is between 80 and 90 and a critical alert if the value is > 90 and should be sent to a few extra folks. So a data model will be created with a threshold of 80. Then we will write Python code to do this check and update META_DATA also to include more people in critical cases.
bw = get_vumetric_value(D, 1, ‘input_bandwidth’) If bw and bw >= 80 and bw < 90: RESULT = True severity = warning elif bw and bw >= 90: RESULT = True severity = critical else: RESULT = False |
Suppose you want to generate an alert only when the transaction success rate is 25% lower than the daily average. You can achieve this by creating a custom evaluation script.
For example, if you have a Data Model named “Success Rates” with metrics “Current Rate” and “Daily Average,” the following script can be used:
current_success = get_vumetric_value(D, 1, ‘Current Rate’) daily_avg = get_vumetric_value(D, ‘1’, ‘Daily Average’) RESULT = True if current_success is not None and daily_avg is not None: if daily_av: # avoid division by zero ratio = current_success/daily_avg else: ratio = 0 If ratio < .75: RESULT = False |
💡Note: In the script provided earlier, the variable RESULT plays a crucial role in determining whether an alert should be generated. If you set RESULT to True, an alert will be triggered for the specific situation. If RESULT is set to False, no alert will be generated. If the script doesn’t modify the value of RESULT, no alert will be generated by default.
In summary, RESULT serves as the output variable that allows the evaluation script to control when alert notifications are generated based on the conditions and logic you define.
You can also access grouping values for a specific alert being evaluated in your scripts. This is useful if your notifications involve multiple grouping levels. For instance, if you’re grouping alerts by hostname and interface name, you can access these values within your script as demonstrated in the example provided.
if grouping_values[0] == ‘AppServer’ and grouping_values[1] == ‘serial-1-1’ RESULT = False else: RESULT = True |
In the example above, we use grouping values to avoid generating alerts for the serial interface on the host “AppServer.” The script accesses grouping values through the “grouping_values” list, which contains the values for each level of grouping. You can access these values using the Python syntax, such as “grouping_values[0]” and “grouping_values[1].” This allows you to customize alert generation based on specific grouping criteria.
You can add new fields to the notifications generated by the system using the evaluation script.
For instance, if you need to include a new field or category with values based on the transaction success rate metric, you can achieve this with the following script snippet.
success_rate = get_vumetric_value(D, 1, ‘Success Rate’) if success_rate and success_rate > 90: DYNAMIC_FIELDS[‘category’] = ‘Normal’ else: DYNAMIC_FIELDS[‘category’] = ‘Need Investigation’ RESULT = True |
As can be seen, any field to be added to the notification generated can be specified in the DYNAMIC_FIELDS dictionary with the corresponding key and value.
You can control the notification channels and recipients in the evaluation script using the ALERT_CHANNELS list. You can add or remove items from this list to include or exclude specific channels for a particular alert. Here are the keywords for different channels:
For example, the following code snippet can be used to add Email as one of the channels and remove the Ticketing system as a channel based on conditions.
success_rate = get_vumetric_value(D, 1, ‘Success Rate’) if success_rate and success_rate > 90: ALERT_CHANNELS.append(‘alertByEmail’) EMAIL_ID_LIST = [‘[email protected]’,’[email protected]’] elif success_rate < 80: ALERT_CHANNELS.append(‘alertByTicket’) RESULT = True |
Within each channel, a similar facility is available to control the recipients. In the above, two email addresses are configured as recipients.
The list of controls available for different channels is shown below
Field | Channel | Description |
EMAILID_LIST | List of email addresses. Eg: [email protected] | |
EMAILGROUP_LIST | List email group names. Eg: Support | |
REPORT_LIST | Report | List of Reports. Eg: the “CXO Report” |
PHONE_NUMBER_LIST | List of phone numbers. Eg: 9881 234 567 | |
RUNBOOK_SCRIPT | Runbook | Runbook script name. Eg: service_restart |
The severity of the alert can be modified using an evaluation script. For example, the severity of the alert is increased to Critical for a certain range of values.
success_rate = get_vumetric_value(D, 1, “Success Rate”) if success_rate and success_rate > 90: DYNAMIC_FIELDS[‘severity’] = ‘critical’ RESULT = True |
Similar to severity, summary and description fields can be modified using an evaluation script.
success_rate = get_vumetric_value(D, 1, ‘Success Rate’) if success_rate and success_rate > 90: DYNAMIC_FIELDS[‘summary’] = ‘Resource Usage High for %g’ DYNAMIC_FIELDS[‘description’] = “Investigation of this server …..” RESULT = True |
As can be seen in the above example, the summary and description created by the script can make use of the format specifiers supported by the system. Please refer to Step 1 of alert creation in this manual for more details on this.
The duration for which the current alarm condition has been active is available in the META_DATA dictionary. This can be used for escalating alerts based on the active duration.
In the below example, the script implements an escalation of the alarm condition by sending out a notification to a larger group, if the alarm condition has continued for more than 6 hours. We are keeping it in.
if META_DATA[‘duration’] > 6 *60: ALERT_CHANNELS.append(‘alertByEmail’) EMAIL_ID_LIST = [‘[email protected]’,’[email protected]’] RESULT = True |
An evaluation script can make use of the history of this alarm condition to decide on the alert behavior. The below example does escalation notification if the condition has been activated more than X number of times in the last 1 week.
if len(META_DATA[‘history’][‘list’]) > 10: ALERT_CHANNELS.append(‘alertByEmail’) EMAIL_ID_LIST = [‘[email protected]’,’[email protected]’] RESULT = True |
We can use the enrichments available in vuSmartMaps to enrich the alert document or to use those enrichments in evaluation scripts for any other purpose.
We have a lookup function available for this named “get_value_from_lookup_file”. It takes the following arguments –
The given “key” can either be a single key or a list of keys
– In case of multi-level lookups
– For ex: key = [“circle1”, “region2”, “code”] in a lookup
– circle1:
region1:
code: ‘255’
region2:
code: ‘254’
– will give ‘254’ as output
RESULT = False if R1: add_fields = D[0][‘Alert for WAN Link Down – BGP State Change SNMP – North’][‘metrics’][‘BGP Peer State’][‘includes’] M1 = get_vumetric_value(D, 2, “LinkUsage”) M2 = get_vumetric_value(D, 2, “bgp_peer_remote_address”) M3 = get_vumetric_value(D, 2, “Circle”) M4 = get_vumetric_value(D, 2, “BranchName”) M5 = get_vumetric_value(D, 2, “BranchCode”) M6 = get_vumetric_value(D, 2, “DeviceIP”) M7 = get_vumetric_value(D, 2, “ISP”) AssignmentGroup =get_value_from_lookup_file(“1”, “1”, “Assignment-Grp.yml”, [M3, M7, “AssignmentGroup”]) DYNAMIC_FIELDS[“Assignment_Group”] = AssignmentGroup DYNAMIC_FIELDS[“Assigned_Organization”] = get_value_from_lookup_file(“1”, “1”, “AssignedOrg.yml”, [AssignmentGroup,”Organization”]) DYNAMIC_FIELDS[“code”] = get_value_from_lookup_file(“1”, “1”, “code.yml”, “Nexus”) |
In certain cases, the decision on alert may have to be made based on the time at which the alert is being generated.
The time of alert is available in the OBSERVATION_TIME variable.
For example, if different thresholds are to be used for business hours and non-business hours, the following logic can be used.
success_rate = get_vumetric_value(D, 1, ‘success_rate’) # Time in the local time zone at which this alert is being generated # OBSERVATION_TIME is a Python datetime object and all # operations/functions supported in datetime object can be used on this # hour of the day hour = int(OBSERVATION_TIME.strftime(“%I”)) If hour >= 9 and hour <= 17: threshold = 80 else threshold = 60 if success_rate and success_rate > threshold: RESULT = True else: RESULT = False |
If different thresholds are to be used for weekdays and weekends, the following logic can be used
success_rate = get_vumetric_value(D, 1, ‘Success Rate’) # Time in the local time zone at which this alert is being generated # OBSERVATION_TIME is a Python datetime object and all # operations/functions supported in datetime object can be used on this # Day of the week day = OBSERVATION_TIME.strftime(“%A”) If day == ‘Sunday’ or day == ‘Saturday’: threshold = 50 else threshold = 70 if success_rate and success_rate > threshold: RESULT = True else: RESULT = False |
Internally, OBSERVATION_TIME is the last alert execution time in the local timezone
OBSERVATION_TIME = self.last_execution_end_time.astimezone(tz.tzlocal()) |
Below is a map of the columns/fields in the Alert Console list view table. The keys are the Column names in the UI table and the values are a list of fields using any of which we can populate the corresponding column in the UI. This has to be done using `DYNAMIC_FIELDS` in the evaluation script.
Backend mapping logic
“Region”: [“location”, “Location”],
“Assignee”: [“assignee”],
“Status”: [“status”],
“Device IP”: [“DeviceIP”, “target”, “host”],
“Affected Category”: [“category”],
“Business Impact”: [“impact”],
“Source”: [“source”],
As you can see, above is the mapping kept in backend where the Key is the actual name of column used in Alert console columns and the value(s) are the fields you can use in Evaluation script (Dynamic fields) for mapping the actual values. For example, to populate `Device IP` column in the Alert Console table, one can add any of the following in the evaluation script:
Dynamic Fields in Evaluation Script
DYNAMIC_FIELDS[‘Device IP’] = ‘1.0.0.0’ OR
DYNAMIC_FIELDS[‘target’] = ‘1.0.0.0’ OR
DYNAMIC_FIELDS[‘host’] = ‘1.0.0.0’
Now suppose you want to give dynamic values to the key instead of a static value, then you can use the following method in evaluation script.
First get the values in a list and then assign this list to the dynamic field. For example, here we are trying to get the IP addresses in the data model (column name: IP) into a list ‘device_ip’.
Then this list ‘device_ip’ is mapped to the dynamic field ‘target’.
device_ip = get_vumetric_value(D, 1,‘IP’) //here IP is the Data model column name
//which has IP addresses
DYNAMIC_FIELDS[‘target’] = device_ip
Currently, only ‘Region’ wise Alert trend is supported. This can be configured in the same way as above.
DYNAMIC_FIELDS[‘location’] = ‘Bengaluru’
OR, if there
city = get_vumetric_value(D,2,’Region’)
DYNAMIC_FIELDS[‘location’] = city
To configure Related dashboards of an alert, use the YAML configuration as shown below. “Device Availability Overview” is the name of the dashboard which is preconfigured and called here.
The output of this configuration is seen in the ‘detailed view’ of the Alerts on the Alert Console.
Click on the name of the dashboard highlighted and the respective dashboard will be displayed.
Browse through our resources to learn how you can accelerate digital transformation within your organisation.
VuNet’s Business-Centric Observability platform, vuSmartMaps™ seamlessly links IT performance to business metrics and business journey performance. It empowers SRE and IT Ops teams to improve service success rates and transaction response times, while simultaneously providing business teams with critical, real-time insights. This enables faster incident detection and response.