4.1 Configuring Observability Sources
4.1.1 Observability Sources
4.1.2 Data Pipeline and Parser Configuration
4.1.3 Data Enrichment Techniques
4.2 Configuring RCABot and ML Models
7.2 Authentication and Security
7.3 Data Management and Data Model Handling
7.3.1 Storage
7.3.2 Retention
7.3.3 Export/Import
7.3.4 Working with Data Model
7.4 Control Center
7.4.1 License Entitlements
7.5 Platform Settings
7.5.1 Definitions
7.5.2 Preferences
7.5.3 About
Alerts and Notification > Flexible Alert Rule Modification
The Alert Evaluation script offers an extensive Python programming interface for customizing alert behavior. You can use this interface for complex alerting scenarios and specific requirements.
For most standard cases, you can skip this section. If needed, refer to the detailed documentation on the Alert Evaluation Script for programming guidance.
Suppose you want to generate an alert only when the transaction success rate is 25% lower than the daily average. You can achieve this by creating a custom evaluation script.
For example, if you have a Data Model named “Success Rates” with metrics “Current Rate” and “Daily Average,” the following script can be used:
current_success = get_vumetric_value(D, 1, ‘Current Rate’)daily_avg = get_vumetric_value(D, ‘1’, ‘Daily Average’) RESULT = True if current_success is not None and daily_avg is not None: if daily_av: # avoid division by zero ratio = current_success/daily_avg else: ratio = 0 If ratio < .75: RESULT = False |
In the script provided earlier, the variable RESULT plays a crucial role in determining whether an alert should be generated. If you set RESULT to True, an alert will be triggered for the specific situation. If RESULT is set to False, no alert will be generated. If the script doesn’t modify the value of RESULT, no alert will be generated by default.
In summary, RESULT serves as the output variable that allows the evaluation script to control when alert notifications are generated based on the conditions and logic you define. |
In the evaluation script, you can access the current values of the metrics used in your alert conditions. Each metric can be accessed through a multi-level Python data structure. To simplify this process, an accessor function is available to help the evaluation script retrieve the values of specific metric columns from the data model used in your alert rule.
This allows you to make informed decisions in your script about whether or not to trigger an alert.
D – Rule result dictionary rule_identifier – Identifier of the rule (1, 2 etc) metric_column – Metric column D – Rule result dictionary rule_identifier – Identifier of the rule (1, 2 etc) metric_column – Metric column Example: get_vumetric_value(D, 1, ‘success_rate’) Internally, the accessor function validates the presence of the metric and returns the value from the multi-level dictionary hierarchy.def get_vumetric_value(D, rule_identifier: int, metric_column: str): “”” Accessor function for use by evaluation script. This gives a wrapper to access metric values in the results from evaluation script
Input —– D – Rule result dictionary rule_identifier – Identifier of the rule result dictionary (1, 2 etc) metric_column – Metric column
Output —— Metric value or None “”” if not len(D) >= rule_identifier: return None
# Get the metric dict from the result list metric_dict = D[rule_identifier – 1]
if not metric_dict: return None
if isinstance(metric_dict, dict): return metric_dict.get(metric_column)
if isinstance(metric_dict, list): if len(metric_dict) == 1: return metric_dict[0].get(metric_column) else: return metric_dict |
current_success = get_vumetric_value(D, 1, ‘success_rate’) daily_avg = get_vumetric_value(D, 1, ‘daily_average’)
RESULT = True
if current_success is not None and daily_avg is not None: Ratio = current_success/daily_avg
If ratio < .75: RESULT = False |
The metric values can be used to decide on whether to generate an alert or not.
For example, to generate alerts only when the Input BW is above 90%, the following code can be used.
bw = get_vumetric_value(D, 1, ‘input_bandwidth’)
if bw and bw > 90: RESULT = True else: RESULT = False |
You can add new fields to the notifications generated by the system using the evaluation script.
For instance, if you need to include a new field or category with values based on the transaction success rate metric, you can achieve this with the following script snippet.
success_rate = get_vumetric_value(D, 1, ‘Success Rate’)
if success_rate and success_rate > 90: DYNAMIC_FIELDS[‘category’] = ‘Normal’ else: DYNAMIC_FIELDS[‘category’] = ‘Need Investigation’
RESULT = True |
As can be seen, any field to be added to the notification generated can be specified in the DYNAMIC_FIELDS dictionary with the corresponding key and value.
The severity of the alert can be modified using an evaluation script. For example, the severity of the alert is increased to Critical for a certain range of values.
success_rate = get_vumetric_value(D, 1, “Success Rate”)
if success_rate and success_rate > 90: DYNAMIC_FIELDS[‘severity’] = ‘critical’
RESULT = True |
Similar to severity, summary and description fields can be modified using an evaluation script.
success_rate = get_vumetric_value(D, 1, ‘Success Rate’)
if success_rate and success_rate > 90: DYNAMIC_FIELDS[‘summary’] = ‘Resource Usage High for %g’ DYNAMIC_FIELDS[‘description’] = “Investigation of this server …..”
RESULT = True |
In this step, you can set up how alert notifications work for this rule. You can configure notification channels, and recipients, enable or disable alarm mode, and control the intervals for active alert rule notifications.
Enable Alarm Mode: When activated, the system monitors the alarm state, sending notifications when the alert condition is met and when it’s cleared. No additional notifications are sent while the condition remains active. For example, when the queue size exceeds 80%, an active alarm notification is sent, but no more are sent while the queue size is above the threshold. When it drops below 80%, a clear notification is sent.
Disable Alarm Mode: If this mode is turned off, notifications are sent at regular intervals as long as the alert condition is active. In this case, the system doesn’t track the alarm state, and no clear notifications are generated. For example, if the queue size stays above 80%, vuSmartMaps will send a notification every 5 minutes.
Non-alarm mode notifications are beneficial when operators want to receive regular updates about the status of a monitored metric or component.
Throttling is only active when alarm mode is turned off. With throttling enabled, the system will not send repeated notifications for the same condition until the configured interval has passed.
For instance, if you set the throttling interval to 2 hours, you will only receive a second notification about high CPU usage for a specific server 2 hours after the initial notification. This is handy to prevent constant notifications when alarm mode is disabled.
Let’s say you want notifications about the number of user logins to an application every 2 hours. To do this, set up an alert rule in non-alarm mode with a metric for the login count and a threshold greater than 0. Configure a throttling duration of 2 hours.
This way, you’ll receive these notifications every 2 hours.
This configuration is handy for avoiding notifications during periods of lower activity, such as weekends or non-business hours, ensuring that you only receive notifications when it’s most relevant.
You can specify an alert’s active period to prevent it from triggering alerts during planned maintenance activities. During this period, the rule won’t generate alert notifications, but you can still access them in the Events section for reference.
You can use the Advanced Configuration to add new functionalities to alert rules through a YAML interface.
It’s a way to configure features that may not be available through the regular menu options.
For instance, you can configure the alert notification level using this interface by specifying it in the advanced configuration text area.
notification_level: 0
For more details on the notification level, check out the examples section.
Related Dashboards:
To config-related dashboards of an alert, use the YAML configuration as shown below.
Tags:
To add tags during Alert configuration, enable the Advanced Configuration section and use a YAML script like the one shown in the attached snapshot below.
Browse through our resources to learn how you can accelerate digital transformation within your organisation.
VuNet Systems is a next-gen visibility and analytics company that uses full-stack AI & Big Data analytics to accelerate digital transformation within an organisation. We provide deep observability into business journeys to reduce failures and enhance overall customer experience.