4.1 Configuring Observability Sources
4.1.1 Observability Sources
4.1.2 Data Pipeline and Parser Configuration
4.1.3 Data Enrichment Techniques
4.2 Configuring RCABot and ML Models
7.2 Authentication and Security
7.3 Data Management and Data Model Handling
7.3.1 Storage
7.3.2 Retention
7.3.3 Export/Import
7.3.4 Working with Data Model
7.4 Control Center
7.4.1 License Entitlements
7.5 Platform Settings
7.5.1 Definitions
7.5.2 Preferences
7.5.3 About
Insights > Template Breakdown
The configuration for the HTML template consists of
The Python script interface for the HTML template comes with predefined variables:
The Python script has access to the contents of the Data Models selected in the configuration. For example, the formatted data value inside a Data Model can be accessed in the script using the following syntax.
DM[‘<DM_name>’][‘<metric_label>’] [‘value’]
The link variable will redirect you to a dashboard or an event. The link can be configured with the Python script interface.
For example, if you want to drill down to the dashboard/event already configured in the Data Model we can use
LINK = DM[‘DM_NAME’][‘METRIC_NAME’][‘view_more’]
Or you can directly assign the dashboard name to the LINK variable.
LINK = ‘Dashboard Name’
For Templates, we are using mustache syntax for HTML. The important part is accessing data in the VALUE dictionary from the HTML Script. To do that all we have to do is put two curly braces around the key whose value we want to access.
Example:
Script: a=DM['Two_bucket']b=0 for i in range(0,len(a)): if a[i][‘Count’] > 1000: b=b+1 if b>10: VALUE[‘val’] = ‘Count is above threshold’ else: VALUE[‘val’] = ‘Count is below threshold’ VALUE[‘count’]=b |
Here is an example of how to change colors and fonts using HTML.
Script: import jsondata=DM['Server Infra Details']a=json.dumps(data) VALUE[‘Memory’]=[data[0][‘Percentage of Memory Utilized’]] VALUE[‘CPU’]=[data[0][‘Percentage of CPU Utilized’]] VALUE[‘Disk’]=[data[0][‘Percentage of Disk Space Utilized’]]
Output:
|
Here’s an example of How to create Bar Graph Insights for Top 5 Severity Distribution.
Now, we can use labels and attribute them to Data Models and Variables. In the future, new Data Models and Variables can be used without having to amend the entire code.
Example:
Data Model Label: dm1
Variable Label: category column
Script: error = False error_msg = “” if len(DM) > 0: is_data = True if “dm1” in DM: data = DM[‘dm1’] category_column = VAR.get(‘category_column’, None) count_column = VAR.get(‘count_column’, None) if category_column is None and count_column is None: error = True error_msg = “Variables category_column and count_column is to be added.” elif category_column is None: error = True error_msg = “Variables category_column is to be added.” elif count_column is None: error = True error_msg = “Variables count_column is to be added.” else: if category_column == count_column: error = True error_msg = “Variables category_column and count_column are the same.” else: dm_columns = [] if len(data) > 0: dm_columns = list(data[0].keys()) if category_column not in dm_columns and count_column not in dm_columns: error = True error_msg = “Coulumns “ + category_column + ” and “ + count_column + ” is not in the data model; Available Columns are “ + “, “.join(dm_columns) elif category_column not in dm_columns: error = True error_msg = “Coulumn “ + category_column + ” is not in the data model; Available Columns are “ + “, “.join(dm_columns) elif count_column not in dm_columns: error = True error_msg = “Coulumn “ + count_column + ” is not in the data model; Available Columns are “ + “, “.join(dm_columns) else: dm0 = data[0] if not isinstance(dm0[category_column], str) and isinstance(dm0[count_column], str): error = True error_msg = “Coulumn “ + category_column + ” is not a string, expected to be a string column; Column “ + count_column + ” is a string, expcted to be a numeric.” elif not isinstance(dm0[category_column], str): error = True error_msg = “Coulumn “ + category_column + ” is not a string, expected to be a string column.” elif isinstance(dm0[count_column], str): error = True error_msg = “Coulumn “ + count_column + ” is a string, expected to be a numeric.” else: no_of_bars = VAR.get(‘no_of_category_to_show’, None) bar_data = [] total = 0 for i in range(len(data)): bar_data.append([data[i][category_column], data[i][count_column]]) total += data[i][count_column]
bar_data.sort(key=lambda x: x[1], reverse = True) if no_of_bars is None: no_of_bars = len(bar_data) title = category_column.title() + ” distribution” else: title = “Top “ + str(no_of_bars) + ” “ + category_column.title() + ” distribution” out_data = [] for i in range(len(bar_data)): if i < no_of_bars: out_data.append({ “category”: bar_data[i][0].title(), “pct”: str(round((bar_data[i][1]/total)*100, 2)) + “%” }) else: break VALUE[‘out_data’] = out_data VALUE[‘title’] = title else: error = True error_msg = “Data model is expected to have ‘dm1’ key” else: is_data = False VALUE[‘is_data’] = is_data VALUE[‘error’] = error VALUE[‘error_msg’] = error_msg HTML Template: <style> .bar-card{ position:absolute; top: 0; left: 0; padding-top: 10px; padding-left: 15px; inset: 0; font-family: ‘Inter-Medium’, sans-serif; } .decline_heading { font-size: 15px; } .bar_table{ padding-top: 6px; width:100%; align-items: left; text-align: left; border-collapse: separate; border-spacing: 10px } .bar_table_head, .bar_distribution_data{ padding-top: 2px; } .bar_distribution{ text-align: right; padding-right: 5px; width: 10%; white-space: nowrap; overflow: hidden; text-overflow: ellipsis; } .uei_cat_bar{ border-radius: 20px; text-align: right; padding-right: 5px; }
.error_div { color: #ba3939; background: #ffe0e0; border: 1px solid #a33a3a; padding: 5px; }
.no_data_div { color: #204a8e; background: #c9ddff; border: 1px solid #4c699b; padding: 5px; text-align: center; } .bar_title { font-size: 16px; font-weight: bold; } </style> {{^error}} <div class=“bar-card”> <div class=“bar_title”> {{title}} </div> <table class=“bar_table”> {{#out_data}} <tr class=“bar_distribution_data”> <td class=“bar_distribution”>{{category}}</td> <td style=“width: 80%; background-color: #c9ddff; border-radius: 10px;”> <div style=“display: flex;”> <div class=“uei_cat_bar” style=“background-color:#4075C5 ; width:{{pct}};”> </div> <!– <div style=”float: left; padding-left: 5px; font-weight: bold;”> {{pct}} </div> –> </div> </td> <td> <div style=“float: left; padding-left: 5px; font-weight: bold;”> {{pct}} </div> </td> </tr> {{/out_data}} </table> </div> {{^is_data}} <div class=“no_data_div”> <span style=“font-weight: bold;”>No Data</span><br> </div> {{/is_data}} {{/error}} {{#error}} <div class=“error_div”> <span style=“font-weight: bold;”>Error:</span><br> {{error_msg}} </div> {{/error}}
|
Here’s an example of How to create tables in insight visualization.
Script: data = DM['Chart_ex1']day_arr = [“Sunday”, “Monday”, “Tuesday”, “Wednesday”, “Thursday”, “Friday”, “Saturday”] val_arr = [0] * 7 for item in data: val_arr[day_arr.index(item.get(“day”))] = item.get(“count”) VALUE[‘chart_label’] = day_arr VALUE[‘chart_value’] = val_arr
Template: <script type=“text/javascript” src=“/assets/Chart.js”></script> <canvas id=“myChart” width=“400” height=“400”></canvas> <script> var ctx = document.getElementById(‘myChart’).getContext(‘2d’); var myChart = new Chart(ctx, { type: ‘bar’, data: { labels: [‘Red’, ‘Blue’, ‘Yellow’, ‘Green’, ‘Purple’, ‘Orange’], datasets: [{ label: ‘# of Votes’, data: [12, 19, 3, 5, 2, 3], backgroundColor: [ ‘rgba(255, 99, 132, 0.2)’, ‘rgba(54, 162, 235, 0.2)’, ‘rgba(255, 206, 86, 0.2)’, ‘rgba(75, 192, 192, 0.2)’, ‘rgba(153, 102, 255, 0.2)’, ‘rgba(255, 159, 64, 0.2)’ ], borderColor: [ ‘rgba(255, 99, 132, 1)’, ‘rgba(54, 162, 235, 1)’, ‘rgba(255, 206, 86, 1)’, ‘rgba(75, 192, 192, 1)’, ‘rgba(153, 102, 255, 1)’, ‘rgba(255, 159, 64, 1)’ ], borderWidth: 1 }] }, options: { scales: { yAxes: [{ ticks: { beginAtZero: true } }] } } }); </script>
|
Here’s an Example of How to create tables in insight visualization:
Script: dm = DM['dm1'] no_of_rows = VAR.get(‘no_of_rows’, 5) no_of_col = VAR.get(‘no_of_col’, None) error = False error_msg = “” table_data = [] if no_of_col is None: error = True error_msg = “Varaiable ‘no_of_col’ nor found” else: column_names = [VAR[‘col’+str(i)] for i in range(1, no_of_col+1)] if len(dm) > 0: dm_keys_len = len(dm[0].keys()) if dm_keys_len >= len(column_names): for i in range(len(dm)): if i < no_of_rows: cell_data = [] for column in column_names: cell_data.append({ “cell_value”: dm[i][column] }) table_data.append({ “row”: cell_data }) else: break table_heading = [{“column_name”: column} for column in column_names] VALUE[‘table_heading’] = table_heading VALUE[‘table_data’] = table_data
HTML Template: <style> .table-container{ width: 100%; overflow-x: auto; box-shadow: 0 0 10px rgba(0,0,0,0.25); border-radius: 10px; font-family: ‘Inter-Medium’, sans-serif; } .insight_table{ width: 100%; border-collapse: collapse; table-layout: fixed; } .insight_table th, .insight_table td{ text-align: left; } .insight_table td { text-align: left; border-bottom: 1px solid #f2f2f2; color: #000000; font-size: 15px; color: #666; line-height: 1rem; font-weight: unset!important; padding-top: 10px; padding-bottom: 10px; padding-left: 20px; border-bottom: 1px solid #f2f2f2; word-wrap: break-word; } .insight_table th { font-size: 18px; color: #fff; line-height: 0.5; font-weight: unset!important; padding-top: 19px; padding-bottom: 19px; padding-left: 20px; background-color: #6c7ae0; border-bottom: 3px solid #6c7ae0; border-top: 3px solid #6c7ae0; font-size: 1.2em; height: 5px; } .insight_table tbody tr:hover { background-color: #ECECFF; } </style> <div class=“table-container”> <table class=“insight_table”> <thead> <tr> {{#table_heading}} <th>{{column_name}}</th> {{/table_heading}} </tr> </thead> <tbody> {{#table_data}} <tr> {{#row}} <td>{{cell_value}}</td> {{/row}} </tr> {{/table_data}} </tbody> </table> </div>
Output:
|
Just like HTML, scripting in this template also has 2 parts. The first is Script and the second is the Template.
The idea is to use Script which would be a Python script to store values in the VALUE dictionary or list and later use Template which would be a markdown script to access and display those values in the insight visualization.
Just like in HTML Template, to access the values from the VALUE dictionary all we have to do is put two curly braces around the key to whose value we want to access. The rest is just Markdown syntax.
Example
Script – VALUE['b'] = str(DM['a']) Template: {{b}}
Output:
|
The Ad-hoc variable type doesn’t support Timescale DB.
Let us consider the use case where you want to monitor the number of failed orders. If any order has failed, then attention is required.
Refer Data Model Chapter to create a Data Model for failed orders.
Now let us create an insight using the above-created DM.
Let’s add the following Python script to determine if an order has failed or not.
# get the DM dictfailed_order_DM = DM[‘Trukart Order Failures’] # get the data from DM failedOrderCount = failed_order_DM[‘Order Failures’][‘value’] # depending on the failed count lets provide a status if failedOrderCount == 0 : text = “No failed orders so far. Looking good.” value = “GOOD” else : text = “The number of failed orders are ” + str(failedOrderCount) value = “ATTENTION” # finally assign the values to ‘TEXT’ and ‘VALUE’ variables TEXT= text VALUE.append(value) |
Creating Headers -Script – value = DM[‘DM For Testing Throttling’][‘Memory Usage’][‘value’] VALUE[‘value’] = value Template – # Memory Usage ## Memory Usage is {{value}} Output- Changing fonts of any text- Script- value = DM[‘DM For Testing Throttling’][‘Memory Usage’][‘formattedValue’] VALUE[‘value’] = value Template for changing font- ## The Memory Usage is **{{value}}** ## The Memory Usage is ***{{value}}*** Creating a bullet point list – Script – value1 = DM[‘DM For Test Alert2’][‘Memory Usage’][‘formattedValue’] value2 = DM[‘DM For Test Alert’][‘CPU%’][‘formattedValue’] value3 = DM[‘DM For Testing Throttling’][‘Memory Usage’][‘formattedValue’] VALUE = [value1, value2, value3] Template – ## Performance # {{#.}} * {{.}} {{/.}} |
data=DM['Top CPU Utilized Server']IP=list(data)[0] VALUE=[IP] TEXT = “Out of total cpu utilized, ” + str(data[IP][‘CPU used %’][‘formattedValue’]) + ” of cpu is used by this Server” |
data1=DM['Out Bandwidth Under Utilization per Interface']data2=DM[‘In Bandwidth Under Utilization per Interface’] count=0 for k in data1: if k!=”level” and k!=”fieldName”: for k1 in data1[k]: if k1!=”level” and k1!=”fieldName”: if data1[k][k1][‘out b/w%’][‘value’]>1000: count=count+1 for k in data2: if k!=”level” and k!=”fieldName”: for k1 in data2[k]: if k1!=”level” and k1!=”fieldName”: if data2[k][k1][‘in b/w%’][‘value’]>1000: count=count+1 if count>0: TEXT=”In last 7 days at least “+str(count)+” interfaces of network devices w.r.t bandwidth are underutilized” VALUE.append(count) else: TEXT=”There were 0 underutilized network devices in last 7 days” VALUE.append(count) |
d=DM['DM for overutilized branch server insight']m_count=0 c_count=0 for key in d: if key!=”level” and key!=”fieldName”: for k2 in d[key]: if k2==”cpu%”: if d[key][‘cpu%’][‘value’]>500: c_count=c_count+1 elif k2==”mem%”: if d[key][‘mem%’][‘value’]>500: m_count=m_count+1 count=m_count+c_count VALUE.append(count) if c_count>0 and m_count>0: TEXT=str(c_count)+” server faced consistent high CPU utilization and “+str(m_count)+” server faced consistent high memory utilization in last 7 days “ elif c_count>0 and m_count==0: TEXT=str(c_count)+” server faced consistent high CPU utilization in last 7 days” elif m_count>0 and c_count==0: TEXT=str(m_count)+” server faced consistent high memory utilization in last 7 days” else: TEXT=”Great !! No server faced consistent high CPU or memory utilization in the last 7 days.” |
d=DM['DM for device down insight']val=[] count=0 for k in d: if k!=”level” and k!=”fieldName”: # if d[k][‘Branch Down’][‘value’]>1000: val.append(k) val.append(d[k][‘Branch Down’][‘value’]) count=count+d[k][‘Branch Down’][‘value’] st=” for i,v in enumerate(val): st+=str(v) if i%2==0: st+=”: “ else: st+=” “ VALUE.append(count) if count>0: TEXT=”Count of fluctuations observed for network devices in last 7 days is “+st else: TEXT=”Super !! No network device faced consistent fluctuations in last 7 days.” |
The “DMs” chosen for the visualization will be available in the DM dictionary variable. For example, if two DMs “DM_1” and “DM_2” are selected, the dict DM[‘DM_1’] will contain the details of DM_1, and DM[‘DM_2] will contain the details of DM_2.
In this example, The DM name is “Avg Page Load Time” and the metric name is “Avg Page Load Time”.The DM dict will be in the following format.
{ “Avg Page Load Time”:{ “metrics”:{ “Avg Page Load Time”:{ “threshold”:[ ], “formattedValue”:”4.63sec”, “status”:1, “metricIcon”:””, “visualization_name”:”Avg Page Load Time”, “visualization_type”:”business_metric”, “success”:True, “groupName”:”Avg Page Load Time”, “label”:”Avg Page Load Time”, “value”:4063.3, “description”:””, “color”:”#05a608″ } } } } |
The insights visualization configuration Python interface will use the above dict to create insights.
pl_DM = DM['Avg Page Load Time']page_loadtime_ms = pl_DM[‘Avg Page Load Time’][‘value’] text = “Average page loading well within 3 secs” value = “GOOD” if page_loadtime_ms > 3000 and page_loadtime_ms <= 6000: text = “Avg page loading (” + str(page_loadtime_ms) + “msec) takes less than 6 secs, but more than 3 sec” value = “FAIR” if page_loadtime_ms > 6000 and page_loadtime_ms <= 10000: text = “Avg page loading (” + str(page_loadtime_ms) + “msec) takes more than 6 secs, you may want to check influencers like network latency or processing time” value = “ATTENTION” if page_loadtime_ms > 10000: text = “Avg Page loading (” + str(page_loadtime_ms) + “msec) takes more than 10 sec, this will be bad user experience. Needs urgent resolution” value = “BAD” TEXT=text VALUE.append(value) |
With buckets configured, the only difference is the DM dict will contain the metrics structure for each sub-bucket under it. So, you can iterate over the sub-buckets and read their metric(s) value. The below example uses URL heartbeat data for each URL and determines the overall URL status by checking the number of URLs Up/Down.
The DM dict will be
{ “URL Latest Status”:{ “https://www.example.com/en-us/plan-your-investments”:{ “fieldName”:”url”, “metrics”:{ “State”:{ “formattedValue”:”Up”, “metricIcon”:””, “color”:”#05a608″, “success”:True, “value”:”Up”, “description”:””, “threshold”:[ ] } } }, “https://www.example.com/en-us/investment-solutions”:{ “fieldName”:”url”, “metrics”:{ “State”:{ “formattedValue”:”Up”, “metricIcon”:””, “color”:”#05a608″, “success”:True, “value”:”Up”, “description”:””, “threshold”:[ ] } } }, “https://www.example.com/en-us/ways-to-invest/online-portal”:{ “fieldName”:”url”, “metrics”:{ “State”:{ “formattedValue”:”Up”, “metricIcon”:””, “color”:”#05a608″, “success”:True, “value”:”Up”, “description”:””, “threshold”:[ ] } } }, “https://www.example.com/en-us/navs”:{ “fieldName”:”url”, “metrics”:{ “State”:{ “formattedValue”:”Up”, “metricIcon”:””, “color”:”#05a608″, “success”:True, “value”:”Up”, “description”:””, “threshold”:[ ] } } }, “https://www.example.com/en-us”:{ “fieldName”:”url”, “metrics”:{ “State”:{ “formattedValue”:”Up”, “metricIcon”:””, “color”:”#05a608″, “success”:True, “value”:”Up”, “description”:””, “threshold”:[ ] } } } } } |
The Insights configuration (for text template) will be like this
allurls = DM['URL Latest Status']state = “UP” statetext = “All Prelogin URLs UP” status_hash = { “Up” : 0, “Down”: 0 } totalurls = len(allurls) if totalurls < 5: state = “Partially UP” statetext = “Unable to get status for some URLs” for url in allurls: url_state = allurls[url][‘State’][‘formattedValue’] status_hash[url_state] += 1 if status_hash[“Up”] > 0 and status_hash[“Down”] > 0: state = “PARTIAL” statetext = “Some URLs Up and Some URLs Down” if status_hash[“Down”] == 5: state = “DOWN” statetext = “All Prelogin URLs Down” TEXT = statetext VALUE.append(state) |
By now, you would have understood how DM dict is structured. In case there are multiple DMs, you will have as many DM dicts as the number of DMs selected.
The below example gives the overall status of a website delivering pages to the user. It uses two DMs giving success and failure counts for each web server delivering the site. DM dicts (please note ‘Web Servers Failures Count’ dict is empty since no web servers were encountering any failures)
DM:{ “Webserver Failures Count”:{ }, “Webserver Success Count”:{ “WS3”:{ “fieldName”:”web_server”, “metrics”:{ “Num-Success”:{ “formattedValue”:”50″, “metricIcon”:””, “color”:”#05a608″, “success”:True, “value”:50, “description”:””, “threshold”:[ ] } } }, “WS4”:{ “fieldName”:”web_server”, “metrics”:{ “Num-Success”:{ “formattedValue”:”150″, “metricIcon”:””, “color”:”#05a608″, “success”:True, “value”:150, “description”:””, “threshold”:[ ] } } }, “WS1”:{ “fieldName”:”web_server”, “metrics”:{ “Num-Success”:{ “formattedValue”:”100″, “metricIcon”:””, “color”:”#05a608″, “success”:True, “value”:100, “description”:””, “threshold”:[ ] } } } } } |
Insight configuration
failws = len(DM['Webserver Failures Count'])succws = len(DM[‘Webserver Success Count’]) state = “OK” statetext = “All 4 Webservers delivering site correctly” if failws > 0 and failws < 4: state = “FAIR” statetext = “One or more Webservers had page load failures in the last hour” if failws >= 4: state = “NOT OK” statetext = “All the 4 Webservers had page load failures in the last hour” if succws == 0 and failws == 0: state = “NO DATA” statetext = “Dont have any synthetic agent data for the last hour” TEXT = statetext VALUE.append(state) |
Browse through our resources to learn how you can accelerate digital transformation within your organisation.
VuNet Systems is a next-gen visibility and analytics company that uses full-stack AI & Big Data analytics to accelerate digital transformation within an organisation. We provide deep observability into business journeys to reduce failures and enhance overall customer experience.
