ML KPIs and Automated Insights > User Experience Index (UEI) and Operational Predictive Index (OPI)
In the world of services, like Internet Banking or specific journeys within a service (e.g., Fund Transfer on a Mobile App), it’s crucial to measure real-time service quality from an external user’s perspective.
Imagine a fund transfer from a Mobile App. Users expect a swift transaction, ideally within a few seconds. If it takes longer, frustration sets in. Beyond 10 seconds, the user experience sharply declines.
The UEI simplifies this experience into a number from 0 to 10—0 being a terrible user experience, and 10 being outstanding. A UEI of 8.5 or higher is generally considered good. The system can automatically set optimal baselines for a particular service, triggering alerts when user experience falls below these standards.
| Model | This is an unsupervised statistical model that gives you a score between 0 and 10. It considers factors like transaction outcome, turnaround time, and psychological parameters for response time thresholds. |
| Hyper Parameters | The system comes with default values for response time psychology, and you can fine-tune them. Also, you can specify the dimensions or categories for which you want to calculate the UEI separately. Easy adjustments for a better user experience! |
| Deployment | An unsupervised technique that can be operational from day 1. |
| Scale | Lightweight model and can scale to thousands of signals with minimal resource consumption |
| Benefits | Ability to track user experience for any system, service, or journey in a standardized way. Provides a way to detect user-impacting problems quickly and consistently. Provides transparent visibility into transaction performance to the leadership team, business teams, and partners. |
| Example Use Cases | The track user experience of various user journeys like Fund Transfer, Balance Enquiry, Bill payment, Account Registration, etc in real-time and detect deterioration in experience in real-time Track user experience provided by partners or third parties like NPCI, Lending solution providers, etc, and detect a drop in quality of service in real-time Track the quality of service provided by important components/services within the bank. For example, Core Banking Systems, API Gateways, etc. |
Scaled UEI is shown on a storyboard detailing a particular customer journey for an instant payment system like UPI. Due to higher technical declines, the UEI is fair, which means customer experience is below the satisfactory level
Low UEI on a particular day due to an extremely high rate of technical declines.
Anticipating and detecting potential problems before they impact users is crucial. For instance, a gradual increase in a message queue could lead to transaction timeouts. Similarly, if the garbage collection of a Java process becomes more frequent without reclaiming enough memory, it might escalate to a service-level problem.
Identifying these potential problems in advance is key to minimizing outages. VuNet’s Operational Predictive Index (OPI) gauges the likelihood of an individual signal or component encountering issues that could impact users in the future. The OPI score, ranging from 0 to 10, indicates the probability of a signal or component causing a problem—0 being highly likely and 10 suggesting a very low chance.
An OPI score of 7 or below signals a high probability of potential future failures. This scoring extends to individual components like databases, web servers, etc., and even at the journey level, providing a predictive index on the probability of a user experience drop.
VuNet’s patented OPI approach not only predicts potential user-impacting problems but also identifies possible causes, offering a unique mechanism to proactively avoid incidents.
OPI forecasting a potential user-impacting problem if the database problem is not rectified
The forecasted problem manifests as a user-impacting incident
| Model | DeepAR-based Auto regressive, Deep learning (Recurrent Neural Network) model looks at the past behavior of the signal and tries to forecast the future probability range of the signal and from there calculates a probability of the score of this signal turning into a bad state. | |
| Hyper Parameters | Deep learning network parameters including Context length, epoch, dropout rate, prediction length, and learning rate | |
| Deployment | Supervised technique that requires a minimum of 2 months past data. Can cold start without any user feedback and the model become better as feedback starts flowing in? The integrated MLOPs layer of the platform takes care of periodic retraining, model versioning, and deployment in an automated way. | |
| Scale | The heavier model compared to unsupervised techniques with higher resources required for training and minimal resources required for inference. Can scale to operate on thousands of signals that are part of a journey. | |
| Benefits | Detection probability of potential problems in advance and taking corrective action to avoid impact on users Provides a standard scoring methodology across signals and components to understand the probability of errors in future | |
| Example Use Cases | ML-Driven Use Cases | Automation |
| Predicting potential application-level slowness and failures in advance by looking at the GC status and statistics | Depending on the level of memory usage and reclamation rate, a playbook is initiated to either restart the respective service or to allocate additional memory/resources to the service. | |
| Predicting the need for an application service switchover to DR based on OPI score for an important component like a database. | Initiate re-routing of requests to DB or requests from users to alternate sites. | |
| Using OPI scores for overall network experience – using network and TCP level signals corresponding to connection resets, packet drops, retransmissions, etc – to predict possible higher latency and impact on transaction turnaround time. | Typical corrective action on this would be connection restarts and interface restarts. Since this may involve Service Providers, automated or semi-automated notifications to relevant stakeholders are also initiated. | |
Browse through our resources to learn how you can accelerate digital transformation within your organisation.
VuNet’s Business-Centric Observability platform, vuSmartMaps™ seamlessly links IT performance to business metrics and business journey performance. It empowers SRE and IT Ops teams to improve service success rates and transaction response times, while simultaneously providing business teams with critical, real-time insights. This enables faster incident detection and response.