How Domain-Centric Approach Enable Better RCA and ML Insights

Domain Expertise

Domain knowledge is a critical factor for successfully solving technical problems and delivering solutions that align with the goals and requirements of the business or industry.

• Understanding of Business Context: IT solutions are developed to facilitate the functional and operational requirements of various business groups. Having domain knowledge allows IT professionals to understand the business context, goals, and challenges. This understanding is essential for aligning technical solutions with business objectives. For instance, an application team focuses solely on technical aspects of the application, such as designing architecture, advanced tech stack, performance improvements, or adding new technical features. However, these efforts might not always address the business objectives or create an impact on user engagement/experience. 
• Effective Communication: IT professionals often need to communicate with stakeholders from various departments, including non-technical ones. Having domain knowledge enables effective communication, as IT professionals can speak the language of the business, making it easier to collaborate and gather requirements.
• Problem Identification and Analysis: Domain knowledge helps in identifying and analyzing problems more accurately. IT professionals with a deep understanding of the domain can recognize the root causes of issues, making it easier to design effective and efficient solutions.
• Customization of Solutions: Every business domain has its unique characteristics and requirements. With domain knowledge, IT professionals can customize solutions to fit the specific needs of the industry they are working in. This ensures that IT solutions are not generic but tailored to the unique challenges of the domain.
• Quick and Informed Decision-Making: When faced with technical challenges or decisions, having domain knowledge allows IT professionals to make quicker and more informed decisions. They can assess the impact of technical choices on business operations, which is crucial for maintaining efficiency and minimizing downtime.
• Innovation and Continuous Improvement: Domain knowledge is essential for fostering innovation within a specific industry. Understanding the nuances of the business allows IT professionals to propose and implement innovative solutions that can drive efficiency, productivity, and competitive advantage.
• Problem-Solving Efficiency: Domain knowledge streamlines the problem-solving process. IT professionals who understand the domain can more efficiently navigate through complex issues, leading to faster and more effective resolutions.
• Risk Mitigation: In any application, understanding the domain helps in anticipating and mitigating potential risks. IT professionals can proactively address challenges related to compliance, security, or specific industry regulations, reducing the likelihood of problems arising during or after implementation.

Machine Learning (ML)

Machine Learning offers numerous benefits in solving technical problems, it’s important to note that its successful implementation often requires a solid understanding of the problem domain, proper data management, and ongoing monitoring and refinement of the ML models. Combining ML with domain knowledge results in more effective and contextually relevant solutions in the realm of IT.

• Predictive Analytics: ML algorithms can analyze historical data to identify patterns and trends, enabling the prediction of future events. This is valuable in IT for predicting system failures, identifying potential security threats, and optimizing resource allocation.
• Anomaly Detection: The accuracy of ML models excels at detecting anomalies on larger historical time-series datasets.. anomaly detection can be leveraged to identify unusual patterns or abnormal behaviors in both businesses (ex. surge/dip in transaction volume, fraudulent activities, etc.) and IT (ex. High Transaction failures, system utilization, or performance issues).
• Automated Decision-Making: ML algorithms can be used to automate decision-making processes. In IT, this can include automated incident response, resource allocation, or even decision-making in areas like network optimization.
• Natural Language Processing (NLP): NLP, a subset of ML, can be used to analyze and understand human language. Datasets like user feedback or experience shared in social media platforms, and knowledge repositories would be ideal for NLP use cases.  For instance, during a critical incident, the system consolidates and processes the data from multiple data sources (resolutions from similar incidents in the past) and precisely highlights the impacted user groups, root cause, and recommendation which can easily be understood by a layman to resolve the incident without any further analysis or investigation. This will eventually help in reducing the MTTR and improving user experience.
• Recommendation Systems: ML-powered recommendation systems are widely used in IT, especially in e-commerce and content delivery platforms. These systems analyze user behavior and preferences to suggest products, services, or content, enhancing user experience.
• Optimization of IT Operations: ML can be employed to optimize various IT operations, such as network management, server provisioning, and workload distribution. This helps in improving efficiency, reducing downtime, and enhancing overall system performance.
• Fraud Detection: ML algorithms can identify patterns indicative of fraudulent activities in financial transactions, user accounts, or other IT-related processes. This is crucial for maintaining the integrity and security of IT systems.
• Personalization: ML enables the personalization of user experiences by analyzing user behavior and preferences. In IT, this can lead to personalized interfaces, content recommendations, and tailored services.
• Continuous Learning and Adaptation: ML models can continuously learn from new data, adapting to changing circumstances and improving their performance over time. This adaptability is particularly useful in dynamic IT environments.

vuCoreML Domain-centric approach

A domain-centric approach is pivotal in Root Cause Analysis (RCA) and Machine Learning (ML) insights for a few key reasons:

• Contextual Understanding: In RCA or ML, understanding the intricacies and nuances of a specific domain is crucial. A domain-centric approach involves deep knowledge of the field or industry in question. This contextual understanding helps in identifying relevant variables, features, or factors that might impact the outcomes being studied. For instance, in any application domain knowledge helps discern between normal and anomalous data patterns. Digital Payment systems have a complex architecture where multiple application touchpoints and infra components are involved, it’s very essential to understand the business journey workflow and its topology to build any observability platform.
• Feature Engineering: Domain knowledge is often utilized in feature engineering in ML. This involves selecting, extracting, or creating the right features from the data that are most relevant to the problem. A domain-centric approach helps in identifying these key business-critical features that might not be obvious from the data alone but are crucial for accurate predictive models. In observability, heterogeneous datasets like metrics, metadata, traces, and logs are leveraged in assessing the internal states of a system’s availability health, or performance. However, not all these golden signals can be categorized as problematic golden signals which might immediately cause an impact on the system’s health or performance. So it is essential to have the domain knowledge to classify the golden signals as lead indicators (which reflect the direct impact on the end-user experience)  and problematic signals (which directly impact the stable state of lead indicators).
• Interpretability and Explainability:In both RCA and ML, interpretability and explainability are vital. Understanding the underlying reasons behind an issue or the workings of an ML model is easier with domain knowledge. It helps in translating complex insights into actionable strategies or explanations that stakeholders can understand and trust.  In general, there are multiple teams involved in identifying the root cause, however, it is limited to their respective domain expertise and experience. In this case, the robust ML system which precisely highlights the impacted user groups, root cause, and recommendation.
• Reducing Noise and False Positives: In complex datasets, irrelevant or noisy data can hinder RCA or lead to inaccurate ML predictions. Domain expertise assists in filtering out noise and reducing false positives by focusing on relevant patterns and meaningful signals within the data. In the domain-centric approach, there is a drastic reduction in noise and false positives, as the input dataset has been cleansed and enriched  with meaningful context from a journey perspective.
• Effective Communication: Communication between the business team and data scientists becomes smoother and more effective with a domain-centric approach. Once the data science team gets a better understanding of business objectives,  various business functions, and operations specific to a given domain they can easily connect with the business team, as they speak in the business language rather than in the technical knowledge which is the case with the IT team. This collaboration helps in framing hypotheses, validating results, and ensuring that the insights derived are practically applicable within the specific domain.

In essence, a domain-centric approach acts as a guiding light, providing a structured and informed way to navigate through data complexities, aiding in more accurate analysis, RCA, and development of ML models that are not just accurate but also practically applicable within a specific domain.

Example:

Digital Payment Systems have complex architecture designed to facilitate real-time, interbank electronic funds transfers in India. While many online/real-time payment systems have been successful in revolutionizing digital payments, they also come with technical challenges due to their intricate design and the multitude of stakeholders involved.

Solving technical problems in any digital payment system with complex architecture requires a multidisciplinary approach involving expertise in software development, security, compliance, and user experience design along with domain understanding.

• Contextual Understanding: In Digital payment systems the transactions can broadly be classified as  Financial and Non- Financial. Financial transactions typically involve debit or credit of money and these transactions will interact with the core banking system, whereas non-financial transactions will not hit the core banking system. Contextual understanding of the transaction flow, and the list of touchpoints involved for each type of financial and non-financial transaction journey are critical inputs for the  ML model to deliver better results.
• Feature Engineering: When users are facing any issues in any digital payment systems due to technical reasons. There will be many alerts related to multiple signals/metrics indicating there is a problem or abnormality in the system. However, all the alerts won’t be impacting the end-users or relevant to the root cause of the issue. Lead indicators like Failures, Volume, and TAT will notify whether the current issue is impacting the end-users or not.  The underlying cause could be due to any combinations of signal/metrics related to a network or server or storage or database performance metrics, like “db stuck threads”, “connection timeouts”,  “server/device down” etc.
• Interpretability and Explainability: In any complex architecture, there are multiple components and technology stacks like Web Applications, Servers, Middleware, Networks, databases, and API calls to external systems involved. The vuCoreML considers the business context, topology, and interdependencies between the touchpoints/applications in the given digital payment system and would precisely explain the exact point of failure for a transaction.
• Reducing Noise and False Positives: There are N number of lead indicators and problematic signals, but all the signal behavior is not the same. Our system can understand whether a given signal is bounded or unbounded and uses appropriate ML models which helps in the reduction of false positives to a great extent.
• Effective Communication: For instance, there is high failure in an application. These failures can be due to technical or business reasons. Raising an anonymous alert for high failure may create unnecessary panic in the Business/Application/Support team. Instead by enriching/enhancing the data with appropriate business context to provide exact details around failures like whether any specific type of transactions or all transactions are failing along the list of impacted touchpoints, components, golden signals, and finally highlighting the probable root cause and recommendation.