Use-Cases: Data Streams and Enrichment >
In the applications of observability, the seamless integration of ContextStreams and data enrichments are integral components for transforming raw data into insightful and contextual information. This comprehensive solution guide delves into the important role played by these elements, discussing how they synchronize within the framework of vuSmartMaps™. As we navigate through the diverse use cases, the tangible impact and significance of ContextStreams and data enrichment in achieving business observability become vividly apparent.
In financial transactions, sessions often involve multiple logs that need to be combined and merged for a holistic view. Using session plugins, the platform seamlessly handles these intricacies, allowing the aggregation of transaction logs—starting and ending—providing a consolidated and insightful perspective for observability.
Scenario: Consider the case of Internet Banking Mobile Banking (IBMB), where various touchpoints, including web servers, IBMB Server, Financial Risk Management (FRM) servers, Core Banking System (CBS) servers, IBMB database, IFSC, and IMPS Services, play crucial roles in transaction journeys. The complex interconnections among these touchpoints make it challenging to detect the root cause of failures for specific transactions. These touchpoints generate diverse logs, resulting in a complex set that is difficult to interpret. VuSmartMaps, acting as a logs observability platform, plays a pivotal role in addressing these challenges.
This complex set of interconnections can be visualized below with a Unified Transaction Map:
Let’s examine a scenario where a user initiates a current balance inquiry through IBMB, involving four key steps:
For each transaction with a unique ID, logs are generated, indicating the status of each of the four steps. The sample logs below:
Raw Logs:
2024-01-11 22:50:22.732+0000 [http-nio-18080-exec-9] INFO i.controllers.api.AccountController - msg [Request to IBMB for getting current balance] ReqData [requestUUID=EXWF03515187, requestId=WNS00876319, requestTime=04:20:22.732272, custID=LZB94303, mobileNumber=0502942510, accountNumber=2147483651] metaData [txnType=BalanceInquiry, txnCategory=Non-Financial, URL=http://172.16.68.168:18080/ibmb/accounts/2147483651/balance] spanContext [traceId=b5f7fad589ebb657d9560f6b6c2d6ffd, spanId=4e6755d605895bbd] 2024-01-11 22:50:22.732+0000 [http-nio-18080-exec-9] INFO ibmb.service.integration.CbsService - msg [Request to CBS for getting current balance] ReqData [requestUUID=EXWF03515187, requestId=WNS00876319, requestTime=04:20:22.732272, custID=LZB94303, mobileNumber=0502942510, accountNumber=2147483651] metaData [txnType=BalanceInquiry, txnCategory=Non-Financial, URL=http://172.16.68.159:7001/cbs/api/v1.0/accounts/balance] spanContext [traceId=b5f7fad589ebb657d9560f6b6c2d6ffd, spanId=a91f24dec1bc29ca] 2024-01-11 22:50:22.753+0000 [http-nio-18080-exec-9] INFO ibmb.service.integration.CbsService - msg [Response from CBS for getting current balance] RespData [requestUUID=EXWF03515187, requestId=WNS00876319, requestTime=04:20:22.732272, custID=LZB94303, mobileNumber=0502942510, accountNumber=2147483651, currentBalance=0.0, rrn=1079554643] statusData [status=Success, respCategory=Approved, respDesc=Success, respCode=0] metaData [txnType=BalanceInquiry, txnCategory=Non-Financial, URL=http://172.16.68.159:7001/cbs/api/v1.0/accounts/balance] spanContext [traceId=b5f7fad589ebb657d9560f6b6c2d6ffd, spanId=a91f24dec1bc29ca] 2024-01-11 22:50:22.753+0000 [http-nio-18080-exec-9] INFO i.controllers.api.AccountController - msg [Final response from IBMB for getting current balance] RespData [requestUUID=EXWF03515187, requestId=WNS00876319, requestTime=04:20:22.732272, custID=LZB94303, mobileNumber=0502942510, accountNumber=2147483651, currentBalance=0.0, rrn=1079554643] statusData [status=Success, respCategory=Approved, respDesc=Success, respCode=0] metaData [txnType=BalanceInquiry, txnCategory=Non-Financial, URL=http://172.16.68.168:18080/ibmb/accounts/2147483651/balance] spanContext [traceId=b5f7fad589ebb657d9560f6b6c2d6ffd, spanId=4e6755d605895bbd]
In financial transactions, sessions often involve multiple logs that need to be combined and merged for a holistic view. Using session plugins, the platform seamlessly handles these intricacies, allowing the aggregation of transaction logs—starting and ending—providing a consolidated and insightful perspective for observability.
Plugin:
{ "aggregate_fields": [ "traceId" ], "logname_field": "leg", "instance_name": "Current Balance", "session_timeout_in_sec": 100, "grace_duration_in_sec": 100, "field_configs": { "Request to IBMB for getting current balance": { "set_fields_latest": { "RequestUUID": "RequestUUID", "RequestId": "RequestId", "RequestTime": "RequestTime", "CustID": "CustID", "MobileNumber": "MobileNumber", "AccountNumber": "AccountNumber", "TxnType": "TxnType", "TxnCategory": "TxnCategory" } }, "Final response from IBMB for getting current balance": { "set_fields_latest": { "RootCause": "RootCause", "TxnStatus": "TxnStatus", "RRN": "RRN", "RespCode": "RespCode", "RespCategory": "RespCategory", "RespDesc": "RespDesc", "CurrentBalance": "CurrentBalance" } } }, "end_identification_code": "(event.has('leg') && event.get('leg').asText() == 'Final response from IBMB for getting current balance');", "event_list_additional_fields": [], "enable_logstash_based_output": "false", "evaluate_time_diff": { "cbs_duration": [ "Request to CBS for getting current balance", "Response from CBS for getting current balance" ] } }
This configuration defines a session plugin for handling transaction logs related to acquiring the current balance from a bank’s service. It orchestrates the grouping of logs based on the “traceId” field, categorizes events using the “leg” field, and establishes a session named “Current Balance”.
The configuration specifies expected fields for two crucial events in the transaction process, defines conditions for session termination, and calculates the duration between specific events to assess the time taken for the transaction. In essence, this session plugin streamlines the observability of current balance transactions by organizing, categorizing, and analyzing logs within a defined context.
The consolidated log after parsing with the session plugin will look like the following, where logs from multiple touchpoints are stitched together:
Parsed Log:
{"session_status":"COMPLETED","@timestamp":"2024-01-11T22:50:22.591Z","traceId":"19c855c26010e409920e8bdec9f3abb4","TxnStatus":"Success","RRN":"1000338325","RespCode":"0","RespCategory":"Approved","RespDesc":"Success","CurrentBalance":"0.0","RequestUUID":"OLKB21914196","RequestId":"VFM20872419","RequestTime":"04:20:22.582960","CustID":"HAG77074","MobileNumber":"7243874141","AccountNumber":"2147483650","TxnType":"BalanceInquiry","TxnCategory":"Non-Financial","cbs_duration":8,"Duration":8,"scaled_uei":10,"Result":"Success","RootCause":"NotApplicable","ibmb_duration":0,"JourneyName":"IBMB","AppName":"IBMB"}
After parsing with the session plugin, the resulting consolidated log provides a comprehensive view of the entire transaction. This log includes critical information such as trace identifiers, transaction status, response details, and specific durations at various stages. Viewable in the Explore section of the platform, this consolidated log serves as valuable input for various consumption layers, including visualizations, storyboards, alerts, and reports, enriching the overall observability of financial transactions.
This use case illustrates how session plugins effectively handle the intricacies of transaction logs, ensuring a seamless and comprehensive understanding of financial transaction journeys.
Scenario: Introducing a Key-Value (KV) logging scenario, where logs follow the KV format with “:” as the key-value separator and “,” as the field separator. Employing a key-value (KV) plugin, the platform processes these KV logs, extracting valuable information and enabling effective financial transaction observability.
Consider the processed log in the above case as below, which has multiple fields in key-value pairs to be extracted through KV parser.
Raw Log:
{"session_status":"COMPLETED","@timestamp":"2024-01-11T22:50:22.591Z","traceId":"19c855c26010e409920e8bdec9f3abb4","TxnStatus":"Success","RRN":"1000338325","RespCode":"0","RespCategory":"Approved","RespDesc":"Success","CurrentBalance":"0.0","RequestUUID":"OLKB21914196","RequestId":"VFM20872419","RequestTime":"04:20:22.582960","CustID":"HAG77074","MobileNumber":"7243874141","AccountNumber":"2147483650","TxnType":"BalanceInquiry","TxnCategory":"Non-Financial","cbs_duration":8,"Duration":8,"scaled_uei":10,"Result":"Success","RootCause":"NotApplicable","ibmb_duration":0,"JourneyName":"IBMB","AppName":"IBMB"}
The following is the KV plugin to be used for this purpose:
Plugin:
{ "target": "body", "source": "body", "field_split": ", ", "value_split": ":" }
The provided parser configuration is designed to process log data containing key-value pairs within the log body. It uses a simple logic where key-value pairs are separated by a comma and space (“, “), and each pair is split by an equal sign (“:”). This transformation allows the extraction of structured information from unstructured log data, making it more accessible for analysis within the logging and observability platform.
Scenario: In the context of financial transactions, consider XML-based logs from a Bill Payment Service (BBPS). vuSmartMaps platform seamlessly converts these XML logs into JSON format, initiating the regular flow within the data pipeline. This conversion enhances the data’s interpretability and sets the stage for comprehensive financial transaction observability.
The Issue:
Emma found interpreting XML logs to be a daunting task, hindering her ability to unravel the intricacies of the billing processes. Recognizing the need for a solution, she sought to streamline the logging mechanism to make data more accessible and insightful.
Enter vuSmartMaps:
Emma’s quest for a resolution led her to vuSmartMaps, a platform that promised to transform XML logs into a more user-friendly JSON format. This could simplify log structures, align with industry standards, and seamlessly integrate into their data pipeline.
The Transformation:
Upon integrating vuSmartMaps, Emma witnessed a transformative shift. The XML logs, once a complex set of information, are seamlessly converted to JSON, offering a clearer and standardized view. This not only simplified her data interpretation but also aligned the logs with industry standards, ensuring compatibility and collaboration with other systems.
The Impact:
With this solution, Emma could now efficiently query and analyze financial transaction data. The JSON format empowered her to identify patterns, anomalies, and trends effectively. The streamlined process not only alleviated Emma’s challenges but also marked a significant step forward in their strategy to optimize financial transaction observability.
Summary:
In Emma’s journey, vuSmartMaps emerged as a transformative ally, simplifying the way she approached financial transaction logs. The user-friendly JSON format makes her job easier and also reflects the commitment to continuous improvement and innovation in their billing services. Emma’s story is a testament to the power of user-focused solutions in overcoming operational challenges.
Scenario: In the logs observability, the need to extract meaningful information from encoded data is a common challenge. The Data Enrichment feature within vuSmartMaps addresses this challenge by allowing users to create Enrichment Tables, mapping key identifiers to corresponding descriptive values. This facilitates a standardized approach to enriching data within streaming environments, empowering users to seamlessly translate coded information into actionable insights. The ensuing user story exemplifies how Data Enrichment transforms raw data into comprehensible and valuable intelligence, enhancing the overall observability and diagnostic capabilities across diverse applications and systems.
The Challenge:
In the complex Apache server logs, Chris identified the need to convert Response Codes into meaningful Response Information. This conversion is essential for quick issue identification and debugging. To address this challenge, Chris utilizes the Data Enrichment feature within vuSmartMaps.
The Solution:
To streamline the process, Chris creates an Enrichment Table within vuSmartMaps, mapping Response Codes to their respective Response Information. This table becomes a reference point for translating codes into meaningful insights. The fields within this table are organized systematically to ensure a standardized approach to data enrichment.
Enrichment Table Fields:
Utilizing Enrichment in Data Stream:
Chris seamlessly integrates the Enrichment Table into the data stream using the vuSmartMaps enrichment plugin. This integration ensures that, as the data flows through the customer’s environment, the Enrichment process smoothly transforms Response Codes into descriptive Response Information.
Impact:
The enrichment of Response Codes significantly enhances Chris’s ability to interpret server status directly from the logs. With this standardized approach, decoding Response Codes becomes an automated and efficient process, allowing for quicker issue resolution and proactive debugging.
Summary:
In Chris’s observability journey, the use of Data Enrichment within vuSmartMaps emerges as a crucial platform. The systematic mapping of Response Codes to Response Information simplifies the interpretation of Apache server logs. This user-focused solution not only streamlines the observability process but also reflects the platform’s commitment to providing practical and efficient features for enhanced server observability.
In banking applications, the efficient orchestration and observability of transactions play an important role in ensuring a seamless and secure customer experience. Data streams, exemplified through use cases like the core banking system’s role in transaction processing, the integrity maintenance in credit card applications, and the efficient orchestration of IVR flows, form a critical backbone. The convergence of raw logs and enriched data, facilitated by advanced platforms like vuSmartMaps, underscores the industry’s commitment to innovation. These data stream use cases not only streamline the observability process but also enhance overall diagnostic capabilities, reflecting a broader trend in leveraging technology for deep observability across diverse banking operations.
Transaction Journey:
When a credit or debit transaction is initiated, be it through a physical card at a point-of-sale terminal or an online/mobile platform, the Core Banking System ensures the proper formatting and secure transmission of transaction details to the core banking system. This core system serves as the backbone of the bank’s operations.
Extending our financial transaction observability, in this use case logs are arriving in the ISO-8583 format. Leveraging a parser, these ISO-8583 logs undergo conversion to JSON, aligning them with the standard data format. Following this transformation, the data follows the routine flow within the platform’s data pipeline, ensuring smooth and consistent observability. Parsing ISO-8583 logs proved to be a stumbling block for Mike, creating friction in the seamless flow of financial data. With the implementation of the vuSmartMaps Log Parser, Mike witnessed a remarkable shift. ISO-8583 logs, initially a complex challenge, seamlessly transformed into JSON, adhering to industry standards.
Transaction Processing and Response:
The core banking system meticulously processes debit or credit requests, verifies customer account details and ensures sufficient funds for debit transactions. Subsequently, it sends a response back to the originating channel, indicating the success or any issues with the transaction.
Logs and Enrichment:
In transaction observability, the raw log data showcases the intricate details of each transaction. To enhance interpretability, Mike utilizes vuSmartMaps to enrich the logs. By creating a specific data adapter.
The Impact:
The hyper-configured data adapter streamlines the observability process, enabling Mike to effortlessly interpret transaction logs. With a standardized approach, decoding raw data becomes an automated and efficient process, enhancing overall observability and diagnostic capabilities.
Summary:
Mike’s user story underscores the power of core banking systems in orchestrating seamless transactions, coupled with vuSmartMaps’ Data streaming feature for comprehensive log interpretation. This user-focused solution exemplifies the platform’s commitment to efficiency and innovation in the observability of critical banking operations.
Meet Jennifer, a vigilant tech professional responsible for overseeing the integrity of credit card applications in a bank, particularly within the credit card journey. One critical touchpoint in this journey is designed to observe users’ status concerning Upgrade offers.
Functionality:
At this specific touchpoint, a crucial check is implemented to determine if a user currently holds an Upgrade offer. If an existing Upgrade offer is found, the system restricts the user from submitting another application for the same offer. This preventive measure is in place to avoid users reapplying for identical Upgrade offers while an active one is already in progress.
Transaction Flow:
When a user initiates an application or an upgrade, the request is processed through various channels, ensuring the integrity of the Upgrade offers. A request is sent, blocking the offer with specific details like AccountNo, ProductCode, ReferenceNo, Source, and more.
Response Handling:
Upon processing, the system generates a response confirming the successful availing of the offer or providing relevant feedback. This response is crucial in determining the status and success of the application.
Data Stream Processing:
In this observability use case, raw logs from these transactions are processed and enriched using vuSmartMaps. The Data Stream Use Case ensures seamless observability of the credit card application journey, allowing Jennifer to interpret the journey effortlessly.
Summary:
Jennifer’s user story highlights the importance of observability in maintaining offer integrity within credit card applications. The integration of vuSmartMaps for data stream processing exemplifies the platform’s commitment to providing a user-focused solution for efficient observability.
Meet Emily, a dedicated customer service representative facing the complex task of analyzing and resolving customer queries within the complex web of Retail Interactive Voice Response (IVR) Flow. The banking IVR, with a large number of options like PIN changes, new PIN generation, balance inquiries, and credit card statement views, presented a complex landscape. The siloed observability tools proved inadequate for tracking and analyzing the diverse journeys customers could embark upon, leading to inefficiencies in issue resolution and impacting customer satisfaction.
The Issue:
Recognizing the need for a solution that could seamlessly navigate the complex Banking IVR, Emily turned to vuSmartMaps. Known for its real-time visibility and actionable insights, vuSmartMaps seemed tailor-made for Emily’s challenges. Her goal was straightforward – enhance the observability of the Retail IVR Flow, identify bottlenecks across various customer journeys, and streamline issue resolution for a seamless and satisfactory customer experience.
vuSmartMaps in Action: ContextStreams and Parsers:
Utilizing a combination of simple grok, mutate, translate, and configured-session plugins, vuSmartMaps’ ContextStreams played a pivotal role in addressing Emily’s challenges. These streams streamlined data, allowing the support team to identify various metrics specific to different banking IVR journeys. From tracking the prompts played during calls, user-selected options, and call regions to discerning service providers and understanding call end reasons, vuSmartMaps provided granular insights.
The Advantages:
Summary:
vuSmartMaps, with its ContextStreams and advanced parsers, emerged as a transformative solution for Emily. It not only navigated the complexities of banking IVR but also elevated the efficiency of observability and issue resolution. The platform’s ability to provide detailed insights, swift issue resolution, and proactive alerts significantly contributed to Emily’s success in delivering exceptional customer service.
vuSmartMaps leads the charge in observability, utilizing dynamic ContextStreams and robust data enrichment to process data from start to finish seamlessly. What sets this platform apart is its capability of organizing raw data, offering practical insights that enhance user experiences and operational efficiency. With a dedicated commitment to performance, scalability, and real-time processing, vuSmartMaps emerges as a game-changer. It tackles challenges through a scalable structure, ensuring top-notch data quality and resource efficiency. vuSmartMaps unlocks the full power of observability, establishing it as a go-to solution for unlocking invaluable insights and ensuring organizational success.
Browse through our resources to learn how you can accelerate digital transformation within your organisation.
VuNet’s Business-Centric Observability platform, vuSmartMaps™ seamlessly links IT performance to business metrics and business journey performance. It empowers SRE and IT Ops teams to improve service success rates and transaction response times, while simultaneously providing business teams with critical, real-time insights. This enables faster incident detection and response.