9.1 Users Management and RBAC

9.2 Authentication and Security

9.3 Data Management and Data Model Handling 

9.3.1 Storage

9.3.2 Retention

9.3.3 Export/Import 

Resource Management

Import Data

9.3.4 Working with Data Model
9.3.5 Data Extraction

9.4 Control Center

9.4.1 License Entitlements

9.5 Platform Settings

9.5.1 Definitions

9.5.2 Preferences

9.5.3 About

Use-Cases: Data Streams and Enrichment >

Use-Cases: Data Streams and Enrichment

Introduction

In the applications of observability, the seamless integration of ContextStreams and data enrichments are integral components for transforming raw data into insightful and contextual information. This comprehensive solution guide delves into the important role played by these elements, discussing how they synchronize within the framework of vuSmartMaps™. As we navigate through the diverse use cases, the tangible impact and significance of ContextStreams and data enrichment in achieving business observability become vividly apparent. 

Use Case 1: Streamlining Transaction Logs with Session Plugins

In financial transactions, sessions often involve multiple logs that need to be combined and merged for a holistic view. Using session plugins, the platform seamlessly handles these intricacies, allowing the aggregation of transaction logs—starting and ending—providing a consolidated and insightful perspective for observability. Scenario: Consider the case of Internet Banking Mobile Banking (IBMB), where various touchpoints, including web servers, IBMB Server, Financial Risk Management (FRM) servers, Core Banking System (CBS) servers, IBMB database, IFSC, and IMPS Services, play crucial roles in transaction journeys. The complex interconnections among these touchpoints make it challenging to detect the root cause of failures for specific transactions. These touchpoints generate diverse logs, resulting in a complex set that is difficult to interpret. VuSmartMaps, acting as a logs observability platform, plays a pivotal role in addressing these challenges. This complex set of interconnections can be visualized below with a Unified Transaction Map: Let’s examine a scenario where a user initiates a current balance inquiry through IBMB, involving four key steps:
  1. Request to IBMB for getting current balance
  2. Request to CBS for getting current balance
  3. Response from CBS for getting current balance
  4. Final response from IBMB for getting current balance
For each transaction with a unique ID, logs are generated, indicating the status of each of the four steps. The sample logs below:

Raw Logs:

2024-01-11 22:50:22.732+0000 [http-nio-18080-exec-9] INFO  i.controllers.api.AccountController - msg [Request to IBMB for getting current balance] ReqData [requestUUID=EXWF03515187, requestId=WNS00876319, requestTime=04:20:22.732272, custID=LZB94303, mobileNumber=0502942510, accountNumber=2147483651] metaData [txnType=BalanceInquiry, txnCategory=Non-Financial, URL=http://172.16.68.168:18080/ibmb/accounts/2147483651/balance] spanContext [traceId=b5f7fad589ebb657d9560f6b6c2d6ffd, spanId=4e6755d605895bbd]
2024-01-11 22:50:22.732+0000 [http-nio-18080-exec-9] INFO  ibmb.service.integration.CbsService - msg [Request to CBS for getting current balance] ReqData [requestUUID=EXWF03515187, requestId=WNS00876319, requestTime=04:20:22.732272, custID=LZB94303, mobileNumber=0502942510, accountNumber=2147483651] metaData [txnType=BalanceInquiry, txnCategory=Non-Financial, URL=http://172.16.68.159:7001/cbs/api/v1.0/accounts/balance] spanContext [traceId=b5f7fad589ebb657d9560f6b6c2d6ffd, spanId=a91f24dec1bc29ca]
2024-01-11 22:50:22.753+0000 [http-nio-18080-exec-9] INFO  ibmb.service.integration.CbsService - msg [Response from CBS for getting current balance] RespData [requestUUID=EXWF03515187, requestId=WNS00876319, requestTime=04:20:22.732272, custID=LZB94303, mobileNumber=0502942510, accountNumber=2147483651, currentBalance=0.0, rrn=1079554643] statusData [status=Success, respCategory=Approved, respDesc=Success, respCode=0] metaData [txnType=BalanceInquiry, txnCategory=Non-Financial, URL=http://172.16.68.159:7001/cbs/api/v1.0/accounts/balance] spanContext [traceId=b5f7fad589ebb657d9560f6b6c2d6ffd, spanId=a91f24dec1bc29ca]
2024-01-11 22:50:22.753+0000 [http-nio-18080-exec-9] INFO  i.controllers.api.AccountController - msg [Final response from IBMB for getting current balance] RespData [requestUUID=EXWF03515187, requestId=WNS00876319, requestTime=04:20:22.732272, custID=LZB94303, mobileNumber=0502942510, accountNumber=2147483651, currentBalance=0.0, rrn=1079554643] statusData [status=Success, respCategory=Approved, respDesc=Success, respCode=0] metaData [txnType=BalanceInquiry, txnCategory=Non-Financial, URL=http://172.16.68.168:18080/ibmb/accounts/2147483651/balance] spanContext [traceId=b5f7fad589ebb657d9560f6b6c2d6ffd, spanId=4e6755d605895bbd]
In financial transactions, sessions often involve multiple logs that need to be combined and merged for a holistic view. Using session plugins, the platform seamlessly handles these intricacies, allowing the aggregation of transaction logs—starting and ending—providing a consolidated and insightful perspective for observability.

Plugin:

{
  "aggregate_fields": [
    "traceId"
  ],
  "logname_field": "leg",
  "instance_name": "Current Balance",
  "session_timeout_in_sec": 100,
  "grace_duration_in_sec": 100,
  "field_configs": {
    "Request to IBMB for getting current balance": {
      "set_fields_latest": {
        "RequestUUID": "RequestUUID",
        "RequestId": "RequestId",
        "RequestTime": "RequestTime",
        "CustID": "CustID",
        "MobileNumber": "MobileNumber",
        "AccountNumber": "AccountNumber",
        "TxnType": "TxnType",
        "TxnCategory": "TxnCategory"
      }
    },
    "Final response from IBMB for getting current balance": {
      "set_fields_latest": {
        "RootCause": "RootCause",
        "TxnStatus": "TxnStatus",
        "RRN": "RRN",
        "RespCode": "RespCode",
        "RespCategory": "RespCategory",
        "RespDesc": "RespDesc",
        "CurrentBalance": "CurrentBalance"
      }
    }
  },
  "end_identification_code": "(event.has('leg') && event.get('leg').asText() == 'Final response from IBMB for getting current balance');",
  "event_list_additional_fields": [],
  "enable_logstash_based_output": "false",
  "evaluate_time_diff": {
    "cbs_duration": [
      "Request to CBS for getting current balance",
      "Response from CBS for getting current balance"
    ]
  }
}
This configuration defines a session plugin for handling transaction logs related to acquiring the current balance from a bank’s service. It orchestrates the grouping of logs based on the “traceId” field, categorizes events using the “leg” field, and establishes a session named “Current Balance”. The configuration specifies expected fields for two crucial events in the transaction process, defines conditions for session termination, and calculates the duration between specific events to assess the time taken for the transaction. In essence, this session plugin streamlines the observability of current balance transactions by organizing, categorizing, and analyzing logs within a defined context. The consolidated log after parsing with the session plugin will look like the following, where logs from multiple touchpoints are stitched together:

Parsed Log:

{"session_status":"COMPLETED","@timestamp":"2024-01-11T22:50:22.591Z","traceId":"19c855c26010e409920e8bdec9f3abb4","TxnStatus":"Success","RRN":"1000338325","RespCode":"0","RespCategory":"Approved","RespDesc":"Success","CurrentBalance":"0.0","RequestUUID":"OLKB21914196","RequestId":"VFM20872419","RequestTime":"04:20:22.582960","CustID":"HAG77074","MobileNumber":"7243874141","AccountNumber":"2147483650","TxnType":"BalanceInquiry","TxnCategory":"Non-Financial","cbs_duration":8,"Duration":8,"scaled_uei":10,"Result":"Success","RootCause":"NotApplicable","ibmb_duration":0,"JourneyName":"IBMB","AppName":"IBMB"}
After parsing with the session plugin, the resulting consolidated log provides a comprehensive view of the entire transaction. This log includes critical information such as trace identifiers, transaction status, response details, and specific durations at various stages. Viewable in the Explore section of the platform, this consolidated log serves as valuable input for various consumption layers, including visualizations, storyboards, alerts, and reports, enriching the overall observability of financial transactions. This use case illustrates how session plugins effectively handle the intricacies of transaction logs, ensuring a seamless and comprehensive understanding of financial transaction journeys.

Use Case 2: Key-Value (KV) Parsing

Scenario: Introducing a Key-Value (KV) logging scenario, where logs follow the KV format with “:” as the key-value separator and “,” as the field separator. Employing a key-value (KV) plugin, the platform processes these KV logs, extracting valuable information and enabling effective financial transaction observability. Consider the processed log in the above case as below, which has multiple fields in key-value pairs to be extracted through KV parser.

Raw Log:

{"session_status":"COMPLETED","@timestamp":"2024-01-11T22:50:22.591Z","traceId":"19c855c26010e409920e8bdec9f3abb4","TxnStatus":"Success","RRN":"1000338325","RespCode":"0","RespCategory":"Approved","RespDesc":"Success","CurrentBalance":"0.0","RequestUUID":"OLKB21914196","RequestId":"VFM20872419","RequestTime":"04:20:22.582960","CustID":"HAG77074","MobileNumber":"7243874141","AccountNumber":"2147483650","TxnType":"BalanceInquiry","TxnCategory":"Non-Financial","cbs_duration":8,"Duration":8,"scaled_uei":10,"Result":"Success","RootCause":"NotApplicable","ibmb_duration":0,"JourneyName":"IBMB","AppName":"IBMB"}
The following is the KV plugin to be used for this purpose:

Plugin:

{
  "target": "body",
  "source": "body",
  "field_split": ", ",
  "value_split": ":"
}
The provided parser configuration is designed to process log data containing key-value pairs within the log body. It uses a simple logic where key-value pairs are separated by a comma and space (“, “), and each pair is split by an equal sign (“:”). This transformation allows the extraction of structured information from unstructured log data, making it more accessible for analysis within the logging and observability platform.

More Use Cases

XML-based Logging in Bill Payment Service

Scenario: In the context of financial transactions, consider XML-based logs from a Bill Payment Service (BBPS). vuSmartMaps platform seamlessly converts these XML logs into JSON format, initiating the regular flow within the data pipeline. This conversion enhances the data’s interpretability and sets the stage for comprehensive financial transaction observability.

Use Case 3: Bridging Data Interpretability Gaps for Financial Clarity

Meet Emma, a diligent financial operations analyst, tasked with ensuring the seamless functioning of Bill Payment Service (BBPS). Emma faced challenges in gaining comprehensive insights into the financial transactions due to the XML-based logs generated by BBPS.

The Issue:

Emma found interpreting XML logs to be a daunting task, hindering her ability to unravel the intricacies of the billing processes. Recognizing the need for a solution, she sought to streamline the logging mechanism to make data more accessible and insightful.

Enter vuSmartMaps:

Emma’s quest for a resolution led her to vuSmartMaps, a platform that promised to transform XML logs into a more user-friendly JSON format. This could simplify log structures, align with industry standards, and seamlessly integrate into their data pipeline.

The Transformation:

Upon integrating vuSmartMaps, Emma witnessed a transformative shift. The XML logs, once a complex set of information, are seamlessly converted to JSON, offering a clearer and standardized view. This not only simplified her data interpretation but also aligned the logs with industry standards, ensuring compatibility and collaboration with other systems.

The Impact:

With this solution, Emma could now efficiently query and analyze financial transaction data. The JSON format empowered her to identify patterns, anomalies, and trends effectively. The streamlined process not only alleviated Emma’s challenges but also marked a significant step forward in their strategy to optimize financial transaction observability.

Summary:

In Emma’s journey, vuSmartMaps emerged as a transformative ally, simplifying the way she approached financial transaction logs. The user-friendly JSON format makes her job easier and also reflects the commitment to continuous improvement and innovation in their billing services. Emma’s story is a testament to the power of user-focused solutions in overcoming operational challenges.

Contextualisation of Data with Enrichment in Data Stream

Scenario: In the logs observability, the need to extract meaningful information from encoded data is a common challenge. The Data Enrichment feature within vuSmartMaps addresses this challenge by allowing users to create Enrichment Tables, mapping key identifiers to corresponding descriptive values. This facilitates a standardized approach to enriching data within streaming environments, empowering users to seamlessly translate coded information into actionable insights. The ensuing user story exemplifies how Data Enrichment transforms raw data into comprehensible and valuable intelligence, enhancing the overall observability and diagnostic capabilities across diverse applications and systems.

Use Case 4: Enhancing Apache Server Observability with Response Code Enrichment

Meet Chris, an IT specialist responsible for observing an Apache server using vuSmartMaps. Chris faces the challenge of deciphering server status from various Response Codes logged by the Apache server. Understanding the significance of these codes, such as 200 for OK, 204 for No Content, and 404 for Not Found, is crucial for effective debugging.

The Challenge:

In the complex Apache server logs, Chris identified the need to convert Response Codes into meaningful Response Information. This conversion is essential for quick issue identification and debugging. To address this challenge, Chris utilizes the Data Enrichment feature within vuSmartMaps.

The Solution:

To streamline the process, Chris creates an Enrichment Table within vuSmartMaps, mapping Response Codes to their respective Response Information. This table becomes a reference point for translating codes into meaningful insights. The fields within this table are organized systematically to ensure a standardized approach to data enrichment.

Enrichment Table Fields:
  1. Key: Response Code
  2. Value: Response Information

Utilizing Enrichment in Data Stream:

Chris seamlessly integrates the Enrichment Table into the data stream using the vuSmartMaps enrichment plugin. This integration ensures that, as the data flows through the customer’s environment, the Enrichment process smoothly transforms Response Codes into descriptive Response Information.

Impact:

The enrichment of Response Codes significantly enhances Chris’s ability to interpret server status directly from the logs. With this standardized approach, decoding Response Codes becomes an automated and efficient process, allowing for quicker issue resolution and proactive debugging.

Summary:

In Chris’s observability journey, the use of Data Enrichment within vuSmartMaps emerges as a crucial platform. The systematic mapping of Response Codes to Response Information simplifies the interpretation of Apache server logs. This user-focused solution not only streamlines the observability process but also reflects the platform’s commitment to providing practical and efficient features for enhanced server observability.

Use Case 5: Streamlining Oracle Server Details with Host IP Enrichment

Meet Sarah, a diligent database administrator leveraging vuSmartMaps to observe Oracle servers. Faced with the challenge of distinguishing servers sharing common host IPs, Sarah employs the Data Enrichment feature. By creating an Enrichment Table associating host IPs with specific details such as AppName, DatabaseName, Environment, Location, and Owner, Sarah transforms raw data into actionable insights. The seamless integration of this enrichment process within the data stream enhances Sarah’s ability to efficiently identify, manage, and diagnose Oracle servers, exemplifying the user-centric power of Data Enrichment in diverse observability scenarios.

ContextStreams Use Cases in Banking Applications

In banking applications, the efficient orchestration and observability of transactions play an important role in ensuring a seamless and secure customer experience. Data streams, exemplified through use cases like the core banking system’s role in transaction processing, the integrity maintenance in credit card applications, and the efficient orchestration of IVR flows, form a critical backbone. The convergence of raw logs and enriched data, facilitated by advanced platforms like vuSmartMaps, underscores the industry’s commitment to innovation. These data stream use cases not only streamline the observability process but also enhance overall diagnostic capabilities, reflecting a broader trend in leveraging technology for deep observability across diverse banking operations.

Use Case 6: Orchestrating Seamless Transactions through Banking Middleware

Mike, an IT professional is responsible for overseeing the integrity of a bank’s transactions facilitated by the core banking application. In the intricate world of banking operations, every credit or debit request, irrespective of its origin, is directed through the core banking interface, acting as a crucial intermediary to the bank’s core system.

Transaction Journey:

When a credit or debit transaction is initiated, be it through a physical card at a point-of-sale terminal or an online/mobile platform, the Core Banking System ensures the proper formatting and secure transmission of transaction details to the core banking system. This core system serves as the backbone of the bank’s operations.

Extending our financial transaction observability, in this use case logs are arriving in the ISO-8583 format. Leveraging a parser, these ISO-8583 logs undergo conversion to JSON, aligning them with the standard data format. Following this transformation, the data follows the routine flow within the platform’s data pipeline, ensuring smooth and consistent observability. Parsing ISO-8583 logs proved to be a stumbling block for Mike, creating friction in the seamless flow of financial data. With the implementation of the vuSmartMaps Log Parser, Mike witnessed a remarkable shift. ISO-8583 logs, initially a complex challenge, seamlessly transformed into JSON, adhering to industry standards.

Transaction Processing and Response:

The core banking system meticulously processes debit or credit requests, verifies customer account details and ensures sufficient funds for debit transactions. Subsequently, it sends a response back to the originating channel, indicating the success or any issues with the transaction.

Logs and Enrichment:

In transaction observability, the raw log data showcases the intricate details of each transaction. To enhance interpretability, Mike utilizes vuSmartMaps to enrich the logs. By creating a specific data adapter.

The Impact:

The hyper-configured data adapter streamlines the observability process, enabling Mike to effortlessly interpret transaction logs. With a standardized approach, decoding raw data becomes an automated and efficient process, enhancing overall observability and diagnostic capabilities.

Summary:

Mike’s user story underscores the power of core banking systems in orchestrating seamless transactions, coupled with vuSmartMaps’ Data streaming feature for comprehensive log interpretation. This user-focused solution exemplifies the platform’s commitment to efficiency and innovation in the observability of critical banking operations.

Use Case 7: Ensuring Offer Integrity in Credit Card Applications Through Credit Card Middleware

Meet Jennifer, a vigilant tech professional responsible for overseeing the integrity of credit card applications in a bank, particularly within the credit card journey. One critical touchpoint in this journey is designed to observe users’ status concerning Upgrade offers.

Functionality:

At this specific touchpoint, a crucial check is implemented to determine if a user currently holds an Upgrade offer. If an existing Upgrade offer is found, the system restricts the user from submitting another application for the same offer. This preventive measure is in place to avoid users reapplying for identical Upgrade offers while an active one is already in progress.

Transaction Flow:

When a user initiates an application or an upgrade, the request is processed through various channels, ensuring the integrity of the Upgrade offers. A request is sent, blocking the offer with specific details like AccountNo, ProductCode, ReferenceNo, Source, and more.

Response Handling:

Upon processing, the system generates a response confirming the successful availing of the offer or providing relevant feedback. This response is crucial in determining the status and success of the application.

Data Stream Processing:

In this observability use case, raw logs from these transactions are processed and enriched using vuSmartMaps. The Data Stream Use Case ensures seamless observability of the credit card application journey, allowing Jennifer to interpret the journey effortlessly.

Summary:

Jennifer’s user story highlights the importance of observability in maintaining offer integrity within credit card applications. The integration of vuSmartMaps for data stream processing exemplifies the platform’s commitment to providing a user-focused solution for efficient observability.

Use Case 8: Navigating the Complexity of Banking IVR with vuSmartMaps

Meet Emily, a dedicated customer service representative facing the complex task of analyzing and resolving customer queries within the complex web of Retail Interactive Voice Response (IVR) Flow. The banking IVR, with a large number of options like PIN changes, new PIN generation, balance inquiries, and credit card statement views, presented a complex landscape. The siloed observability tools proved inadequate for tracking and analyzing the diverse journeys customers could embark upon, leading to inefficiencies in issue resolution and impacting customer satisfaction.

The Issue:

Recognizing the need for a solution that could seamlessly navigate the complex Banking IVR, Emily turned to vuSmartMaps. Known for its real-time visibility and actionable insights, vuSmartMaps seemed tailor-made for Emily’s challenges. Her goal was straightforward – enhance the observability of the Retail IVR Flow, identify bottlenecks across various customer journeys, and streamline issue resolution for a seamless and satisfactory customer experience.

vuSmartMaps in Action: ContextStreams and Parsers:

Utilizing a combination of simple grok, mutate, translate, and configured-session plugins, vuSmartMaps’ ContextStreams played a pivotal role in addressing Emily’s challenges. These streams streamlined data, allowing the support team to identify various metrics specific to different banking IVR journeys. From tracking the prompts played during calls, user-selected options, and call regions to discerning service providers and understanding call end reasons, vuSmartMaps provided granular insights.

The Advantages:

  • Granular Insights for Personalized Service: ContextStreams empowered Emily to distinguish and understand the dynamics of each customer interaction within the banking IVR. Whether a user sought balance information or navigated through credit card-related options, Emily now had the platforms to offer personalized and effective service.
  • Swift Issue Resolution for Improved SatisfactionArmed with contextual information, Emily could swiftly identify issues specific to each banking IVR journey. Whether it was a glitch in PIN validation or a hiccup in credit card inquiries, vuSmartMaps contributed to quicker resolutions and a substantial boost in customer satisfaction.
  • Comprehensive Dashboards for Strategic Decision-Making: vuSmartMaps offered Emily comprehensive dashboards tailored to the nuances of banking IVR. These included total call counts, Registered/ Non-registered Mobile Number distribution, location-wise statistics, service provider-wise stats, call end reasons, and more – all critical for strategic decision-making.
  • Proactive Alerts for Timely ActionThreshold-based alerts were configured to notify Emily and her team about critical issues within specific banking IVR journeys. Alerts were triggered for failures exceeding predefined thresholds, non-responsive components, volume dips, and infrastructure alerts based on CPU, memory, and disk utilization.
  • Backend Connection InsightsThe solution allowed Emily to discern which backend had more connections based on user-selected options. Whether it was a surge in connections to core banking applications for balance inquiries or a spike in credit card-related interactions directed to the credit card backend, vuSmartMaps provided valuable insights.

Summary:

vuSmartMaps, with its ContextStreams and advanced parsers, emerged as a transformative solution for Emily. It not only navigated the complexities of banking IVR but also elevated the efficiency of observability and issue resolution. The platform’s ability to provide detailed insights, swift issue resolution, and proactive alerts significantly contributed to Emily’s success in delivering exceptional customer service.

Conclusion

vuSmartMaps leads the charge in observability, utilizing dynamic ContextStreams and robust data enrichment to process data from start to finish seamlessly. What sets this platform apart is its capability of organizing raw data, offering practical insights that enhance user experiences and operational efficiency. With a dedicated commitment to performance, scalability, and real-time processing, vuSmartMaps emerges as a game-changer. It tackles challenges through a scalable structure, ensuring top-notch data quality and resource efficiency. vuSmartMaps unlocks the full power of observability, establishing it as a go-to solution for unlocking invaluable insights and ensuring organizational success.

Further Reading

  1. Data Streams
  2. Data Enrichment

Resources

Browse through our resources to learn how you can accelerate digital transformation within your organisation.

Unveiling our all powerful IBMB Observability ExperienceCenter. Click Here