9.1 Users Management and RBAC

9.2 Authentication and Security

9.3 Data Management and Data Model Handling 

9.3.1 Storage

9.3.2 Retention

9.3.3 Export/Import 

Resource Management

Import Data

9.3.4 Working with Data Model
9.3.5 Data Extraction

9.4 Control Center

9.4.1 License Entitlements

9.5 Platform Settings

9.5.1 Definitions

9.5.2 Preferences

9.5.3 About

Data Streams >

1. Introduction to vuSmartMaps™

2. Getting Started with vuSmartMaps™

3. Installing vuSmartMaps™

4. Configuring vuSmartMaps™

Observability Sources

Data Pipeline and Parser Configuration

Data Enrichment Techniques

Onboarding Applications and Business Journeys
Configuring RCABot and ML Models

Tuning Hyperparameters and Model Training

5. Observability Through vuSmartMaps™

Dashboards and Visualization 

Business and Operational Storyboard
RCA Storyboard

Alert Console and Correlation

Alerts and Notification

Rule-based and Dynamic threshold-based Alerts

Programmable Alerts

Notifications and Triggers

Alert Customization Notification

RCABot and ChatBot Interactions

Ved – Gen AI Bot

Comprehensive Reporting

Log Management and Analysis

Application Observability

Journey Observability

6. Understanding vuSmartMaps™

Platform Architecture and Key Services
Insights from Logs, Traces and Metrics
How domain-centric approach enables better RCA and ML insights?
Security and Compliance

7. Administering vuSmartMaps™

Dashboard Creation

Visualizations 

Panels

Custom Panels

• UTM Visualization

• Matrix Visualization

• Insights

Alert Creation

Data Source Management

Report Generation

Data Onboarding and Instrumentation

Mobile Dashboard Configuration

9. Managing vuSmartMaps™

Users Management and RBAC

Authentication and Security

• Data Management and Data Model Handling 

Storage

Retention

• Export/Import 

• Resource Management

• Import Data

Working with Data Model
Data Extraction

• Control Center

License Entitlements

• Platform Settings

Definitions

Preferences

About

10. Glossary

11. Support and Troubleshooting

ContextStreams

Introduction

ContextStream, also sometimes referred to as Data Streams, are an integral part of the data flow process within the vuSmartMaps™ platform. ContextStream is a comprehensive data processing engine that receives the data, processes, enriches, massages, and transforms the data, and then writes it into a data store for the end user’s consumption. It is highly scalable, capable of handling multi-million records, and engineered to ensure there are no single points of failure and no data loss.

Data pipelines are constructed with multiple data transformation plugins such as session plugin, data manipulation, ISO-8583, etc. These pipelines are highly versatile, enabling users to transform and enrich data in various ways to suit their specific needs. The primary function of Data pipelines is to read data from a data stream, process it, and then send it to another Data Stream/database.

ContextStreams: Transforming Data for Comprehensive Observability

ContextStreams within vuSmartMaps serve as sophisticated orchestrators that not only transport raw data but also transform it with rich context, making it meaningful for the observability journey. The process of contextualization involves the enhancement of raw data by adding metadata, categorizing information, and ensuring it aligns with the specific requirements of observability.

Why Contextualization Matters

Raw data, in its unstructured form, often lacks the necessary context for effective analysis. Contextualization bridges this gap by adding layers of information that provide insights into the who, what, when, where, and why of each data point. For example, associating a transaction with a specific user, timestamp, and geographic location. By contextualizing data, organizations can unlock a deeper understanding of their systems, transactions, and user interactions.

The Journey of Raw Data through ContextStreams

  1. Ingestion: The journey begins with the ingestion of raw data from diverse sources, such as network devices, servers, applications, and more. This data may vary widely in formats, structures, and levels of granularity.
  2. Transformation: As data enters the ContextStream, parsing mechanisms break down complex structures, and standardization processes ensure uniformity. This step is crucial for harmonizing diverse data formats.
  3. Enrichment: Enrichment techniques come into play, adding contextual information to the data. This can include augmenting data with details about the source, applying geolocation information, or correlating data with external databases.
  4. Categorization: ContextStreams categorize data based on predefined rules or machine learning algorithms. This step helps in organizing data into logical groups, simplifying subsequent analysis.
  5. Routing: Once enriched and categorized, data is intelligently routed to the appropriate storage indices/tables, ensuring that it is accessible for specific use cases, whether it be performance observability, security analysis, or business intelligence.
  6. Storage Indices: The processed and enriched data is stored in data storage indices through data connectors. It enables seamless retrieval and analysis for actionable insights on the consumption layer with visualizations, storyboards, alerts, and reports.

Examples of Contextualization

  1. Transaction Context: For a banking application, ContextStreams may enrich transaction logs with details such as user identities, transaction types, and timestamps, providing a holistic view of each transaction’s journey.
  2. Geospatial Context: In a network observability scenario, IP addresses can be enriched with geospatial information, enabling organizations to visualize network traffic patterns across regions.
  3. Service Correlation: For a microservices architecture, ContextStreams may correlate logs from different services, providing a cohesive narrative of complex transactions spanning multiple components.

ContextStreams Features and Capabilities

Components of ContextStreams

The data is collected from the target system through Observability Sources. The data then undergoes a significant transformation during the data processing phase, thanks to Data Streams. This phase occurs in the distinct sections: I/O Streams, Data Pipeline, and DataStore Connectors, with each having a unique role in processing data. Additionally, the  Flows tab gives a display of the data processing journey.

  1. I/O Streams: The data ingested from the customer environment enters the vuSmartMaps platform through an input stream within the I/O Streams. This initial step is vital for preparing the data, and setting the foundation for subsequent transformations. After processing within the I/O Streams and data pipeline, the structured data seamlessly flows into the output stream, ready for further utilization in the platform.
  2. Data Pipeline: Subsequently, the processed data from the input Streams flows through a series of transformations within the Data Pipeline. This phase is pivotal for data processing, refining raw data into a structured and meaningful format. After processing in the Data pipeline, the data is directed to an output stream for further utilization.
  3. DataStore Connectors: The role of DataStore Connectors is to facilitate the transfer of processed data to designated storage destinations. These connectors play a crucial role in directing data to various storage solutions such as Elasticsearch, or the  Hyperscale Data Store.
  4. Flows: The Flows section visually represents the entire data flow journey, acting as a valuable reference to understand how data moves from the Input Stream through the Data Pipeline and DataStore Connectors to its ultimate storage destination. This visual representation is an essential feature for system observability, providing insights into the flow and transformation of data within the platform.

To visualize the journey of data through ContextStreams, consider the following diagrammatic overview:

This diagram provides a visual narrative of how ContextStreams orchestrate the entire journey of raw data, transforming it into a valuable asset for comprehensive observability. Now Let’s look into the live data pipeline, showcasing a real-world example of the contextualization of raw logs from a typical Internet and mobile banking application through a sophisticated ensemble of data adapters and parsers.

Debugging Feature: I/O Streams offer a “Preview” option, enabling users to verify and download streamed data, facilitating quick debugging. Data pipelines provide three debug options — For Published pipeline, Draft pipeline, and at the Block level—allowing users to inspect the output at different stages, aiding in effective debugging and diagnosis of data processing workflows.

In the subsequent sections of this user guide, we will explore ContextStreams in more detail, including their configuration, key functionalities, and how they contribute to an enhanced data flow experience.

Supported Data Transformation Plugins

In the data transformation process within vuSmartMaps, a diverse set of plugins empowers users to shape and enrich the data with Contextstreams. These plugins play a pivotal role in tailoring the data flow to specific needs. Explore the following supported plugins to understand their functionalities and discover how they can enhance your data processing experience.

  1. Aggregation Plugin: Aggregates real-time streaming data based on a set of bucketing fields, enabling dynamic data summarization.
  2. Arithmetic Plugin: Used to find the difference between values of successive events.
  3. C24 Plugin: Parses logs from specific formats (RuPay/ IMPS formatter/ SHC/ L7LB) and extracts required information for further processing.
  4. Correlate Plugin: Correlates two independent sources based on specified criteria, facilitating the correlation of related data.
  5. Data Enrichment Plugin: Enriches the incoming stream using static values, enhancing the data with additional context.
  6. Data Manipulate Plugin: Enables manipulation of the incoming stream using Java/JS functions, allowing for dynamic transformations.
  7. Date Time Plugin: Casts, converts, or updates date-time values based on ISO or custom formats and timezones, ensuring consistent date-time handling.
  8. Elapsed Plugin: Tracks the duration of an event, providing insights into the time taken for specific processes.
  9. Event Clone Plugin: Clones events, providing duplication for specific use cases such as parallel processing or backup scenarios.
  10. Event Split Plugin: Splits an event into multiple events based on a field present in the configuration, facilitating event segmentation.
  11. Filter Plugin: Filters the incoming stream based on a valid Java-based condition, allowing users to focus on specific data subsets.
  12. GeoIP Plugin: Adds information about the geographical location of IP addresses, enriching data with location-based insights.
  13. GSub Plugin: Replaces all occurrences of a string in another string based on the input Java regex pattern, aiding in string manipulation.
  14. Grok Plugin: Parses unstructured log data into a structured and queryable format, facilitating efficient log analysis.
  15. ISO 8583 Plugin: Parses an ISO message into different formats, supporting interoperability with ISO 8583-based systems.
  16. KV Plugin: Extracts key-value pairs from a message (or mentioned field) based on a specified delimiter, aiding in data extraction.
  17. Session Plugin: Tracks the progress and status of transactions from logs, providing insights into transactional workflows.
  18. Stream Key Plugin: Changes the Kafka message key of an incoming event, allowing for key-based event processing.
  19. Stream Timestamp Plugin: Parses dates from fields and utilizes them as Kafka timestamps, aiding in time-sensitive event processing.
  20. Truncate Plugin: Allows the truncation of fields longer than a given length, operating on byte values for efficient field length control.
  21. User Agent Plugin: Adds information about the user agent, including name, version, operating system, and device, enhancing user-related data.
  22. XML Plugin: Takes a field containing XML and expands it into an actual data structure, enabling efficient handling of XML data.

These plugins collectively offer a versatile toolkit for users to shape and transform their data streams according to specific needs and scenarios, enhancing the flexibility and efficiency of data processing within the vuSmartMaps platform. For more specific details on each plugin and its usage, refer to the vuSmartMaps Plugin Documentation.

ContextStreams Workflows

vuSmartMaps provides two distinct methods for data processing to cater to diverse user requirements. The first method involves seamless data contextualization using standard O11ySources. The second approach allows users to create data streams based on their unique requirements.

Automated Data Management with Standard O11ySources

Opting for data ingestion through standard O11ySources provides a seamless and automated data contextualization experience. The system takes care of creating the essential components — I/O streams, data pipelines, and DataStore Connectors — specific to the chosen O11ySource. Enabling the O11ySource triggers the automatic creation of corresponding I/O Streams, initialization of the necessary Data Pipeline, and establishment of required DataStore Connectors. This automated workflow ensures efficient organization, transformation, and secure storage of data, eliminating the need for manual intervention. If required, you can enhance the data processing and contextualization logic by modifying the data pipeline configuration.

Refer to the O11ySources user guide to understand the specific workflow for O11ySources here. Detailed instructions for managing ContextStreams can be found in subsequent sections of this user guide.

Hyper-configurable ContextStreams Creation

The second approach involves creating a customized data stream tailored to specific requirements, requiring users to meticulously analyze the entire data flow journey and consider all touchpoints. The creation process involves defining the input stream, designing the data pipeline with necessary intermediate blocks for processing (each containing multiple plugins), and establishing an output stream. Additionally, DataStore Connectors are created to forward processed data to storage. Before initiating this creation process, a thorough analysis is crucial to accurately identify the required blocks. Depending on processing requirements, the design of the pipeline, blocks, and plugins should be ready before moving on to creating the ContextStream on the platform. The steps for creating the ContextStreams are discussed here in detail.

Accessing Data Streams UI

The Data Streams page can be accessed from the platform left navigation menu by navigating to Data Ingestion > Data Stream.

The Data Streams landing page will look like this where you can create/configure the I/O streams, data pipelines, or DataStore Connectors with the different options.

The user interface of the Data Stream section is composed of four primary tabs (I/O Streams, Data Pipeline, DataStore Connectors, Flows), each designed to facilitate specific actions and configurations, enhancing your ability to harness the full potential of ContextStream management.

Further Reading

  1. Creating ContextStreams
  2. Managing ContextStreams
  3. Debugging ContextStreams

Resources

Browse through our resources to learn how you can accelerate digital transformation within your organisation.

Unveiling our all powerful IBMB Observability ExperienceCenter. Click Here