ContextStreams – For a Business Centric Observability Platform “Context is all you need”

The role of the Observability pipeline has become critical for organizations looking to utilize the full potential of their data. Imagine a scenario in which your data pipeline evolves beyond a simple channel for data transfer. Imagine it as an intricate system that intelligently transforms and enriches logs, metrics, and traces with vital domain context, elevating data into ‘decision-grade-information’. Instead of managing data, it’s about leveraging data as a foundational element of business intelligence, operational agility, and strategic decision-making.

With this end-user impact in mind, and with deep engineering efforts in working and optimising at scale, we are proud to unveil a major milestone “ContextStreams” for vuSmartMaps™, our domain-centric Observability pipeline into our latest release.

At the heart of vuSmartMaps™ lies ContextStreams, a component so vital that we prefer to dub it the “business context engine” of our Observability platform.

Building Blocks of ContextStreams

ContextStreams is made up of three components.

1. I/O Streams – a distributed message bus that holds the data to be processed. It is responsible for data availability and makes sure that data is not lost due to any issues.

2. Data Pipelines – this is the place where real magic happens. The data pipelines are responsible for the transformation and enrichment of data with vital domain context.

3. DataStore Connectors: it allows you to integrate the contextualized data with one or more downstream data stores. Currently, we support the commonly used data analytical stores.

How ContextStreams Works?

Collect with Ease

ContextStreams stands out by allowing the ingestion of data at high volumes and velocities (you can ingest hundreds of thousands of EPS on commodity hardware) from many data sources. These data sources could be API endpoints, agents streaming data, other databases, cloud streaming engines, log sources etc.

VuNet’s library of O11ySources (Prepackaged source integration and corresponding context stream building blocks. Examples: A VMWare Hypervisor O11ySource, AWS EKS Cluster O11ySource) allows you to seamlessly ingest data and it would take care of the creation/configuration of essential components — I/O streams, data pipelines, and DataStore Connectors for the chosen data source. You won’t have to worry about the format or structure of the incoming data with pre-packaged O11ySources.

Contextualize Efficiently

Data pipelines are where you can transform, enrich, and filter data using out-of-the-box plugins. Our library of plugins ranges from regular data transformation operations like time conversion, field enrichments, arithmetic operations, and metrics aggregations to complex actions like micro-transactions stage tracking, and dynamic transaction ID-based correlation of events.

ContextStreams offers domain-centric adapters specifically built to apply your business, domain and environment context to logs, metrics, and traces. There are adapters built to transform data from a specific application, with business, domain or environment context for business functions (eg: adapters that understand payment transactions, transaction stages, payloads, error cases, and error codes) to create better compliance reports. More importantly, ContextStreams comes with an arsenal of 25+ domain adapters for the Banking and Financial Services Industry.

You can also seamlessly build custom plugins that adhere to your business use case ensuring that the data is not just processed but imbued with meaningful context using our low code adapter framework.

ContextStreams allows you to build highly customizable data pipelines via a plug-and-play plugin architecture. Everything is configuration-driven with the user having to write little to no code. Make use of the “Preview” feature and tailored debug options to ensure that data is verified and validated at every stage. Additionally, our interactive UI enables you to seamlessly build, validate and deploy pipelines.

Cast Intelligently

After transformation, the data is ready to be consumed. ContextStreams facilitates this through its data store connectors, allowing the transfer of transformed data to a storage destination of your choice for later consumption and analytics – VuNet’s HyperScale Data Store that is built for cost-effective and efficient storage of data, Elasticsearch, TimescaleDB, or a storage solution of your choice.

Scalability

ContextStreams can handle events in the order of millions of events per second thanks to resources efficient multi-threading and horizontally scalable architecture. You can ingest data at scale and the data pipeline contextualises large volumes of data in real-time.

Data Loss Prevention and Cost Efficiency

ContextStreams is designed to ensure no data loss during ingestion and transformation, using an at-least-once-delivery guarantee architecture. Given that the raw data is transformed and needless portions of data can be dropped the final output is intricately business-specific and thereby optimized to save cost. Additionally, since the platform transforms logs into metrics which once again contributes to storage savings.

Context is All You Need

Like the seminal paper on Generative AI that highlighted that “attention is all you need”, we believe “business context” should be at the epicentre of your objectives from an Observability Platform.

The flexibility and scalability offered by ContextStreams mean that businesses can now fully control their data’s journey from input to insight. By leveraging the three pillars of observability (logs, metrics, and traces) from chosen sources and storing processed data on a cost-effective platform, organizations can unlock valuable business insights. This not only ensures that infrastructure scales with business needs but also empowers business teams to make informed decisions swiftly.

In conclusion, ContextStreams for vuSmartMaps™ is transforming the landscape of business intelligence. Its ability to collect, contextualize, and consume data efficiently makes it an indispensable tool for businesses looking to leverage their data for actionable insights. With ContextStreams, the path from raw data to real-time business intelligence has never been easier.