Data Modelling Workspace

Introduction

In today's data-driven world, the ability to effectively manage and interpret vast amounts of information is crucial. Data Models play a pivotal role in this process by providing a structured and user-friendly framework for querying and refining data.

Users can create and manipulate data models that encapsulate complex business logic, making it easier to derive insights and make data-driven decisions. By simplifying database queries, Data Models enable users to focus on defining golden signals and their dimensions, applying transformations, and enriching data with domain-specific knowledge. This empowers users to build comprehensive data pipelines, enhance data with contextual information, and ultimately streamline the analysis process, ensuring that critical insights are readily accessible and actionable. It powers the functionality of Data Visualization, Alerts, Reports, and Insights, ensuring a comprehensive and intuitive data-handling experience.

Getting Started with Data Modelling Workspace

1. Accessing Data Modelling Workspace: Navigating your Data Modelling Workspace is a breeze. vuSmartMaps Data Modelling can be accessed from the left navigation menu by clicking Data Modelling Workspace under Configure Observability. Once you're there, you'll find a list of all the data models. You can easily manage your Data Models by performing actions such as creating, viewing, editing, and deleting them. For enhanced control, you can customize permissions for different roles—whether it's granting viewing rights or permitting modifications.
2. Data Store: A data store is a repository for persistently storing and managing collections of data. Accessing and handling the data stores is simple and can be found at the top-left corner of the Data Models listing page. vuSmartMaps supports a variety of backend databases, making it highly adaptable to your needs. You can effortlessly query these Data Stores using a standard Query Editor or tailor your queries to the specific capabilities of each Data Store. Currently, Data Stores officially supports PostgreSQL, MySQL, Elasticsearch, and Hyperscale DB.
3. New Data Model Creation: Creating a new Data Model is a breeze with our intuitive user interface. The process consists of four straightforward steps: Start by providing the basic information, set up your data query, apply any necessary transformations, and finally, define the properties of your Data Model.
   • Basic Information: You are required to provide a unique and intuitive name and a brief description of the data model.
   • Prepare Data Query: This involves crafting queries to fetch data from your databases. The process allows you to choose between building queries using a user-friendly Query Builder or directly specifying a Raw query, giving you flexibility in how you model your data.
   • Do Transform: These transformations enable data manipulation to suit your specific needs. While not mandatory for data models with a single query, they become vital, especially join and concatenate transformations, when working with multiple queries.
   • Additional Attributes: you can select the metrics and buckets you want to expose through this data model. In the 'Set Properties' step, you can choose, and define properties for the metrics and buckets to be accessible to Data Model users.
4. Support for Dynamic Filters in Dashboards VuNet's dashboard lets you use dynamic filters for data viewing. The addition of the data model to the dashboard occurs indirectly via Insights. Configure your queries with custom filters to accurately filter the data on the dashboard.

Accessing Data Modelling Workspace

vuSmartMaps Data Modelling Workspace can be accessed from the left navigation menu (Configure Observability > Data Modelling Workspace). Upon clicking 'Data Modelling Workspace', you are taken to the 'Landing Page’.

Landing Page

All the Data Models created in the system are visible here. Utilize features in the listing table, such as search and filters, to locate specific Data Models.

The listing gives information on the following:

• Name: The data model's name. Click it to view the data model in non-edit mode.
• Data Store Name: The name of the Data Store used by this Data Model.
• Type: Indicates if it's a data model that comprises metrics or events.
• Created By: The user who created the data model.
• Created At: The date and time when the data model was created.
• Modified By: The user who recently modified the data model.
• Modified At: The date and time of the most recent modification.
• Actions: Edit or Delete your Data Models using the buttons under this column.
• + button: It is used to create a new Data Model. You will learn how to create a Data Model under the New Data Model Creation Section

Actions

The Data Modelling Workspace's home page can be used for various basic operations such as, Edit and Delete Data Models.

Edit- Click on this icon to edit the respective Data Model.

Delete: You can delete a Data Model by clicking the delete option under the Action column. Please note that the Data Models can't be deleted if Insights Alerts or Reports are using it.

Deleting Multiple Data Models: Select the multiple Data Models by checking the boxes on the extreme left and clicking the Delete button on the top right.

Setting Permissions for a Data Model

We use RBAC (Role Based Access Control) to let you grant access to Object Level Permissions which are made available in 'Edit' Mode. This ensures that only authorized users can access specific functions, resources, or information.

• Click on Edit for the data model you want to manage.
• You'll be directed to the Set Properties page.
• Click on Permissions to access and manage Object Level Permissions.

The screen will look like this, For each role, you can assign three types of permissions

• None: There are no permissions given.
• View: The selected user can only view the Data Modelling Workspace.
• Modify: The selected can also modify and make changes to the Data Modelling Workspace.

Data Store

Data Store refers to the Sql and NoSql databases available in vuSmartMaps. vuSmartMaps supports multiple databases.

Supported Data Stores

The following Data Stores are officially supported:

• PostgreSQL 14.7 and above
• MySQL - 14.14 and above
• Elasticsearch 7.0 and above
• Hyperscale DB 23.8+ and above

Data Store Management

vuSmartMaps comes with the default configurations of NoSql and Hyperscale DB. You can easily add other supported data stores to the available data stores. Click on the Data Store button at the top right corner to see the listing of already configured data stores and also manage them.

Add New Data Store

1. Click Data Store on the right top corner of the landing page.

2. The Data Store landing page shows a list of previously configured Data Stores. Now click on the + icon on the top right to create a new Data Store.

   • Name: User-defined unique name for the Data Store.
   • Data Store Type: Type of the selected Data Store (e.g., PostgreSQL, Elasticsearch).
   • Created By: Name of the user who created the Data Store.
   • Created At: Date and time when the Data Store was created.
   • Modified By: Name of the user who made the most recent modifications to the Data Store's configuration.
   • Modified At: Date and time of the most recent configuration changes.
   • Actions: Each Data Store has two buttons:
     • Edit: Click to make changes to the Data Store's configuration.
     • Delete: Click to delete the Data Store (further details are in the next section).

3. You can see the list of supported Data Stores. You can search for any specific supported Data Store in the search bar. Click on the Data Store of your choice.

4. Follow the Data Store's instructions for configuration, then click the Save & Test button.

   • PSQL Data Store: All the fields are mandatory. The platform engineer or an admin should know the access details. _ Data Store Name: Enter the name of the Data Store.
     _ Description: Give a brief description.
     _ Database: Admin to enter the Database name.
     _ Host: Enter the Database Host IP address.
     _ User: Enter the user name of the Database
     _ Password: Enter the Database password.
     _ Port: Enter the Port number.
     _ Timezone: You can manually choose the Timezone or click on the Detect Time Zone to automatically select the timezone.
     _ Selecting a Time Zone at the Data Store level ensures correct time-range data is fetched from the Data Store.
     _ isSSL: Enable it if you have SSL (Secure Sockets Layer) certified and enter the CA certificate details, Client Certificate, and vuSmartMaps.dataModels.labels.client_key.

     

   • Elastic Search Data Store: All the fields are mandatory. The platform engineer or an admin should know the access details. _ Data Store Name: Enter the name of the Data Store.
     _ Description: Give a brief description.
     _ Host: Enter the Database Host IP address.
     _ Port: Enter the Port number.
     _ ES Version: Choose the Elasticsearch version from the dropdown.
     _ Timezone: You can manually choose the Timezone or click on the Detect Time Zone to automatically select the timezone. It is a mandatory field.

     

     • Hyperscale Data Store: All the fields are mandatory. The platform engineer or an admin should know the access details.

       • Data Store Name: Enter the name of the Data Store.
       • Description: Give a brief description.
       • Database: Admin to enter the Database name.
       • Host: Enter the Database Host IP address.
       • User: Enter the user name of the Database
       • Password: Enter the Database password.
       • Port: Enter the Port number.
       • Timezone: You can manually choose the Timezone or click on the Detect Time Zone to automatically select the timezone.
       • isSSL: Enable it if you have SSL (Secure Sockets Layer) certified and enter the CA certificate details, Client Certificate, and vuSmartMaps.dataModels.labels.client_key.

       

5. If the Data Store connection is successful, you'll be directed to the listing page, and your newly added Data Store should be visible.

note

If the hostname, user, and password are incorrect then the Datastore will not get created and immediately throws an error message.

Delete

Use this option to delete the Data Stores that are no longer required.

• Click the Delete button under the “Actions" column for the specific Data Store you want to delete.
• A confirmation window will appear to ensure you want to proceed with the deletion.
• Click Delete to confirm and permanently remove the Data Store.
• If you change your mind, you can click Cancel to abort the deletion process.

Please note that deleting a Data Store is irreversible, so be certain of your decision before confirming the deletion.

note

You can’t delete the Data Store if it is being used by any of the Data Models. The system will immediately throw an error.

With that, you have successfully created a new Data Store.

New Data Model Creation

To add a new Data Model, simply click the + icon on the Data Models listing page.

Data Model creation is a 4 step process. These steps are as follows:

1. Basic Information: Fill in the basic details.
2. Prepare Data Query: Configure the data query.
3. Do Transform: Apply transformations as needed.
4. Additional Attributes: Define the Additional Attributes of your Data Model.

FAQs

What are Data Stores in vuSmartMaps?

Data Stores in vuSmartMaps are repositories for persistent data storage and management. Key features include:

• Supported Databases: vuSmartMaps supports PostgreSQL, MySQL, Elasticsearch, and Hyperscale DB.
• Data Handling: You can manage and access these Data Stores through the Data Modelling Workspace, enabling you to store and retrieve large volumes of data efficiently.
• Configuration: Set up Data Stores by specifying connection details, database type, and authentication settings.

Can I customize my data models to suit my specific needs?

Yes! You can customize your data models by defining properties, applying transformations, and adding new fields. You can also use arithmetic operations, expressions, or custom values to modify data models based on your specific requirements.

What should I do if I encounter issues while connecting to a Data Store?

• Check Credentials: Ensure that the hostname, username, password, and port are correctly entered.
• Network Configuration: Verify that network settings and firewall rules allow connections to the Data Store.
• SSL Configuration: If using SSL, confirm that the CA certificate, client certificate, and client key are properly configured.

What is the advantage of using the Query Builder?

The Query Builder offers:

• Intuitive Interface: Simplifies query creation with a visual interface, making it easy for non-SQL users.
• Cross-Database Compatibility: Supports various databases like PostgreSQL, Elasticsearch, and Hyperscale DB.
• Efficiency: Accelerates query building with pre-configured templates, reducing errors and saving time.

How can I filter time series data in my query?

To filter time series data:

• Timestamp Field: Select the appropriate timestamp field in your query.
• Time Range: Specify the time range (e.g., last 24 hours) to filter relevant data.
• Date Histogram: Use the date histogram aggregation for time-based data grouping.

What is the purpose of Dynamic Filters?

Dynamic Filters enable:

• Real-Time Filtering: Adjust filters based on real-time data to refine results dynamically.
• Flexible Analysis: Allows you to modify filters without altering the base query.
• Interactive Dashboards: Enhances dashboards by enabling dynamic data exploration based on changing criteria.

How can I run and preview my queries?

To run and preview queries:

• Run Query: Execute individual queries to verify accuracy.
• Run All Queries: Execute all queries collectively to see the combined results.
• Preview Results: View the query results in the preview pane to validate output before saving.

What should I do if I encounter validation errors during query creation?

If validation errors occur due to missing or unavailable data, you can click Skip Validation to proceed to the next step. This bypasses the data check and allows you to continue building the data model.

Are there additional options available for managing queries?

Yes, you can perform operations like:

• Copy: Duplicate existing queries for reuse.
• Delete: Remove unnecessary queries.
• Disable: Temporarily deactivate queries without deleting them.
• Saved Search: Use saved searches to auto-populate queries with pre-configured filters.

Is it possible to add new fields to query outputs?

Yes, you can add new fields using the “Add Field” transformation. Options include:

• Arithmetic Operations: Perform calculations between fields.
• Custom Values: Add static values.
• Expressions: Create new fields using complex formulas or conditions.

How do severity rules work in the context of the "Set Properties" step?

Severity rules allow you to:

• Define Value Ranges: Set thresholds for metrics to trigger alerts.
• Thresholding Options: Choose between no thresholding, dynamic thresholding (ML-based), or static thresholding (fixed values).
• Notifications: Configure notifications based on severity thresholds to ensure timely alerting for critical metrics.