VuNet Query Language

VuNet Query Language (VQL) is a powerful text based query language designed for comprehensive data analysis. It supports a range of functions and advanced query syntax to enable users to interact with and analyze data effectively. VQL allows for case-sensitive and case-insensitive searches, pattern matching, logical conditions, and field-specific queries, making it a versatile solution for querying large datasets.

String Search

VQL supports string searches by allowing the user to add string in the form of “<value>” , i.e., strings values enclosed within double quotes

For example: log_group:”group1″. 

This searches for all the rows which contain the string `group1` in their log_group column.

Now in certain cases few characters enclosed in double quotes have to be escaped since they have a special meaning. Below is the list of such special characters:

– \ (back slashes)

– ” (double quotes)

We escape these characters by adding backslashes to it. So to search for `group\new` the user has to write `group\\new`. Similarly to search for `group”name` the user has to type `group\”new`

To represent new line characters users can use

\n. Ex: `group\nname`

To represent a tab users can use 

\t. Ex: `group\tname`

Token Search

Tokens are essentially a continuous sequence of alphanumeric values or characters. Token Search filters entries based on the provided value. It performs a case-insensitive search. The syntax for using Token Search is as follows:

field_name:value

The field parameter is optional. If it is not provided, the search will be performed only in the message field of the table. Please note that when searching in a table that does not contain a message field, you must specify the field parameter.

For example, to search for “Error” in the message field, use the following:

Error

Similarly, to search for “Error” in a field named severity, use the following:

severity:Error

Phrase Search

Phrase search refines results based on the specified phrase. It conducts a case-insensitive search and accommodates spaces and special characters. The syntax for using phrase search is as follows,

field_name:"phrase to be searched"

The field parameter is optional. If it is not provided, the search will be performed only in the message field of the table. Please note that when searching in a table that does not contain a message field, you must specify the field parameter.

For example, to search for “Error in VuBlock” in the message field, use the following:

“Error in VuBlock”

Similarly, to search for “log collector” in a field named vublock_name, use the following:

vublock_name:“log collector”

Please note that phrase search is currently not supported for any VQL functions like case(), starts() or ends().

Regex Filter

The Regex filter in Vunet Query Language (VQL) enables advanced pattern matching and text extraction within your data. Unlike standard filters that search for exact values or simple conditions, the Regex filter allows you to define complex patterns to capture a wide variety of entries, even those that do not conform to a simple string or exact match.

Syntax:

• Search in the message field:

  regex("pattern")

For example, to search for entries containing a 12-digit number, use the following: 

regex("\d{12}")

• Search in a specific field:
  

  field_name:regex("pattern")

For example, to search for a 12-digit number in the log_uuid field, use the following: 

log_uuid:regex("\d{12}")

Use Cases:

• Identify Specific Patterns: Use regex to identify specific types of errors, such as stack traces, exception logs, or custom error messages.
• Complex Pattern Matching: Filter logs based on varying formats, like different IP address structures, various date formats, or partial matches following a specific pattern.

Case Sensitive Search

By default, all VQL operations are case insensitive. To support case sensitivity users can use case-sensitive search which refines results based on exact matching and respects case sensitivity. The syntax for using case sensitive search is as follows,

field_name:case(value)

The field parameter is optional. If it is not provided, the search will be performed only in the message field of the table. Please note that when searching in a table that does not contain a message field, you must specify the field parameter.

For example, to search for “VuAlert” in the message field, use the following:

case(VuAlert)

Similarly, to search for “Linux” in a field named log_group, use the following:

log_group:case(Linux)

Prefix Search

Prefix search filters results based on values that start with the specified value. This search is case insensitive. The syntax for using prefix search is as follows,

field_name:starts(value)

The field parameter is optional. If it is not provided, the search will be performed only in the message field of the table. Please note that when searching in a table that does not contain a message field, you must specify the field parameter.

For example, to search for entries, whose message field starts with “err” , use the following:

starts(err)

This will yield results starting with ‘err’ within the message field.

Similarly, to search for entries, whose log_group field starts with “lin”, use the following:

log_group:starts(lin)

This will yield results starting with ‘lin’ in field log_group.

New Addition: VQL now supports searching for phrases within the starts function. This allows users to filter results based on values that start with a specific phrase, including spaces and special characters.

For example, to search for entries where timestamp starts with “2024-08-07,” use the following:

Suffix Search

Suffix search filters results based on values that end with the specified value. This search is case insensitive. The syntax for using suffix search is as follows,

field_name:ends(value)

The field parameter is optional. If it is not provided, the search will be performed only in the message field of the table. Please note that when searching in a table that does not contain a message field, you must specify the field parameter.

For example, to search for entries, whose message field ends with “ror”, use the following:

ends(ror)

This will yield results ending with ‘ror’ within the message field.

Similarly, to search for entries, whose log_group field ends with “lin”, use the following:

log_group:ends(ux)

This will yield results ending with ‘ux’ in field log_group.

New Addition: VQL now supports searching for phrases within the ends function. This enables users to filter results based on values that end with a specific phrase, accommodating spaces and special characters.

For example, to search for entries where the message field ends with “has empty bucket,” use the following:

Finding Entries with Provided Values

Match entries having one of the provided values and return them. This is case sensitive. The syntax is as follows,

field_name:in(value1, value2, …)

Returns entries containing any of the exact specified values. This function does not support phrases. The field parameter is optional. If it is not provided, the search will be performed only in the message field of the table. Please note that when searching in a table that does not contain a message field, you must specify the field parameter.

For example, to search for entries, whose severity field is either ‘error’ or ‘warning’, use the following:

severity:in(error, warning)

Searching for entries which has a non-null value

Returns entries where a specified field has a non-null value. The syntax is as follows,

field_name:exists

The field parameter is optional. If it is not provided, the filtering will be performed only in the message field of the table. Please note that when searching in a table that does not contain a message field, you must specify the field parameter.

For example, to retrieve entries that do not have a null value in the message field, use the following:

exists

Similarly, to retrieve entries that do not have a null value in the log_group field, use the following:

log_group:exists

💡Note: If we have to search for the word “exists”, we need to enclose it with double quotes.

Comparison Operators

Equals to

The equals to operator allows you to do precise filtering based on exact matches. The syntax for using equals to operator is as follows,

field_name:=value

The field parameter is mandatory.

For example, to find entries that match “9000” in a field named port, use the following:

port:=9000

Similarly, the equals to operator can be used to match string values. For example, to find entries that match “VuAlert” in the field module, use the following:

module:=VuAlert

Greater than

The greater-than operator enables filtering based on values that exceed a specified threshold. The syntax for using greater-than operator is as follows,

field_name:>threshold

The field parameter is mandatory. Here the threshold represents a numeric value. Note that this operator is supported only for fields of numeric type.

For example, to find entries in the field error_count where the value is greater than “20”,

error_count:>20

Less than

The less-than operator enables filtering based on values that are lower than a specified threshold. The syntax for using less-than operator is as follows,

field_name:<threshold

The field parameter is mandatory. Here the threshold represents a numeric value. Note that this operator is supported only for fields of numeric type.

For example, to find entries in the field error_count where the value is less than “20”,

error_count:<20

Greater than or equal to

The greater-than-equal-to operator enables filtering based on values that meet or exceed a specified threshold. The syntax for using the greater-than-equal-to operator is as follows,

field_name:>=threshold

The field parameter is mandatory. Here the threshold represents a numeric value. Note that this operator is supported only for fields of numeric type.

For example, to find entries in the field error_count where the value is greater-than-equal-to “20”,

error_count:>=20

Less than or equal to

The less-than-equal-to operator enables filtering based on values that meet or are lower than a specified threshold. The syntax for using less-than-equal-to operator is as follows,

field_name:<=threshold

The field parameter is mandatory. Here the threshold represents a numeric value. Note that this operator is supported only for fields of numeric type.

For example, to find entries in the field error_count where the value is less-than-equal-to “20”,

error_count:<=20

Filter for Entries within a Range

Filter entries containing values within a specified range. The syntax is as follows,

field_name:[start:end]

The field, start and end parameters are mandatory. Here, the start and end parameters specify the beginning and end of the specified range value and are numeric values. Note that this operator is supported only for fields of numeric type.

For example, to find entries in the field error_count where error count is between 0 and 5.

error_count:[0:5]

Negating a Query

To filter out entries by excluding a particular value, you can use the negation operator (~). The syntax is as follows,

~field_name:value
//Compatible with other functions and operators.
~field_name:case(value)
~field_name:>threshold

The field parameter is optional. If it is not provided, the operation will be performed only in the message field of the table. Please note that when searching in a table that does not contain a message field, you must specify the field parameter.

For example, to return entries that do not contain “success,” use the following,

~success

Similarly, to return entries where the log_group field does not start with “Lin,” use the following,

~log_group:starts(Lin)

Combining Multiple Queries

To combine multiple queries, use the AND/OR logical operators. An AND operation can be performed by using a blank space or a “+” between two queries. An OR operation can be performed using “|” between two queries.

Example of an AND operation:

log_group:Linux error_count:[0:5]
//This can also be written as follows
log_group:Linux + error_count:[0:5]

Example for an OR operation

log_group:Linux | module:=VuAlert

It is also possible to combine AND/OR operators to create more complex queries. For example,

log_group:Linux | module:=VuAlert + error_count:[0:5]

Please note that brackets are not currently supported, so operators are applied in the order they appear.

$__VQL() Macro

The $__VQL() macro is available to enhance query capabilities in data models. VuNet Query Language (VQL) can be utilized in data models to create filter conditions by specifying (VQL) queries within a $__VQL macro. Presently, it functions only within the Data Modelling Workspace – Write Query section, specifically designed for hyperscale datastores.

The $__VQL macro facilitates filtering and searching tables, with future plans for aggregation and ordering. To use this macro, it must be integrated into the WHERE clause within Hyperscale’s query. For example,

SELECT * from vulog WHERE $__VQL(case(Error) + django)

Users can also employ the $__dynamicVariable() macro within $__VQL for dynamic value population. Utilize the raw option for proper functionality. For example,

SELECT * from vulog WHERE $__VQL(case(Error) + $__dynamicVariable(server, django, 'raw'))