Deployment & Installation > On-Prem Deployment & Installation > Deployment on Existing Kubernetes Cluster
1. Getting Started with vuSmartMaps™
3. Console
5. Configuration
6. Data Management
9. Monitoring and Managing vuSmartMaps™
Logbeat is a lightweight shipper for forwarding and centralizing log data. Installed as an agent on your servers, Logbeat monitors the log files or locations that you specify, collects log events, and forwards them to the configured destination (typically vuSmartMaps shipper).
cp (AGENT_HOME)/etc/init.d/logbeat /etc/init.d/logbeat cp (AGENT_HOME)/lib/systemd/system/logbeat.service /lib/systemd/system/logbeat.service sudo systemctl enable logbeat or sudo update-rc.d logbeat
sudo systemctl disable logbeat rm /etc/init.d/logbeat rm /lib/systemd/system/logbeat.service
(USERNAME) ALL=NOPASSWD:/usr/bin/systemctl stop logbeat, /usr/bin/systemctl start logbeat, /usr/bin/systemctl reload logbeat , /usr/bin/systemctl restart logbeat, /usr/bin/systemctl status logbeat, /usr/bin/systemctl disable logbeat, /bin/cp logbeat.service /lib/systemd/system/, /bin/cp logbeat /etc/init.d/, /sbin/service logbeat stop, /sbin/service logbeat start, /sbin/service logbeat restart, /sbin/service logbeat status, /sbin/service logbeat force-reload, /bin/rm /lib/systemd/system/logbeat.service, /bin/rm /etc/init.d/logbeat
Note: (USERNAME) is the placeholder for the non root user running the agent.
bash ./install_logbeat.sh
This will run through the installation wizard.
/home/$USER - The installation will create Logbeat directory inside /home/$USER/ /home/$USER/agents - The agents directory should exist in the system. Installation will create Logbeat directory /home/$USER/agents/logbeat/
sudo (AGENT_HOME)/logbeat/usr/share/logbeat/bin/enable_logbeat_service.sh
systemctl start logbeat systemctl stop logbeat systemctl status logbeat
(OR)
service logbeat start service logbeat stop service logbeat status
Note: Please refer to the ‘Prerequisite’ ‘To enable a non-privileged user (non root or non sudo) to stop and start the service using systemctl or service commands’
If services are not enabled then you can manually start/stop the Logbeat using the below commands:
(AGENT_HOME)/usr/share/logbeat/bin/logbeat.sh start (AGENT_HOME)/usr/share/logbeat/bin/logbeat.sh stop (AGENT_HOME)/usr/share/logbeat/bin/logbeat.sh status
Note:
(AGENT_HOME)/etc/logbeat/logbeat.yml
Note: In general, the agent installation is performed only after knowing the data input collection method/protocol and the listening port configured at the vuSmartMaps end. If this information is not known yet, please contact vuSmartMaps support team.
hosts: ["localhost:5050"]
For example if the shipper address is 192.168.9.1 and port number 5050, then configuration will be
hosts: ["192.168.9.1:5050"]
#----------------------------- Kafka output --------------------------------
output.kafka:
#initial brokers for reading cluster metadata
hosts: ["127.0.0.1:9092"]
###message topic selection + partitioning
topic: '{KAFKA_TOPIC}'
required_acks: 1
compression: gzip
max_message_bytes: 100000
channel_buffer_size: 100000
#key: '%{[message_key]}'
#-----------------------------------------------------------------------------#----------------------------- Logstash output -------------------------------- output.logstash: # The Logstash hosts hosts: ["127.0.0.1:5040"] #ssl.enable: true #ssl.verification_mode: none # Optional SSL. By default it is off. # List of root certificates for HTTPS server verifications #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"] # Certificate for SSL client authentication #ssl.certificate: "/etc/pki/client/cert.pem" # Client Certificate Key #ssl.key: "/etc/pki/client/cert.key" #-------------------------------------------------------------------------------
vuSmartMaps uses 9094 for TLS and 9092 for plaintext communication from agent to broker.
To use encrypted communication with Kafka broker, the following steps should be performed.
For one-way SSL encryption, i.e. server authentication by the client, only one property needs to be added to the outputs.kafka section, i.e. certificate_authorities. This refers to the Root CA of the certificate used by the broker. In default vuSmartMaps installation, the broker is set up with a certificate signed by custom Root CA and this CA certificate is pre-packaged with the agent truststore. However, in case of a requirement to use your organization certificates, they have to be used during the install time of kafka broker while deploying vuSmartMaps and should also be added to agent truststore manually post agent installation.
ssl.certificate_authorities:["(path-to-client-truststore (.pem))"]
The certificate and key properties are required for client authentication. If client authentication is not required, no need to specify these properties.
ssl.certificate: "(path-to-client-certificate (.pem))"
ssl.key: "(path-to-client-key (.key))"
#If implementing only 1-way SSL
ssl.verification_mode: “none”
The server may be running a valid CA signed certificate or a self-signed certificate depending on the installation. In case CA signed certificate being used by the server, the client trust store must have the CA root certificate of the signing authority on its trust store. If a self-signed certificate is used by the server, the server certificate itself should be added to the trust store of the client.
Please note the agent uses PEM format for certificate_authorities.
You can manually configure the log inputs under (AGENT_HOME)/etc/logbeat/inputs.d/log-config.yml
You can also configure the log inputs by running configure_logs.sh under (AGENT_HOME)/configure_logs.sh
For example:
cd /home/user/logbeat/ bash ./configure_logs.sh
This will run through the installation wizard:
Do you want to configure logs [Y|n]: y Enter the unique name of log group [e.g. log, apache_access] [default will be log]: java-log Enter a valid log path [eg: /var/log/apache2/access-*.log ]: /var/log/java/java-exception.log Do you want to configure another log path for this group [Y|n], [default is 'n' ]: n Enter time for 'ignore_older' [default will comment this setting, which means all the log files will be read]: 1h Are you reading the multiline pattern [Y|n] [default is 'n']: y Enter the setting for multiline.negate [true/false] [default is true]: true Enter the setting for multiline.match [before/after] [default is after]: after Do you want to configure metadata [y/n] [default is 'n'] : y Verify the configuration. Is this correct? [Y|n]: y Do you want to configure another log group [y/n] [default is 'n' ]: n
Note: Understanding of YAML file format is required before you run the configure_logs.sh or edit the yml file.
The list of inputs can be specified under the filebeat.inputs section of the logbeat.yml or in the inputs.d/log-config.yml
It is recommended you configure all the logs paths in inputs.d/
Inputs specify how Logbeat locates and processes input data.
The list is a YAML array, so each input begins with a dash (-).
You can specify multiple inputs or you can specify the same input ‘type‘ more than once.
Below is the example of inputs:
##-----------------------------------## filebeat.inputs: #Note filebeat.inputs header all the log types will be configured under this section. #If logs are configured under inputs.d/log-config.yml. It will directly start from "-type: log" - type: log #input definition defining the type. The value 'log' in most of the cases enabled: true #enabled can be true or false. If true Logbeat harvests the log path. If 'false' disables the harvesting of the files. paths: #A list of glob-based paths that will be crawled and fetched. - /var/log/system.log #log file path - /var/path2/wifi.log #log file path ignore_older: 2h #ignore_older. If this option is enabled, Logbeat ignores any files that were modified before the specified timespan. #Configuring ignore_older can be especially useful if you keep log files for a long time. #For example, if you want to start Logbeat, but only want to send the newest files and files from last week, you can configure this option. fields: #custom fields. This is an optional setting. document_type: linux-group type: linux-logs #metadata field name and value app_type: myapplication #metadata field name and value fields_under_root: true #if true will put all the metadata at the top level or root level in the output document instead of grouping it under #fields sub-directory the output section - type: log paths: - "/var/log/apache2/*" fields: document_type: apache-logs apache: true fields_under_root: true ##-----------------------------------##
In the above example:
Logbeat harvests lines from two files: system.log and wifi.log and uses fields configuration to add metadata type: linux-logs and app_type: myapplication to the output.
Logbeat harvests lines from every file in the apache2 directory, and uses the fields configuration option to populate the metadata field apache: true to the output.
For the most basic configuration, define a single input with a single path. For example:
##-----------------------------------## filebeat.inputs: - type: log enabled: true paths: - /var/log/*.log ##-----------------------------------##
The input in this example harvests all files in the path /var/log/*.log, which means that Logbeat will harvest all files in the directory /var/log/ that end with .log.
To fetch all files from a predefined level of subdirectories, use this pattern: /var/log/*/*.log. This fetches all .log files from the subfolders of /var/log. It does not fetch log files from the /var/log folder itself.
This allows you to define custom fields/metadata, which are optional fields, sent in the log event to make it more meaningful or contextual.
For example, if you want to enrich the document with fields like Country, ApplicationType
For example:
Country: India ApplicationType: log-processor
A list of regular expressions to match the lines that you want Logbeat to exclude. Logbeat drops any lines that match a regular expression in the list.
By default, no lines are dropped. Empty lines are ignored.
##-----------------------------------## - type: log enabled: true paths: - /var/log/*.log exclude_lines: ['^DBG'] ##-----------------------------------##
A list of regular expressions to match the lines that you want Logbeat to include. Logbeat exports only the lines that match a regular expression in the list. By default, all lines are exported. Empty lines are ignored.
##-----------------------------------## - type: log enabled: true paths: - /var/log/*.log exclude_lines: ['^DBG'] ##-----------------------------------##
Note:
If both include_lines and exclude_lines are defined, Logbeat executes include_lines first and then executes exclude_lines.
The order in which the two options are defined doesn’t matter. The include_lines option will always be executed before the exclude_lines option, even if exclude_lines appear before include_lines in the config file.
##-----------------------------------## - type: log ... include_lines: ['sometext'] exclude_lines: ['^DBG'] ##-----------------------------------##
The files harvested by Logbeat may contain messages that span multiple lines of text. For example, multiline messages are common in files that contain Java stack traces.
In order to correctly handle these multiline events, you need to configure multiline settings in the logbeat.yml or inputs.d/config-log.yml file to specify which lines are part of a single event.
You can specify the following options in the filebeat.inputs section of the logbeat.yml config file to control how Logbeat deals with messages that span multiple lines.
The following example shows how to configure Logbeat to handle a multiline message where the first line of the message begins with a bracket ([).
##-----------------------------------## - type: log enabled:true paths: -/var/log/java/error.log multiline.type: pattern #default is pattern multiline.pattern: '^\[' #Specifies the regular expression pattern to match multiline.negate: true #Defines whether the pattern is negated. The default is false. multiline.match: after #Specifies how Logbeat combines matching lines into an event. The settings are after or before. #The behavior of these settings depends on what you specify for negate ##-----------------------------------##
In the above example, Logbeat takes all the lines that do not start with “[” and combines them with the previous line that does match. For example, you could use this configuration to join the following lines of a multiline message into a single event.
[beat-logstash-some-name-832-2022.01.28] IndexNotFoundException[no such index] at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.resolve(IndexNameExpressionResolver.java:566) at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:133) at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:77) at org.elasticsearch.action.admin.indices.delete.TransportDeleteIndexAction.checkBlock(TransportDeleteI:ndexAction.java:75)
Note: Refer to the logbeat.reference.yml for reference.
b _|
c |
b | => cbb
b _|
b | => bba
a _|
b |
b | => bbc
c _|
b |
a | => bac
c _|
b |
d | => bde
e _|
a | => acb
c _|
b |
d | => deb
e _|
Java stack traces consist of multiple lines, with each line after the initial line beginning with whitespace, as in this example:
#--------------- Exception in thread "main" java.lang.NullPointerException at com.example.myproject.Book.getTitle(Book.java:16) at com.example.myproject.Author.getBookTitles(Author.java:25) at com.example.myproject.Bootstrap.main(Bootstrap.java:14) #---------------
To consolidate these lines into a single event in Logbeat, use the following multiline configuration:
multiline.type: pattern multiline.pattern: '^[[:space:]]' multiline.negate: false multiline.match: after
The above configuration merges any line that begins with whitespace up to the previous line.
Here’s a Java stack trace that presents a slightly more complex example:
#--------------- Exception in thread "main" java.lang.IllegalStateException: A book has a null property at com.example.myproject.Author.getBookIds(Author.java:38) at com.example.myproject.Bootstrap.main(Bootstrap.java:14) Caused by: java.lang.NullPointerException at com.example.myproject.Book.getId(Book.java:22) at com.example.myproject.Author.getBookIds(Author.java:35) ... 1 more #---------------
To consolidate these lines into a single event in Logbeat, use the following multiline configuration:
multiline.type: pattern
multiline.pattern: '^[[:space:]]+(at|\.{3})[[:space:]]+\b|^Caused by:'
multiline.negate: false
multiline.match: after
In this example, the pattern matches the following lines:
Several programming languages use the backslash (\) character at the end of a line to denote that the line continues, as in this example:
#---------------
printf ("%10.10ld \t %10.10ld \t %s\
%f", w, x, y, z );
#---------------
To consolidate these lines into a single event in Logbeat, use the following multiline configuration:
multiline.type: pattern multiline.pattern: '\\ multiline.negate: false multiline.match: before
This configuration merges any line that ends with the \ character with the line that follows.
Activity logs from services such as Elasticsearch/Logstash typically begin with a timestamp, followed by information on the specific activity, as in this example:
#--------------- [2022-07-24 10:49:13,389][INFO ][stack ] [Jack] using [1] data paths, mounts [[/ (/dev/disk1)]], net usable_space [34.5gb], net total_space [118.9gb], types [hfs] #---------------
To consolidate these lines into a single event in Logbeat, use the following multiline configuration:
multiline.type: pattern
multiline.pattern: '^\[[0-9]{4}-[0-9]{2}-[0-9]{2}'
multiline.negate: true
multiline.match: after
This configuration uses the negate: true and match: after settings to specify that any line that does not match the specified pattern belongs to the previous line.
Sometimes your application logs contain events, that begin and end with custom markers, such as the following example:
#--------------- [2022-07-24 10:49:13,500] Start new event [2022-07-24 10:49:13,500] Content of processing something [2022-07-24 10:49:13,500] End event #---------------
To consolidate this as a single event in Logbeat, use the following multiline configuration:
multiline.type: pattern multiline.pattern: 'Start new event' multiline.negate: true multiline.match: after multiline.flush_pattern: 'End event'
The flush_pattern option specifies a regex at which the current multiline will be flushed. If you think of the pattern option specifying the beginning of an event, the flush_pattern option will specify the end or last line of the event.
Warning Note:
This example will not work correctly if start/end log blocks are mixed with non-multiline logs, or if different start/end log blocks overlap with each other. For instance, some other log lines in the following example will be merged into a single multiline document because they neither match multiline.pattern nor multiline.flush_pattern, and multiline.negate is set to true.
#--------------- [2022-07-24 10:49:13,389] Start new event [2022-07-24 10:49:13,395] Content of processing something [2022-07-24 10:49:13,399] End event [2022-07-24 10:50:14,389] Some other log [2022-07-24 10:50:14,395] Some other log [2022-07-24 10:50:14,399] Some other log [2022-07-24 10:51:14,389] Start new event [2022-07-24 10:51:14,395] Content of processing something [2022-07-24 10:51:14,399] End event #---------------
(AGENT_HOME)/var/log/logbeat/
(AGENT_HOME)/usr/share/logbeat/bin/logbeat -e -v -c (AGENT_HOME)/etc/logbeat/logbeat.yml
To uninstall navigate to agent installation path
cd (AGENT_HOME)/ bash ./uninstall_logbeat.sh
For example: if healthbeat is installed under /home/agent/logbeat
cd /home/agent/logbeat ./uninstall_logbeat.sh
Uninstallation is now complete.
Browse through our resources to learn how you can accelerate digital transformation within your organisation.
VuNet’s Business-Centric Observability platform, vuSmartMaps™ seamlessly links IT performance to business metrics and business journey performance. It empowers SRE and IT Ops teams to improve service success rates and transaction response times, while simultaneously providing business teams with critical, real-time insights. This enables faster incident detection and response.