Deployment & Installation > On-Prem Deployment & Installation > Deployment on Existing Kubernetes Cluster

Logbeat 8.1.3 – Linux Readme

Introduction

Logbeat is a lightweight shipper for forwarding and centralizing log data. Installed as an agent on your servers, Logbeat monitors the log files or locations that you specify, collects log events, and forwards them to the configured destination (typically vuSmartMaps shipper).

Pre-requisites

  • A sudo user with elevated privilege or a superuser access is required to create startup scripts during installation.
    cp (AGENT_HOME)/etc/init.d/logbeat /etc/init.d/logbeat
    
    cp (AGENT_HOME)/lib/systemd/system/logbeat.service /lib/systemd/system/logbeat.service
    
    sudo systemctl enable logbeat or sudo update-rc.d logbeat
  • To remove/disable the startup script during agent uninstallation.
    sudo systemctl disable logbeat
    
    rm /etc/init.d/logbeat
    
    rm /lib/systemd/system/logbeat.service
  • The agent does not need any superuser privileges for running, it can run as a normal user.
  • Firewalls, if any, should allow the outgoing TCP port of the vuSmartMaps data collector/shipper from the node on which this agent is installed.
  • To enable a non-privileged user (non root or  non sudo) to stop and start the service using systemctl or service commands, the following entries can be added to  /etc/sudoers.
    (USERNAME) ALL=NOPASSWD:/usr/bin/systemctl stop logbeat, /usr/bin/systemctl start logbeat, /usr/bin/systemctl reload logbeat , /usr/bin/systemctl restart logbeat, /usr/bin/systemctl status logbeat, /usr/bin/systemctl disable logbeat,  /bin/cp logbeat.service /lib/systemd/system/, /bin/cp logbeat /etc/init.d/, /sbin/service logbeat stop, /sbin/service logbeat start, /sbin/service logbeat restart, /sbin/service logbeat status, /sbin/service logbeat force-reload, /bin/rm /lib/systemd/system/logbeat.service, /bin/rm /etc/init.d/logbeat


    Note: (USERNAME) is the placeholder for the non root user running the agent.

  • Read access to the log file/path which has to be shipped.

Supported OS

  • Ubuntu – 16.04, 18.04, 20.04
  • RHEL – 7.x and above
  • CentOS – 7.x and above

Installation

  1. Extract the package and run the Logbeat installation script.
    bash ./install_logbeat.sh


    This will run through the installation wizard.

  2. Choose the installation option:
    1. Select “1” to install the agent for the first time
    2. Select “2” to upgrade the agent. The upgrade option will only update the binaries and leave the configurations intact.
  3. Enter to accept the default directory. If you want the agent to be installed elsewhere, please specify the directory path.

    Example,
    /home/$USER          - The installation will create Logbeat directory inside /home/$USER/
    
    /home/$USER/agents  - The agents directory should exist in the system. Installation will create Logbeat directory /home/$USER/agents/logbeat/
  4. The installation will ask to provide superuser / sudo access to create init.d/systemd services for the startup on system reboot.Enter [y] to accept this and provide the superuser / sudo username and password to proceed with.If you don’t have superuser / sudo access enter ‘n’ and skip the above steps. But, you have to run these steps manually with sudo access.

Managing Logbeat Service

Enabling services at startup manually

  1. Login with sudo user and run the below commands
    sudo 
    
    (AGENT_HOME)/logbeat/usr/share/logbeat/bin/enable_logbeat_service.sh
  2. Once the systemd or service commands are enabled you can start and stop the agent using the below command:
    systemctl start logbeat 
    
    systemctl stop logbeat
    
    systemctl status logbeat

    (OR)

    service logbeat start
    
    service logbeat stop
    
    service logbeat status


    Note:
    Please refer to the ‘Prerequisite’ ‘To enable a non-privileged user (non root or  non sudo) to stop and start the service using systemctl or service commands’

    If services are not enabled then you can manually start/stop the Logbeat using the below commands:

    (AGENT_HOME)/usr/share/logbeat/bin/logbeat.sh start
    
    (AGENT_HOME)/usr/share/logbeat/bin/logbeat.sh stop
    
    (AGENT_HOME)/usr/share/logbeat/bin/logbeat.sh status


    Note:

    • Use the above command to start/stop the agent if you have not enabled the systemd/service commands.
    • Also if you have started the service using systemctl/service then you have to stop the agent using systemctl/service commands.

Configuration

  1. The configuration file is available at the following location
    (AGENT_HOME)/etc/logbeat/logbeat.yml
  2. During the installation, the script will prompt for configuration of  agent, press y to configure the agent on the go or press n to to proceed with default configuration and you  have to manually configure it by updating

    (AGENT_HOME)/etc/logbeat/logbeat.yml
  3. The configuration will update the following attributes:
    • Shipper/Target IP  –  The remote vuSmartMaps system where the data is to be sent
    • PROTOCOL  – The protocol to be used to communicate to the remote machine.We support below protocols:
      • kafka
      • beats
    • PORT  – The port on which the agent has to be connected on the remote machine.
    • TOPIC  – The topic name where we have to send the data incase of kafka protocol.

    Note: In general, the agent installation is performed only after knowing the data input collection method/protocol and the listening port configured at the vuSmartMaps end. If this information is not known yet, please contact vuSmartMaps support team.

  4. The configuration file is available at the location (AGENT_HOME)/etc/logbeat/conf/logbeat.yml
  5. Change the shipper ip address and port in the yml file. Replace localhost with shipper ip address.
    hosts: ["localhost:5050"]


    For example if the shipper address is 192.168.9.1 and port number 5050, then configuration will be

    hosts: ["192.168.9.1:5050"]
  6. For kafka, the settings will look like below in the configuration file (logbeat.yml)
    #----------------------------- Kafka output --------------------------------
    
    output.kafka:
    
    #initial brokers for reading cluster metadata
    
    hosts: ["127.0.0.1:9092"]
    
    ###message topic selection + partitioning
    
    topic: '{KAFKA_TOPIC}'
    
    required_acks: 1
    
    compression: gzip
    
    max_message_bytes: 100000
    
    channel_buffer_size: 100000
    
    #key: '%{[message_key]}'
    
    #-----------------------------------------------------------------------------
  7. For logstash/beats, the settings will look like below in the configuration file
    #----------------------------- Logstash output --------------------------------
    
    output.logstash:
    
    # The Logstash hosts
    
    hosts: ["127.0.0.1:5040"]
    
    #ssl.enable: true
    
    #ssl.verification_mode: none
    
    
    
    
    # Optional SSL. By default it is off.
    
    # List of root certificates for HTTPS server verifications
    
    #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
    
    
    
    
    # Certificate for SSL client authentication
    
    #ssl.certificate: "/etc/pki/client/cert.pem"
    
    
    
    
    # Client Certificate Key
    
    #ssl.key: "/etc/pki/client/cert.key"
    
    #-------------------------------------------------------------------------------

Encryption of Agent Communication

vuSmartMaps uses 9094 for TLS and 9092 for plaintext communication from agent to broker.

To use encrypted communication with Kafka broker, the following steps should be performed.

For one-way SSL encryption, i.e. server authentication by the client, only one property needs to be added to the outputs.kafka section, i.e. certificate_authorities. This refers to the Root CA of the certificate used by the broker. In default vuSmartMaps installation, the broker is set up with a certificate signed by custom Root CA and this CA certificate is pre-packaged with the agent truststore. However, in case of a requirement to use your organization certificates, they have to be used during the install time of kafka broker while deploying vuSmartMaps and should also be added to agent truststore manually post agent installation.

ssl.certificate_authorities:["(path-to-client-truststore (.pem))"]

The certificate and key properties are required for client authentication. If client authentication is not required, no need to specify these properties.

ssl.certificate: "(path-to-client-certificate (.pem))"
ssl.key: "(path-to-client-key (.key))"

#If implementing only 1-way SSL
ssl.verification_mode: “none”

  • certificate_authorities – The list of root(CA) certificates for server verifications. If certificate_authorities is empty or not set, the trusted certificate authorities of the host system are used. Simple SSL encryption requires only 1-way authentication, i.e., server authentication. This is done by the CA which has signed and trusted the server’s public key (certificate).’
  • certificate – The path to the certificate for SSL client authentication. If the certificate is not specified, client authentication is not available.
  • key – The client certificate key used for client authentication. This option is required if a certificate is specified.

The server may be running a valid CA signed certificate or a self-signed certificate depending on the installation. In case CA signed certificate being used by the server, the client trust store must have the CA root certificate of the signing authority on its trust store. If a self-signed certificate is used by the server, the server certificate itself should be added to the trust store of the client.

Please note the agent uses PEM format for certificate_authorities.

Configuring Log Inputs for Shipping

You can manually configure the log inputs under (AGENT_HOME)/etc/logbeat/inputs.d/log-config.yml

You can also configure the log inputs by running configure_logs.sh  under (AGENT_HOME)/configure_logs.sh

For example:

cd /home/user/logbeat/

bash ./configure_logs.sh

This will run through the installation wizard:

Do you want to configure logs [Y|n]: y

Enter the unique name of log group [e.g. log, apache_access] [default will be log]: java-log      

Enter a valid log path [eg: /var/log/apache2/access-*.log ]: /var/log/java/java-exception.log

Do you want to configure another log path for this group [Y|n], [default is 'n' ]: n

Enter time for 'ignore_older' [default will comment this setting, which means all the log files will be read]: 1h

Are you reading the multiline pattern [Y|n] [default is 'n']: y

Enter the setting for multiline.negate [true/false] [default is true]: true

Enter the setting for multiline.match [before/after] [default is after]: after

Do you want to configure metadata [y/n] [default is 'n'] : y

Verify the configuration. Is this correct? [Y|n]: y

Do you want to configure another log group [y/n] [default is 'n' ]: n

Note: Understanding of YAML file format is required before you run the configure_logs.sh or edit the yml file.

 

YAML Format for Log Configuration

The list of inputs can be specified under the filebeat.inputs section of the logbeat.yml or in the inputs.d/log-config.yml

It is recommended you configure all the logs paths in inputs.d/

Inputs specify how Logbeat locates and processes input data.

The list is a YAML array, so each input begins with a dash (-).

You can specify multiple inputs or you can specify the same input ‘type‘ more than once.

Below is the example of inputs:

YAML Structure for Configuring Log Inputs

##-----------------------------------##

filebeat.inputs:                  #Note filebeat.inputs header all the log types will be configured under this section. 

                                  #If logs are configured under inputs.d/log-config.yml. It will directly start from "-type: log"




- type: log                       #input definition defining the type. The value 'log' in most of the cases

  enabled: true                   #enabled can be true or false. If true Logbeat harvests the log path. If 'false' disables the harvesting of the files.

                                  

  paths:                          #A list of glob-based paths that will be crawled and fetched.

    - /var/log/system.log         #log file path

    - /var/path2/wifi.log         #log file path




  ignore_older: 2h                 #ignore_older. If this option is enabled, Logbeat ignores any files that were modified before the specified timespan.                                                                   #Configuring ignore_older can be especially useful if you keep log files for a long time.                                                                                               #For example, if you want to start Logbeat, but only want to send the newest files and files from last week, you can configure this option.  




  fields:                         #custom fields. This is an optional setting.

    document_type: linux-group

    type: linux-logs              #metadata field name and value

    app_type: myapplication       #metadata field name and value

  fields_under_root: true         #if true will put all the metadata at the top level or root level in the output document instead of grouping it under                                                                   #fields sub-directory the output section




- type: log

  paths:

    - "/var/log/apache2/*"

  fields:

    document_type: apache-logs

    apache: true

  fields_under_root: true

##-----------------------------------##

In the above example:

Logbeat harvests lines from two files: system.log and wifi.log and uses fields configuration to add metadata type: linux-logs  and app_type: myapplication to the output.

Logbeat harvests lines from every file in the apache2 directory, and uses the fields configuration option to populate the metadata field apache: true to the output.

For the most basic configuration, define a single input with a single path. For example:

##-----------------------------------##

filebeat.inputs:

- type: log

  enabled: true

  paths:

    - /var/log/*.log

##-----------------------------------##

The input in this example harvests all files in the path /var/log/*.log, which means that Logbeat will harvest all files in the directory /var/log/ that end with .log.

To fetch all files from a predefined level of subdirectories, use this pattern: /var/log/*/*.log.  This fetches all .log files from the subfolders of /var/log. It does not fetch log files from the /var/log folder itself.

Fields

This allows you to define custom fields/metadata,  which are optional fields, sent in the log event to make it more meaningful or contextual.

For example, if you want to enrich the document with fields like Country, ApplicationType

For example:

Country: India

ApplicationType: log-processor

Exclude Lines

A list of regular expressions to match the lines that you want Logbeat to exclude. Logbeat drops any lines that match a regular expression in the list.

By default, no lines are dropped. Empty lines are ignored.

##-----------------------------------##

- type: log

  enabled: true

  paths:

    - /var/log/*.log

  exclude_lines: ['^DBG']

##-----------------------------------##

Include Lines

A list of regular expressions to match the lines that you want Logbeat to include. Logbeat exports only the lines that match a regular expression in the list. By default, all lines are exported. Empty lines are ignored.

##-----------------------------------##

- type: log

  enabled: true

  paths:

    - /var/log/*.log

  exclude_lines: ['^DBG']

##-----------------------------------##

Note:

If both include_lines and exclude_lines are defined, Logbeat executes include_lines first and then executes exclude_lines.

The order in which the two options are defined doesn’t matter. The include_lines option will always be executed before the exclude_lines option, even if exclude_lines appear before include_lines in the config file.

##-----------------------------------##

- type: log

  ...

  include_lines: ['sometext']

  exclude_lines: ['^DBG']

##-----------------------------------##

 

Handling Multiline Messages

The files harvested by Logbeat may contain messages that span multiple lines of text. For example, multiline messages are common in files that contain Java stack traces.

In order to correctly handle these multiline events, you need to configure multiline settings in the logbeat.yml or inputs.d/config-log.yml file to specify which lines are part of a single event.

Configuration Options

You can specify the following options in the filebeat.inputs section of the logbeat.yml config file to control how Logbeat deals with messages that span multiple lines.

The following example shows how to configure Logbeat to handle a multiline message where the first line of the message begins with a bracket ([).

##-----------------------------------##

- type: log

  enabled:true

  paths:

    -/var/log/java/error.log




  multiline.type: pattern    #default is pattern

  multiline.pattern: '^\['   #Specifies the regular expression pattern to match

  multiline.negate: true     #Defines whether the pattern is negated. The default is false.

  multiline.match: after     #Specifies how Logbeat combines matching lines into an event. The settings are after or before. 

                             #The behavior of these settings depends on what you specify for negate

##-----------------------------------##

In the above example, Logbeat takes all the lines that do not start with “[” and combines them with the previous line that does match. For example, you could use this configuration to join the following lines of a multiline message into a single event.

Sample log line for the above multiline pattern

[beat-logstash-some-name-832-2022.01.28] IndexNotFoundException[no such index]

    at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.resolve(IndexNameExpressionResolver.java:566)

    at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:133)

    at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:77)

    at org.elasticsearch.action.admin.indices.delete.TransportDeleteIndexAction.checkBlock(TransportDeleteI:ndexAction.java:75)

Note: Refer to the logbeat.reference.yml for reference.

Multiline Options Description

  1. multiline.type:  Defines which aggregation method to use. The default is the pattern. The other options are count which lets you aggregate a constant number of lines and while_pattern which aggregates lines by pattern without a match option.
  2. multiline.pattern: Specifies the regular expression pattern to match. See Regular expression support for a list of supported regexp patterns. Depending on how you configure other multiline options, lines that match the specified regular expression are considered either continuations of a previous line or the start of a new multiline event. You can set the negate option to negate the pattern.
  3. multiline.negate: Defines whether the pattern is negated. The default is false.
  4. multiline.match: Specifies how Logbeat combines matching lines into an event. The settings are after or before. The behavior of these settings depends on what you specify for negate. Below are some examples of negate and match:
    1. negate: false, match: after, pattern: ^b
      Result: Consecutive lines that match the pattern are appended to the previous line that doesn’t match.
      Example:
         _
      a   |
      b   | => abb

      b  _|

      c   |

      b   | => cbb

      b  _|

    2. negate: false, match: before, pattern: ^b
      Result: Consecutive lines that match the pattern are prepended to the next line that doesn’t match.
      Example:
         _
      b   |

      b   | => bba

      a  _|

      b   |

      b   | => bbc

      c  _|

    3. negate: true, match: after, pattern: ^b
      Result: Consecutive lines that don’t match the pattern are appended to the previous line that does match.
      Example:
         _

      b   |

      a   | => bac

      c  _|

      b   |

      d   | => bde

      e  _|

    4. negate: true, match: before, pattern: ^b
      Result: Consecutive lines that don’t match the pattern are prepended to the next line that does match.
      Example:
         _
      b   |

      a   | => acb

      c  _|

      b   |

      d   | => deb

      e  _|

  5. multiline.flush_pattern: Specifies a regular expression, in which the current multiline will be flushed from memory, ending the multiline-message. Work only with ‘pattern’ type
  6. multiline.max_lines: The maximum number of lines that can be combined into one event. If the multiline message contains more than max_lines, any additional lines are discarded. The default is 500.
  7. multiline.timeout: After the specified timeout, Logbeat sends the multiline event even if no new pattern is found to start a new event. The default is 5s.
  8. multiline.count_lines: The number of lines to aggregate into a single event.
  9. multiline.skip_newline: When set, multiline events are concatenated without a line separator.

Examples of Multiline Configuration

  1. Combining a Java stack trace into a single event
  2. Combining C-style line continuations into a single event
  3. Combining multiple lines from time-stamped events
  4. Combining multiple lines from Application events

Java Stack Traces

Java stack traces consist of multiple lines, with each line after the initial line beginning with whitespace, as in this example:

#---------------
Exception in thread "main" java.lang.NullPointerException

at com.example.myproject.Book.getTitle(Book.java:16)

at com.example.myproject.Author.getBookTitles(Author.java:25)

at com.example.myproject.Bootstrap.main(Bootstrap.java:14)

#---------------

To consolidate these lines into a single event in Logbeat, use the following multiline configuration:

multiline.type: pattern
multiline.pattern: '^[[:space:]]'

multiline.negate: false

multiline.match: after

The above configuration merges any line that begins with whitespace up to the previous line.

Here’s a Java stack trace that presents a slightly more complex example:

#---------------
Exception in thread "main" java.lang.IllegalStateException: A book has a null property

at com.example.myproject.Author.getBookIds(Author.java:38)

at com.example.myproject.Bootstrap.main(Bootstrap.java:14)

Caused by: java.lang.NullPointerException

at com.example.myproject.Book.getId(Book.java:22)

at com.example.myproject.Author.getBookIds(Author.java:35)

... 1 more

#---------------

To consolidate these lines into a single event in Logbeat, use the following multiline configuration:

multiline.type: pattern
multiline.pattern: '^[[:space:]]+(at|\.{3})[[:space:]]+\b|^Caused by:'

multiline.negate: false

multiline.match: after

In this example, the pattern matches the following lines:

  • a line that begins with spaces followed by the word at or …
  • a line that begins with the words Caused by:

Combining C-style line continuations into a single event

Several programming languages use the backslash (\) character at the end of a line to denote that the line continues, as in this example:

#---------------
printf ("%10.10ld  \t %10.10ld \t %s\

%f", w, x, y, z );

#---------------

To consolidate these lines into a single event in Logbeat, use the following multiline configuration:

multiline.type: pattern
multiline.pattern: '\\
multiline.negate: false
multiline.match: before

This configuration merges any line that ends with the \ character with the line that follows.

Combining multiple lines from time-stamped events

Activity logs from services such as Elasticsearch/Logstash typically begin with a timestamp, followed by information on the specific activity, as in this example:

#---------------

[2022-07-24 10:49:13,389][INFO ][stack        ] [Jack] using [1] data paths, mounts [[/

(/dev/disk1)]], net usable_space [34.5gb], net total_space [118.9gb], types [hfs]

#---------------

To consolidate these lines into a single event in Logbeat, use the following multiline configuration:

multiline.type: pattern

multiline.pattern: '^\[[0-9]{4}-[0-9]{2}-[0-9]{2}'

multiline.negate: true

multiline.match: after

This configuration uses the negate: true and match: after settings to specify that any line that does not match the specified pattern belongs to the previous line.

 

Combining multiple lines from Application events

Sometimes your application logs contain events, that begin and end with custom markers, such as the following example:

#---------------

[2022-07-24 10:49:13,500] Start new event

[2022-07-24 10:49:13,500] Content of processing something

[2022-07-24 10:49:13,500] End event

#---------------

To consolidate this as a single event in Logbeat, use the following multiline configuration:

multiline.type: pattern

multiline.pattern: 'Start new event'

multiline.negate: true

multiline.match: after

multiline.flush_pattern: 'End event'

The flush_pattern option specifies a regex at which the current multiline will be flushed. If you think of the pattern option specifying the beginning of an event, the flush_pattern option will specify the end or last line of the event.

Warning Note:

This example will not work correctly if start/end log blocks are mixed with non-multiline logs, or if different start/end log blocks overlap with each other. For instance, some other log lines in the following example will be merged into a single multiline document because they neither match multiline.pattern nor multiline.flush_pattern, and multiline.negate is set to true.

#---------------

[2022-07-24 10:49:13,389] Start new event

[2022-07-24 10:49:13,395] Content of processing something

[2022-07-24 10:49:13,399] End event

[2022-07-24 10:50:14,389] Some other log

[2022-07-24 10:50:14,395] Some other log

[2022-07-24 10:50:14,399] Some other log

[2022-07-24 10:51:14,389] Start new event

[2022-07-24 10:51:14,395] Content of processing something

[2022-07-24 10:51:14,399] End event

#---------------
 

Logging

  • Logs are written in the following path:
    (AGENT_HOME)/var/log/logbeat/
  • The logging configurations like log level, log file rotation, etc can be modified in the logbeat.yml.

How to Run for Local Testing?

(AGENT_HOME)/usr/share/logbeat/bin/logbeat -e -v -c

(AGENT_HOME)/etc/logbeat/logbeat.yml

Uninstall

To uninstall navigate to  agent installation path

cd (AGENT_HOME)/

bash ./uninstall_logbeat.sh

For example: if healthbeat is installed under /home/agent/logbeat

cd /home/agent/logbeat

./uninstall_logbeat.sh

Uninstallation is now complete.

Resources

Browse through our resources to learn how you can accelerate digital transformation within your organisation.