Agent Setup - Bronto

Generate an API Key

Before configuring your chosen log collection agent you will need to create an API Key. You will need to copy the API key to a secure location, because we do not make this available to you again, i.e. you cannot retrieve the full API key again after generating it.

You must have admin privileges to create API keys.

Choose Agent

Fluent Bit must be installed on the server hosting your logs to ensure efficient and reliable log transportation.

Use the following single line command to configure the Fluent Bit repository and install the Fluent Bit log collector.

curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh

Manual instructions

Add Server GPG Key

sudo apt update
sudo apt install ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://packages.fluentbit.io/fluentbit.key | sudo gpg --dearmor -o /etc/apt/keyrings/fluentbit.gpg
sudo chmod a+r /etc/apt/keyrings/fluentbit.gpg

Add Repo

In order to download the latest version of Fluent Bit, add the Fluent Bit repository to your source list.

echo  "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/fluentbit.gpg] https://packages.fluentbit.io/ubuntu/"$(. /etc/os-release && echo "$VERSION_CODENAME")"  "$(. /etc/os-release && echo "$VERSION_CODENAME")" main" |  sudo tee /etc/apt/sources.list.d/fluentbit.list > /dev/null

Install Package

Install the Fluent Bit from the repository setup in the previous step

sudo apt update
sudo apt install fluent-bit

Configure Fluent Bit

After installation, Fluent Bit requires an output configuration to communicate with the Bronto system.

[SERVICE]
    flush           10
    daemon          off
    log_level       info

[INPUT]
    name            tail
    path            /path/to/your/logs
    tag             <OPTIONAL_LOG_TAG>
    refresh_interval 5
    mem_buf_limit   5MB
    skip_long_lines on

[OUTPUT]
    name            http
    tls             on
    match           *
    host            ingestion.<REGION>.bronto.io
    port            443
    format          json_lines
    compress        gzip
    header          x-bronto-api-key <YOUR_API_KEY>
    header          x-bronto-service-name <YOUR_SERVICE_NAME>
    header          x-bronto-service-name-namespace <YOUR_SERVICE_NAMESPACE>
    header          x-bronto-tags: <key1=value1,key2=value2>

Parameter	Value	Requirement	Description
name	HTTP	Required	Specifies the output plugin used by Fluent Bit.
tls	on	Required	Enables the use of TLS for secure communication.
host	ingestion<REGION>.bronto.io	Required	The Bronto endpoint responsible for ingesting log data. Supported regions: EU & US.
port	443	Required	Specifies the network port for communication.
format	json_lines or msgpack	Required	Specify the data format to be used in the HTTP request body, by default it uses msgpack.
compress	gzip	Optional	Recommended. Compresses the payload in GZIP format. Bronto supports and recommends using gzip for efficient data transmission.
header (x-bronto-service-name)	Your Service Name	Required	Specifies the name of the service.
header (x-bronto-service-namespace)	Your Service Namespace	Optional	Specifies the namespace for organizational purposes.
header (x-bronto-tags)	Tags	Optional	Specifies the tags to apply to logs. A tag is a key-value pair applied to a log to hold metadata about that log. Each tag is a label consisting of a key and a required value.

Verify Log Collection

Once you have applied your configuration and restarted Fluent Bit, you can expect to see your log data being ingested to Bronto and accessible via the Search page.

Use the following single line command to configure the Fluent Bit repository and install the Fluent Bit log collector.

curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh

Manual instructions

Add Server GPG Key

sudo apt update
sudo apt install ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://packages.fluentbit.io/fluentbit.key | sudo gpg --dearmor -o /etc/apt/keyrings/fluentbit.gpg
sudo chmod a+r /etc/apt/keyrings/fluentbit.gpg

Add Repo

In order to download the latest version of Fluent Bit, add the Fluent Bit repository to your source list.

echo  "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/fluentbit.gpg] https://packages.fluentbit.io/ubuntu/"$(. /etc/os-release && echo "$VERSION_CODENAME")"  "$(. /etc/os-release && echo "$VERSION_CODENAME")" main" |  sudo tee /etc/apt/sources.list.d/fluentbit.list > /dev/null

Install Package

Install the Fluent Bit from the repository setup in the previous step

sudo apt update
sudo apt install fluent-bit

Configure Fluent Bit

After installation, Fluent Bit requires an output configuration to communicate with the Bronto system.

[SERVICE]
    flush           10
    daemon          off
    log_level       info

[INPUT]
    name            tail
    path            /path/to/your/logs
    tag             <OPTIONAL_LOG_TAG>
    refresh_interval 5
    mem_buf_limit   5MB
    skip_long_lines on

[OUTPUT]
    name            http
    tls             on
    match           *
    host            ingestion.<REGION>.bronto.io
    port            443
    format          json_lines
    compress        gzip
    header          x-bronto-api-key <YOUR_API_KEY>
    header          x-bronto-service-name <YOUR_SERVICE_NAME>
    header          x-bronto-service-name-namespace <YOUR_SERVICE_NAMESPACE>
    header          x-bronto-tags: <key1=value1,key2=value2>

Parameter	Value	Requirement	Description
name	HTTP	Required	Specifies the output plugin used by Fluent Bit.
tls	on	Required	Enables the use of TLS for secure communication.
host	ingestion<REGION>.bronto.io	Required	The Bronto endpoint responsible for ingesting log data. Supported regions: EU & US.
port	443	Required	Specifies the network port for communication.
format	json_lines or msgpack	Required	Specify the data format to be used in the HTTP request body, by default it uses msgpack.
compress	gzip	Optional	Recommended. Compresses the payload in GZIP format. Bronto supports and recommends using gzip for efficient data transmission.
header (x-bronto-service-name)	Your Service Name	Required	Specifies the name of the service.
header (x-bronto-service-namespace)	Your Service Namespace	Optional	Specifies the namespace for organizational purposes.
header (x-bronto-tags)	Tags	Optional	Specifies the tags to apply to logs. A tag is a key-value pair applied to a log to hold metadata about that log. Each tag is a label consisting of a key and a required value.

Verify Log Collection

Once you have applied your configuration and restarted Fluent Bit, you can expect to see your log data being ingested to Bronto and accessible via the Search page.

Use the following single line command to configure the Fluent Bit repository and install the Fluent Bit log collector.

curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh

Manual instructions

Add Repo

In order to download the latest version of Fluent Bit, add the Fluent Bit repository to your yum repo list.

echo -e \
"[fluent-bit]\n"\
"name = Fluent Bit\n"\
"baseurl = https://packages.fluentbit.io/centos/$(rpm -q --queryformat '%{VERSION}' centos-release)/\n"\
"gpgcheck=1\n"\
"gpgkey=https://packages.fluentbit.io/fluentbit.key\n"\
"repo_gpgcheck=1\n"\
"enabled=1\n" | \
sudo tee /etc/yum.repos.d/fluent-bit.repo > /dev/null

Install Package

Install the Fluent Bit from the repository setup in the previous step.

sudo yum install fluent-bit

Configure Fluent Bit

After installation, Fluent Bit requires an output configuration to communicate with the Bronto system.

[SERVICE]
    flush           10
    daemon          off
    log_level       info

[INPUT]
    name            tail
    path            /path/to/your/logs
    tag             <OPTIONAL_LOG_TAG>
    refresh_interval 5
    mem_buf_limit   5MB
    skip_long_lines on

[OUTPUT]
    name            http
    tls             on
    match           *
    host            ingestion.<REGION>.bronto.io
    port            443
    format          json_lines
    compress        gzip
    header          x-bronto-api-key <YOUR_API_KEY>
    header          x-bronto-service-name <YOUR_SERVICE_NAME>
    header          x-bronto-service-name-namespace <YOUR_SERVICE_NAMESPACE>
    header          x-bronto-tags: <key1=value1,key2=value2>

Parameter	Value	Requirement	Description
name	HTTP	Required	Specifies the output plugin used by Fluent Bit.
tls	on	Required	Enables the use of TLS for secure communication.
host	ingestion<REGION>.bronto.io	Required	The Bronto endpoint responsible for ingesting log data. Supported regions: EU & US.
port	443	Required	Specifies the network port for communication.
format	json_lines or msgpack	Required	Specify the data format to be used in the HTTP request body, by default it uses msgpack.
compress	gzip	Optional	Recommended. Compresses the payload in GZIP format. Bronto supports and recommends using gzip for efficient data transmission.
header (x-bronto-service-name)	Your Service Name	Required	Specifies the name of the service.
header (x-bronto-service-namespace)	Your Service Namespace	Optional	Specifies the namespace for organizational purposes.
header (x-bronto-tags)	Tags	Optional	Specifies the tags to apply to logs. A tag is a key-value pair applied to a log to hold metadata about that log. Each tag is a label consisting of a key and a required value.

Verify Log Collection

Once you have applied your configuration and restarted Fluent Bit, you can expect to see your log data being ingested to Bronto and accessible via the Search page.

To retrieve the latest Fluent Bit docker image use the following command.

docker pull fluent/fluent-bit:latest

Configure Fluent Bit

After installation, Fluent Bit requires an output configuration to communicate with the Bronto system.

[SERVICE]
    flush           10
    daemon          off
    log_level       info

[INPUT]
    name            tail
    path            /path/to/your/logs
    tag             <OPTIONAL_LOG_TAG>
    refresh_interval 5
    mem_buf_limit   5MB
    skip_long_lines on

[OUTPUT]
    name            http
    tls             on
    match           *
    host            ingestion.<REGION>.bronto.io
    port            443
    format          json_lines
    compress        gzip
    header          x-bronto-api-key <YOUR_API_KEY>
    header          x-bronto-service-name <YOUR_SERVICE_NAME>
    header          x-bronto-service-name-namespace <YOUR_SERVICE_NAMESPACE>
    header          x-bronto-tags: <key1=value1,key2=value2>

Parameter	Value	Requirement	Description
name	HTTP	Required	Specifies the output plugin used by Fluent Bit.
tls	on	Required	Enables the use of TLS for secure communication.
host	ingestion<REGION>.bronto.io	Required	The Bronto endpoint responsible for ingesting log data. Supported regions: EU & US.
port	443	Required	Specifies the network port for communication.
format	json_lines or msgpack	Required	Specify the data format to be used in the HTTP request body, by default it uses msgpack.
compress	gzip	Optional	Recommended. Compresses the payload in GZIP format. Bronto supports and recommends using gzip for efficient data transmission.
header (x-bronto-service-name)	Your Service Name	Required	Specifies the name of the service.
header (x-bronto-service-namespace)	Your Service Namespace	Optional	Specifies the namespace for organizational purposes.
header (x-bronto-tags)	Tags	Optional	Specifies the tags to apply to logs. A tag is a key-value pair applied to a log to hold metadata about that log. Each tag is a label consisting of a key and a required value.

Verify Log Collection

Once you have applied your configuration and restarted Fluent Bit, you can expect to see your log data being ingested to Bronto and accessible via the Search page.

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

brew install fluent-bit

Configurate Fluent Bit

After installation, Fluent Bit requires an output configuration to communicate with the Bronto system.

/opt/homebrew/etc/fluent-bit.conf

/usr/local/etc/fluent-bit.conf

[SERVICE]
    flush           10
    daemon          off
    log_level       info

[INPUT]
    name            tail
    path            /path/to/your/logs
    tag             <OPTIONAL_LOG_TAG>
    refresh_interval 5
    mem_buf_limit   5MB
    skip_long_lines on

[OUTPUT]
    name            http
    tls             on
    match           *
    host            ingestion.<REGION>.bronto.io
    port            443
    format          json_lines
    compress        gzip
    header          x-bronto-api-key <YOUR_API_KEY>
    header          x-bronto-service-name <YOUR_SERVICE_NAME>
    header          x-bronto-service-name-namespace <YOUR_SERVICE_NAMESPACE>
    header          x-bronto-tags: <key1=value1,key2=value2>

Parameter	Value	Requirement	Description
name	HTTP	Required	Specifies the output plugin used by Fluent Bit.
tls	on	Required	Enables the use of TLS for secure communication.
host	ingestion<REGION>.bronto.io	Required	The Bronto endpoint responsible for ingesting log data. Supported regions: EU & US.
port	443	Required	Specifies the network port for communication.
format	json_lines or msgpack	Required	Specify the data format to be used in the HTTP request body, by default it uses msgpack.
compress	gzip	Optional	Recommended. Compresses the payload in GZIP format. Bronto supports and recommends using gzip for efficient data transmission.
header (x-bronto-service-name)	Your Service Name	Required	Specifies the name of the service.
header (x-bronto-service-namespace)	Your Service Namespace	Optional	Specifies the namespace for organizational purposes.
header (x-bronto-tags)	Tags	Optional	Specifies the tags to apply to logs. A tag is a key-value pair applied to a log to hold metadata about that log. Each tag is a label consisting of a key and a required value.

Verify log collection

Once you have applied your configuration and restarted Fluent Bit, you can expect to see your log data being ingested to Bronto and accessible via the Search page.

To make Fluent Bit available on every node, it should be installed as a DaemonSet. The recommended way to deploy Fluent Bit is with the official Helm Chart: https://github.com/fluent/helm-charts

To add the Fluent Helm Charts repo use the following command

helm repo add fluent https://fluent.github.io/helm-charts

To validate that the repo was added you can run helm search repo fluent to ensure the charts were added. The default chart can then be installed by running the following

helm upgrade --install fluent-bit fluent/fluent-bit

The table below shows the default mappings and the annotations that can be used to organise your kubernetes logs in Bronto.

Parameter		Kubernetes Attribute		Override Annotation
bronto-service-name		container_name		service_name
bronto-service-namespace		namespace_name		service_namespace

Configure Fluent Bit

After installation, Fluent Bit requires an output configuration to communicate with the Bronto system.

[SERVICE]
    daemon             off
    flush              10
    log_level          info
    parsers_file       /fluent-bit/etc/parsers.conf
    parsers_file       /fluent-bit/etc/conf/custom_parsers.conf
    http_server        on
    http_listen        0.0.0.0
    http_port          2020
    health_check       on

[INPUT]
    name               tail
    path               /var/log/containers/*.log
    multiline.parser   docker, cri
    tag                kube.*
    mem_buf_limit      5MB
    skip_long_lines    on

[INPUT]
    name               systemd
    tag                host.*
    systemd_filter     _SYSTEMD_UNIT=kubelet.service
    read_from_tail     on

[FILTER]
    name                kubernetes
    match               kube.*
    merge_Log           on
    keep_log            off
    k8s-logging.parser  on
    k8s-logging.exclude on

[OUTPUT]
    name                http
    tls                 on
    match               kube.*
    host                ingestion.<REGION>.bronto.io
    port                443
    format              json_lines
    compress            gzip
    header              x-bronto-api-key <YOUR_API_KEY>

[OUTPUT]
    name                http
    tls                 on
    match               host.*
    host                ingestion.<REGION>.bronto.io
    port                443
    format              json_lines
    compress            gzip
    header              x-bronto-api-key <YOUR_API_KEY>

Parameter	Value	Requirement	Description
name	HTTP	Required	Specifies the output plugin used by Fluent Bit.
tls	on	Required	Enables the use of TLS for secure communication.
host	ingestion<REGION>.bronto.io	Required	The Bronto endpoint responsible for ingesting log data. Supported regions: EU & US.
port	443	Required	Specifies the network port for communication.
format	json_lines or msgpack	Required	Specify the data format to be used in the HTTP request body, by default it uses msgpack.
compress	gzip	Optional	Recommended. Compresses the payload in GZIP format. Bronto supports and recommends using gzip for efficient data transmission.
header (x-bronto-service-name)	Your Service Name	Required	Specifies the name of the service.
header (x-bronto-service-namespace)	Your Service Namespace	Optional	Specifies the namespace for organizational purposes.
header (x-bronto-tags)	Tags	Optional	Specifies the tags to apply to logs. A tag is a key-value pair applied to a log to hold metadata about that log. Each tag is a label consisting of a key and a required value.

Verify log collection

Once you have applied your configuration and restarted Fluent Bit, you can expect to see your log data being ingested to Bronto and accessible via the Search page.

Fluent Bit must be installed on the server hosting your logs to ensure efficient and reliable log transportation.

Use the following single line command to configure the Fluent Bit repository and install the Fluent Bit log collector.

curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh

Manual instructions

Add Server GPG Key

sudo apt update
sudo apt install ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://packages.fluentbit.io/fluentbit.key | sudo gpg --dearmor -o /etc/apt/keyrings/fluentbit.gpg
sudo chmod a+r /etc/apt/keyrings/fluentbit.gpg

Add Repo

In order to download the latest version of Fluent Bit, add the Fluent Bit repository to your source list.

echo  "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/fluentbit.gpg] https://packages.fluentbit.io/ubuntu/"$(. /etc/os-release && echo "$VERSION_CODENAME")"  "$(. /etc/os-release && echo "$VERSION_CODENAME")" main" |  sudo tee /etc/apt/sources.list.d/fluentbit.list > /dev/null

Install Package

Install the Fluent Bit from the repository setup in the previous step

sudo apt update
sudo apt install fluent-bit

Configure Fluent Bit

After installation, Fluent Bit requires an output configuration to communicate with the Bronto system.

[SERVICE]
    flush           10
    daemon          off
    log_level       info

[INPUT]
    name            tail
    path            /path/to/your/logs
    tag             <OPTIONAL_LOG_TAG>
    refresh_interval 5
    mem_buf_limit   5MB
    skip_long_lines on

[OUTPUT]
    name            http
    tls             on
    match           *
    host            ingestion.<REGION>.bronto.io
    port            443
    format          json_lines
    compress        gzip
    header          x-bronto-api-key <YOUR_API_KEY>
    header          x-bronto-service-name <YOUR_SERVICE_NAME>
    header          x-bronto-service-name-namespace <YOUR_SERVICE_NAMESPACE>
    header          x-bronto-tags: <key1=value1,key2=value2>

Parameter	Value	Requirement	Description
name	HTTP	Required	Specifies the output plugin used by Fluent Bit.
tls	on	Required	Enables the use of TLS for secure communication.
host	ingestion<REGION>.bronto.io	Required	The Bronto endpoint responsible for ingesting log data. Supported regions: EU & US.
port	443	Required	Specifies the network port for communication.
format	json_lines or msgpack	Required	Specify the data format to be used in the HTTP request body, by default it uses msgpack.
compress	gzip	Optional	Recommended. Compresses the payload in GZIP format. Bronto supports and recommends using gzip for efficient data transmission.
header (x-bronto-service-name)	Your Service Name	Required	Specifies the name of the service.
header (x-bronto-service-namespace)	Your Service Namespace	Optional	Specifies the namespace for organizational purposes.
header (x-bronto-tags)	Tags	Optional	Specifies the tags to apply to logs. A tag is a key-value pair applied to a log to hold metadata about that log. Each tag is a label consisting of a key and a required value.

Verify Log Collection

Once you have applied your configuration and restarted Fluent Bit, you can expect to see your log data being ingested to Bronto and accessible via the Search page.

Use the following single line command to configure the Fluent Bit repository and install the Fluent Bit log collector.

curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh

Manual instructions

Add Server GPG Key

sudo apt update
sudo apt install ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://packages.fluentbit.io/fluentbit.key | sudo gpg --dearmor -o /etc/apt/keyrings/fluentbit.gpg
sudo chmod a+r /etc/apt/keyrings/fluentbit.gpg

Add Repo

In order to download the latest version of Fluent Bit, add the Fluent Bit repository to your source list.

echo  "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/fluentbit.gpg] https://packages.fluentbit.io/ubuntu/"$(. /etc/os-release && echo "$VERSION_CODENAME")"  "$(. /etc/os-release && echo "$VERSION_CODENAME")" main" |  sudo tee /etc/apt/sources.list.d/fluentbit.list > /dev/null

Install Package

Install the Fluent Bit from the repository setup in the previous step

sudo apt update
sudo apt install fluent-bit

Configure Fluent Bit

After installation, Fluent Bit requires an output configuration to communicate with the Bronto system.

[SERVICE]
    flush           10
    daemon          off
    log_level       info

[INPUT]
    name            tail
    path            /path/to/your/logs
    tag             <OPTIONAL_LOG_TAG>
    refresh_interval 5
    mem_buf_limit   5MB
    skip_long_lines on

[OUTPUT]
    name            http
    tls             on
    match           *
    host            ingestion.<REGION>.bronto.io
    port            443
    format          json_lines
    compress        gzip
    header          x-bronto-api-key <YOUR_API_KEY>
    header          x-bronto-service-name <YOUR_SERVICE_NAME>
    header          x-bronto-service-name-namespace <YOUR_SERVICE_NAMESPACE>
    header          x-bronto-tags: <key1=value1,key2=value2>

Parameter	Value	Requirement	Description
name	HTTP	Required	Specifies the output plugin used by Fluent Bit.
tls	on	Required	Enables the use of TLS for secure communication.
host	ingestion<REGION>.bronto.io	Required	The Bronto endpoint responsible for ingesting log data. Supported regions: EU & US.
port	443	Required	Specifies the network port for communication.
format	json_lines or msgpack	Required	Specify the data format to be used in the HTTP request body, by default it uses msgpack.
compress	gzip	Optional	Recommended. Compresses the payload in GZIP format. Bronto supports and recommends using gzip for efficient data transmission.
header (x-bronto-service-name)	Your Service Name	Required	Specifies the name of the service.
header (x-bronto-service-namespace)	Your Service Namespace	Optional	Specifies the namespace for organizational purposes.
header (x-bronto-tags)	Tags	Optional	Specifies the tags to apply to logs. A tag is a key-value pair applied to a log to hold metadata about that log. Each tag is a label consisting of a key and a required value.

Verify Log Collection

Once you have applied your configuration and restarted Fluent Bit, you can expect to see your log data being ingested to Bronto and accessible via the Search page.

Use the following single line command to configure the Fluent Bit repository and install the Fluent Bit log collector.

curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh

Manual instructions

Add Repo

In order to download the latest version of Fluent Bit, add the Fluent Bit repository to your yum repo list.

echo -e \
"[fluent-bit]\n"\
"name = Fluent Bit\n"\
"baseurl = https://packages.fluentbit.io/centos/$(rpm -q --queryformat '%{VERSION}' centos-release)/\n"\
"gpgcheck=1\n"\
"gpgkey=https://packages.fluentbit.io/fluentbit.key\n"\
"repo_gpgcheck=1\n"\
"enabled=1\n" | \
sudo tee /etc/yum.repos.d/fluent-bit.repo > /dev/null

Install Package

Install the Fluent Bit from the repository setup in the previous step.

sudo yum install fluent-bit

Configure Fluent Bit

After installation, Fluent Bit requires an output configuration to communicate with the Bronto system.

[SERVICE]
    flush           10
    daemon          off
    log_level       info

[INPUT]
    name            tail
    path            /path/to/your/logs
    tag             <OPTIONAL_LOG_TAG>
    refresh_interval 5
    mem_buf_limit   5MB
    skip_long_lines on

[OUTPUT]
    name            http
    tls             on
    match           *
    host            ingestion.<REGION>.bronto.io
    port            443
    format          json_lines
    compress        gzip
    header          x-bronto-api-key <YOUR_API_KEY>
    header          x-bronto-service-name <YOUR_SERVICE_NAME>
    header          x-bronto-service-name-namespace <YOUR_SERVICE_NAMESPACE>
    header          x-bronto-tags: <key1=value1,key2=value2>

Parameter	Value	Requirement	Description
name	HTTP	Required	Specifies the output plugin used by Fluent Bit.
tls	on	Required	Enables the use of TLS for secure communication.
host	ingestion<REGION>.bronto.io	Required	The Bronto endpoint responsible for ingesting log data. Supported regions: EU & US.
port	443	Required	Specifies the network port for communication.
format	json_lines or msgpack	Required	Specify the data format to be used in the HTTP request body, by default it uses msgpack.
compress	gzip	Optional	Recommended. Compresses the payload in GZIP format. Bronto supports and recommends using gzip for efficient data transmission.
header (x-bronto-service-name)	Your Service Name	Required	Specifies the name of the service.
header (x-bronto-service-namespace)	Your Service Namespace	Optional	Specifies the namespace for organizational purposes.
header (x-bronto-tags)	Tags	Optional	Specifies the tags to apply to logs. A tag is a key-value pair applied to a log to hold metadata about that log. Each tag is a label consisting of a key and a required value.

Verify Log Collection

Once you have applied your configuration and restarted Fluent Bit, you can expect to see your log data being ingested to Bronto and accessible via the Search page.

To retrieve the latest Fluent Bit docker image use the following command.

docker pull fluent/fluent-bit:latest

Configure Fluent Bit

After installation, Fluent Bit requires an output configuration to communicate with the Bronto system.

[SERVICE]
    flush           10
    daemon          off
    log_level       info

[INPUT]
    name            tail
    path            /path/to/your/logs
    tag             <OPTIONAL_LOG_TAG>
    refresh_interval 5
    mem_buf_limit   5MB
    skip_long_lines on

[OUTPUT]
    name            http
    tls             on
    match           *
    host            ingestion.<REGION>.bronto.io
    port            443
    format          json_lines
    compress        gzip
    header          x-bronto-api-key <YOUR_API_KEY>
    header          x-bronto-service-name <YOUR_SERVICE_NAME>
    header          x-bronto-service-name-namespace <YOUR_SERVICE_NAMESPACE>
    header          x-bronto-tags: <key1=value1,key2=value2>

Parameter	Value	Requirement	Description
name	HTTP	Required	Specifies the output plugin used by Fluent Bit.
tls	on	Required	Enables the use of TLS for secure communication.
host	ingestion<REGION>.bronto.io	Required	The Bronto endpoint responsible for ingesting log data. Supported regions: EU & US.
port	443	Required	Specifies the network port for communication.
format	json_lines or msgpack	Required	Specify the data format to be used in the HTTP request body, by default it uses msgpack.
compress	gzip	Optional	Recommended. Compresses the payload in GZIP format. Bronto supports and recommends using gzip for efficient data transmission.
header (x-bronto-service-name)	Your Service Name	Required	Specifies the name of the service.
header (x-bronto-service-namespace)	Your Service Namespace	Optional	Specifies the namespace for organizational purposes.
header (x-bronto-tags)	Tags	Optional	Specifies the tags to apply to logs. A tag is a key-value pair applied to a log to hold metadata about that log. Each tag is a label consisting of a key and a required value.

Verify Log Collection

Once you have applied your configuration and restarted Fluent Bit, you can expect to see your log data being ingested to Bronto and accessible via the Search page.

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

brew install fluent-bit

Configurate Fluent Bit

After installation, Fluent Bit requires an output configuration to communicate with the Bronto system.

/opt/homebrew/etc/fluent-bit.conf

/usr/local/etc/fluent-bit.conf

[SERVICE]
    flush           10
    daemon          off
    log_level       info

[INPUT]
    name            tail
    path            /path/to/your/logs
    tag             <OPTIONAL_LOG_TAG>
    refresh_interval 5
    mem_buf_limit   5MB
    skip_long_lines on

[OUTPUT]
    name            http
    tls             on
    match           *
    host            ingestion.<REGION>.bronto.io
    port            443
    format          json_lines
    compress        gzip
    header          x-bronto-api-key <YOUR_API_KEY>
    header          x-bronto-service-name <YOUR_SERVICE_NAME>
    header          x-bronto-service-name-namespace <YOUR_SERVICE_NAMESPACE>
    header          x-bronto-tags: <key1=value1,key2=value2>

Parameter	Value	Requirement	Description
name	HTTP	Required	Specifies the output plugin used by Fluent Bit.
tls	on	Required	Enables the use of TLS for secure communication.
host	ingestion<REGION>.bronto.io	Required	The Bronto endpoint responsible for ingesting log data. Supported regions: EU & US.
port	443	Required	Specifies the network port for communication.
format	json_lines or msgpack	Required	Specify the data format to be used in the HTTP request body, by default it uses msgpack.
compress	gzip	Optional	Recommended. Compresses the payload in GZIP format. Bronto supports and recommends using gzip for efficient data transmission.
header (x-bronto-service-name)	Your Service Name	Required	Specifies the name of the service.
header (x-bronto-service-namespace)	Your Service Namespace	Optional	Specifies the namespace for organizational purposes.
header (x-bronto-tags)	Tags	Optional	Specifies the tags to apply to logs. A tag is a key-value pair applied to a log to hold metadata about that log. Each tag is a label consisting of a key and a required value.

Verify log collection

Once you have applied your configuration and restarted Fluent Bit, you can expect to see your log data being ingested to Bronto and accessible via the Search page.

To make Fluent Bit available on every node, it should be installed as a DaemonSet. The recommended way to deploy Fluent Bit is with the official Helm Chart: https://github.com/fluent/helm-charts

To add the Fluent Helm Charts repo use the following command

helm repo add fluent https://fluent.github.io/helm-charts

To validate that the repo was added you can run helm search repo fluent to ensure the charts were added. The default chart can then be installed by running the following

helm upgrade --install fluent-bit fluent/fluent-bit

The table below shows the default mappings and the annotations that can be used to organise your kubernetes logs in Bronto.

Parameter		Kubernetes Attribute		Override Annotation
bronto-service-name		container_name		service_name
bronto-service-namespace		namespace_name		service_namespace

Configure Fluent Bit

After installation, Fluent Bit requires an output configuration to communicate with the Bronto system.

[SERVICE]
    daemon             off
    flush              10
    log_level          info
    parsers_file       /fluent-bit/etc/parsers.conf
    parsers_file       /fluent-bit/etc/conf/custom_parsers.conf
    http_server        on
    http_listen        0.0.0.0
    http_port          2020
    health_check       on

[INPUT]
    name               tail
    path               /var/log/containers/*.log
    multiline.parser   docker, cri
    tag                kube.*
    mem_buf_limit      5MB
    skip_long_lines    on

[INPUT]
    name               systemd
    tag                host.*
    systemd_filter     _SYSTEMD_UNIT=kubelet.service
    read_from_tail     on

[FILTER]
    name                kubernetes
    match               kube.*
    merge_Log           on
    keep_log            off
    k8s-logging.parser  on
    k8s-logging.exclude on

[OUTPUT]
    name                http
    tls                 on
    match               kube.*
    host                ingestion.<REGION>.bronto.io
    port                443
    format              json_lines
    compress            gzip
    header              x-bronto-api-key <YOUR_API_KEY>

[OUTPUT]
    name                http
    tls                 on
    match               host.*
    host                ingestion.<REGION>.bronto.io
    port                443
    format              json_lines
    compress            gzip
    header              x-bronto-api-key <YOUR_API_KEY>

Parameter	Value	Requirement	Description
name	HTTP	Required	Specifies the output plugin used by Fluent Bit.
tls	on	Required	Enables the use of TLS for secure communication.
host	ingestion<REGION>.bronto.io	Required	The Bronto endpoint responsible for ingesting log data. Supported regions: EU & US.
port	443	Required	Specifies the network port for communication.
format	json_lines or msgpack	Required	Specify the data format to be used in the HTTP request body, by default it uses msgpack.
compress	gzip	Optional	Recommended. Compresses the payload in GZIP format. Bronto supports and recommends using gzip for efficient data transmission.
header (x-bronto-service-name)	Your Service Name	Required	Specifies the name of the service.
header (x-bronto-service-namespace)	Your Service Namespace	Optional	Specifies the namespace for organizational purposes.
header (x-bronto-tags)	Tags	Optional	Specifies the tags to apply to logs. A tag is a key-value pair applied to a log to hold metadata about that log. Each tag is a label consisting of a key and a required value.

Verify log collection

Once you have applied your configuration and restarted Fluent Bit, you can expect to see your log data being ingested to Bronto and accessible via the Search page.

Vector, a log collector and forwarding agent, must be installed on the server hosting your logs to ensure efficient and reliable log transportation.

Version 0.28.1 or later is supported - the latest version is preferred.

Use the following single line command to configure the Vector repository and install the Vector log collector.

curl --proto '=https' --tlsv1.2 -sSfL https://sh.vector.dev | bash

Configure Vector

After installation, a Vector http sink must be configured to communicate with Bronto.

sources:
  app_logs:
    type: file
    include:
      - /path/to/your/logs
sinks:
  bronto:
    type: "http"
    inputs: ["app_logs"]
    uri: "https://ingestion.<REGION>.bronto.io:443/"
    compression: "gzip"
    encoding:
      codec: "json"
    framing:
      method: "newline_delimited"
    method: "post"
    request:
      headers:
        x-bronto-api-key:  <YOUR_API_KEY>
        x-bronto-service-name: <YOUR_SERVICE_NAME>
        x-bronto-service-namespace:  <YOUR_SERVICE_NAMESPACE>
        x-bronto-tags: <key1=value1,key2=value2>

Use the following single line command to configure the Vector repository and install the Vector log collector.

curl --proto '=https' --tlsv1.2 -sSfL https://sh.vector.dev | bash

Configure Vector

After installation, a Vector http sink must be configured to communicate with Bronto.

sources:
  app_logs:
    type: file
    include:
      - /path/to/your/logs
sinks:
  bronto:
    type: "http"
    inputs: ["app_logs"]
    uri: "https://ingestion.<REGION>.bronto.io:443/"
    compression: "gzip"
    encoding:
      codec: "json"
    framing:
      method: "newline_delimited"
    method: "post"
    request:
      headers:
        x-bronto-api-key:  <YOUR_API_KEY>
        x-bronto-service-name: <YOUR_SERVICE_NAME>
        x-bronto-service-namespace:  <YOUR_SERVICE_NAMESPACE>
        x-bronto-tags: <key1=value1,key2=value2>

OpenTelemetry, a log collector and forwarding agent, must be installed on the server hosting your logs to ensure efficient and reliable log transportation.

For detailed installation instructions check out the OpenTelemetry documentation site.

Configure OpenTelemetry

Copy the below configuration to your open telemetry configuration file. On most linux distributions, this is located at /etc/otel/config.yaml.

receivers:
  filelog/<SERVICE_NAMESPACE>:
    include:
      - /path/to/your/logs
    resource:
      service.name: <YOUR_SERVICE_NAME>
      service.namespace: <YOUR_SERVICE_NAMESPACE>

processors:
  batch:

exporters:
  otlphttp/bronto:
    logs_endpoint: "https://ingestion.<REGION>.bronto.io/v1/logs"
    compression: none
    headers:
      x-bronto-api-key: <YOUR_API_KEY>

service:
  pipelines:
    logs:
      receivers: [filelog/<LOGSET_LOG>]
      processors: [batch]
      exporters: [otlphttp/bronto]
  # Useful for debugging    
  # telemetry:
    # logs:
      # level: "debug"
      # output_paths: [/path/to/debug.log]

Verify Log Collection

Once you have applied your configuration and restarted Fluent Bit, you can expect to see your log data being ingested to Bronto and accessible via the Search page.

Configure OpenTelemetry

Copy the below configuration to your open telemetry configuration file. On most linux distributions, this is located at /etc/otel/config.yaml.

receivers:
  filelog/<SERVICE_NAMESPACE>:
    include:
      - /path/to/your/logs
    resource:
      service.name: <YOUR_SERVICE_NAME>
      service.namespace: <YOUR_SERVICE_NAMESPACE>

processors:
  batch:

exporters:
  otlphttp/bronto:
    logs_endpoint: "https://ingestion.<REGION>.bronto.io/v1/logs"
    compression: none
    headers:
      x-bronto-api-key: <YOUR_API_KEY>

service:
  pipelines:
    logs:
      receivers: [filelog/<LOGSET_LOG>]
      processors: [batch]
      exporters: [otlphttp/bronto]
  # Useful for debugging    
  # telemetry:
    # logs:
      # level: "debug"
      # output_paths: [/path/to/debug.log]

Verify Log Collection

Once you have applied your configuration and restarted Fluent Bit, you can expect to see your log data being ingested to Bronto and accessible via the Search page.

Configure OpenTelemetry

Copy the below configuration to your open telemetry configuration file. On most linux distributions, this is located at /etc/otel/config.yaml.

receivers:
  filelog/<SERVICE_NAMESPACE>:
    include:
      - /path/to/your/logs
    resource:
      service.name: <YOUR_SERVICE_NAME>
      service.namespace: <YOUR_SERVICE_NAMESPACE>

processors:
  batch:

exporters:
  otlphttp/bronto:
    logs_endpoint: "https://ingestion.<REGION>.bronto.io/v1/logs"
    compression: none
    headers:
      x-bronto-api-key: <YOUR_API_KEY>

service:
  pipelines:
    logs:
      receivers: [filelog/<LOGSET_LOG>]
      processors: [batch]
      exporters: [otlphttp/bronto]
  # Useful for debugging    
  # telemetry:
    # logs:
      # level: "debug"
      # output_paths: [/path/to/debug.log]

Verify Log Collection

Once you have applied your configuration and restarted Fluent Bit, you can expect to see your log data being ingested to Bronto and accessible via the Search page.

Configure OpenTelemetry

Copy the below configuration to your open telemetry configuration file. On most linux distributions, this is located at /etc/otel/config.yaml.

receivers:
  filelog/<SERVICE_NAMESPACE>:
    include:
      - /path/to/your/logs
    resource:
      service.name: <YOUR_SERVICE_NAME>
      service.namespace: <YOUR_SERVICE_NAMESPACE>

processors:
  batch:

exporters:
  otlphttp/bronto:
    logs_endpoint: "https://ingestion.<REGION>.bronto.io/v1/logs"
    compression: none
    headers:
      x-bronto-api-key: <YOUR_API_KEY>

service:
  pipelines:
    logs:
      receivers: [filelog/<LOGSET_LOG>]
      processors: [batch]
      exporters: [otlphttp/bronto]
  # Useful for debugging    
  # telemetry:
    # logs:
      # level: "debug"
      # output_paths: [/path/to/debug.log]

Verify Log Collection

Once you have applied your configuration and restarted Fluent Bit, you can expect to see your log data being ingested to Bronto and accessible via the Search page.

Configure OpenTelemetry

Copy the below configuration to your open telemetry configuration file. On most linux distributions, this is located at /etc/otel/config.yaml.

receivers:
  filelog/<SERVICE_NAMESPACE>:
    include:
      - /path/to/your/logs
    resource:
      service.name: <YOUR_SERVICE_NAME>
      service.namespace: <YOUR_SERVICE_NAMESPACE>

processors:
  batch:

exporters:
  otlphttp/bronto:
    logs_endpoint: "https://ingestion.<REGION>.bronto.io/v1/logs"
    compression: none
    headers:
      x-bronto-api-key: <YOUR_API_KEY>

service:
  pipelines:
    logs:
      receivers: [filelog/<LOGSET_LOG>]
      processors: [batch]
      exporters: [otlphttp/bronto]
  # Useful for debugging    
  # telemetry:
    # logs:
      # level: "debug"
      # output_paths: [/path/to/debug.log]

Verify Log Collection

Once you have applied your configuration and restarted Fluent Bit, you can expect to see your log data being ingested to Bronto and accessible via the Search page.

Fluentd must be installed on the server hosting your logs to ensure efficient and reliable log transportation.

For detailed installation instructions check out the Fluentd documentation site.

Configurate FluentD

After installation, FluentD requires the output configuration to communicate with Bronto system.

<source>
  @type tail
  path /path/to/your/logs
  tag <OPTIONAL_LOG_TAG>
  refresh_interval 5s
  <parse>
    @type json
  </parse>
  pos_file /var/log/td-agent/buffer/fluentd.pos
</source>

<filter **>
  @type record_transformer
  <record>
    hostname "#{Socket.gethostname}"
  </record>
</filter>

<match **>
  @type http
  endpoint https://ingestion.<REGION>.bronto.io:443
  http_method post

  <buffer>
    @type file
    path /var/log/td-agent/buffer/http
    flush_interval 10s
    chunk_limit_size 5MB
    overflow_action block
  </buffer>

  <format>
    @type json
  </format>

  headers {"x-bronto-api-key": "<YOUR_API_KEY>","x-bronto-service-name": "<YOUR_SERVICE_NAME>","x-bronto-service-namespace": "<YOUR_SERVICE_NAMESPACE>"}
</match>

Parameter	Value	Required	Description
`source.@type`	`tail`	Yes	Type of source plugin; here, it follows file contents and collects logs.
`source.path`	`/path/to/your/logs`	Yes	Path to the log files to be tailed.
`source.tag`	`<OPTIONAL_LOG_TAG>`	No	Optional tag for the log entry.
`source.refresh_interval`	`5s`	No	Interval for refreshing the file list to check for new logs.
`source.parse`	`{ "@type": "json" }`	Yes	Specifies the parser type, here JSON format
`source.pos_file`	/var/log/td-agent/buffer/fluentd.pos	Yes	Position file to remember where it last read
`filter.@type`	`record_transformer`	Yes	Type of filter plugin to transform the log records
`filter.record`	`{ "hostname": "#{Socket.gethostname}" }`	Yes	Transformation record to add the hostname to the log entry
`match.@type`	`http`	Yes	Type of output plugin; here, to send logs via HTTP
`match.endpoint`	`https://ingestion.<REGION>.bronto.io:443`	Yes	HTTP endpoint to send the log data
`match.http_method`	`post`	Yes	HTTP method to use when sending data
`match.buffer.@type`	`file`	Yes	Configuration for buffering before sending.
`match.path`	`"/var/log/td-agent/buffer/http"`	Yes	Configuration for buffering before sending
`match.chunk_limit_size`	`5mb`	Yes	Configuration for buffering before sending logs
`match.flush_interval`	`10s`	No	Interval for flushing data from the buffer
`match.chunk_limit_size`	`5MB`	No	Maximum size of a data chunk before it’s flushed
`match.overflow_action`	`block`	No	Action to take when the buffer queue is full
`match.format`	`{ "@type": "json" }`	Yes	Specifies the format of data when being sent, here JSON format
`match.headers`	`{"x-bronto-api-key": "<YOUR_API_KEY>", "x-bronto-service-name": "<YOUR_SERVICE_NAME>", "x-bronto-service-namespace": "<YOUR_SERVICE_NAMESPACE>"}`	Yes	Headers to be sent along with HTTP requests

Verify Log Collection

Once you have applied your configuration and restarted Fluent Bit, you can expect to see your log data being ingested to Bronto and accessible via the Search page.

We recommend the use of Fluent-Bit or Vector but for migration we also support the datadog agent.

Configure Datadog Agent

The datadog agent can be configured to forward data to Bronto in addition to Datadog.

logs_config:
  use_http: true
  additional_endpoints:
  - api_key: "YOUR-API-KEY"
    host: "ingestion.<REGION>.bronto.io"
    port: 443

Parameter	Value	Requirement	Description
use_http	true	Required	Specifies the transport protocol to transfer log data, only http is supported
inputs	api_key	Required	Your Bronto API key
host	ingestion.<REGION>.bronto.io	Required	The Bronto endpoint responsible for ingesting log data. Supported regions are EU & US
port	443	Required	The Bronto endpoint port responsible for ingesting log data.

Verify Log Collection

Once you have applied your configuration and restarted the datadog agent, you can expect to see your log data being ingested to Bronto and accessible via the Search page.

We recommend the use of Fluent-Bit or Vector but for migration we also supported other agents like Logstash to help you migrate your logs.

Configurate Logstash

Create a new configuration file or update your existing to include the following:

input {
  file {
    id => "file_input_id"
    path => <YOUR_LOG_PATH>
  }
}

filter {
  grok {
    match => { "message" => "<%{NONNEGINT:pri}>%{NONNEGINT:version} +(?:%{TIMESTAMP_ISO8601:log.timestamp}|-) +(?:%{HOSTNAME:hostname}|-) +(?:-|%{NOTSPACE:app.name}) +(?:-|%{NOTSPACE:procid}) (?:-|%{NOTSPACE:msgid}) +(?:[%{DATA:structured.data}]|-|) +%{GREEDYDATA:message}"}
  }
}

output {
  http {
    url => "https://ingestion.<REGION>.bronto.io"
    id => "my_plugin_id"
    headers => {
      "x-bronto-api-key" => <YOUR_API_KEY>
      "x-bronto-service-name" => <YOUR_SERVICE_NAME>
      "x-bronto-service-namespace" => <YOUR_SERVICE_NAMEMESPACE>
      "Content-Type" => "text/html"
    }
    http_method => "post"
  }

 file {
   path => "test.log"
 }

}

Verify Log Collection

Once you have applied your configuration and restarted Fluent Bit, you can expect to see your log data being ingested to Bronto and accessible via the Search page.

Getting Started

Send Data

Search and Visualize

Monitors and Alerting

Usage Explorer

Account Management

​Generate an API Key

​Choose Agent

​Manual instructions

​Add Server GPG Key

​Add Repo

​Install Package

​Configure Fluent Bit

​Verify Log Collection

​Manual instructions

​Add Server GPG Key

​Add Repo

​Install Package

​Configure Fluent Bit

​Verify Log Collection

​Manual instructions

​Add Repo

​Install Package

​Configure Fluent Bit

​Verify Log Collection

​Configure Fluent Bit

​Verify Log Collection

​Configurate Fluent Bit

​Verify log collection

​Configure Fluent Bit

​Verify log collection

​Manual instructions

​Add Server GPG Key

​Add Repo

​Install Package

​Configure Fluent Bit

​Verify Log Collection

​Manual instructions

​Add Server GPG Key

​Add Repo

​Install Package

​Configure Fluent Bit

​Verify Log Collection

​Manual instructions

​Add Repo

​Install Package

​Configure Fluent Bit

​Verify Log Collection

​Configure Fluent Bit

​Verify Log Collection

​Configurate Fluent Bit

​Verify log collection

​Configure Fluent Bit

​Verify log collection

​Configure Vector

​Configure Vector

​Configure OpenTelemetry

​Verify Log Collection

​Configure OpenTelemetry

​Verify Log Collection

​Configure OpenTelemetry

​Verify Log Collection

​Configure OpenTelemetry

​Verify Log Collection

​Configure OpenTelemetry

​Verify Log Collection

​

​Configurate FluentD

​Verify Log Collection

​Configure Datadog Agent

​Verify Log Collection

​Configurate Logstash

​Verify Log Collection

Generate an API Key

Choose Agent

Manual instructions

Add Server GPG Key

Add Repo

Install Package

Configure Fluent Bit

Verify Log Collection

Manual instructions

Add Server GPG Key

Add Repo

Install Package

Configure Fluent Bit

Verify Log Collection

Manual instructions

Add Repo

Install Package

Configure Fluent Bit

Verify Log Collection

Configure Fluent Bit

Verify Log Collection

Configurate Fluent Bit

Verify log collection

Configure Fluent Bit

Verify log collection

Manual instructions

Add Server GPG Key

Add Repo

Install Package

Configure Fluent Bit

Verify Log Collection

Manual instructions

Add Server GPG Key

Add Repo

Install Package

Configure Fluent Bit

Verify Log Collection

Manual instructions

Add Repo

Install Package

Configure Fluent Bit

Verify Log Collection

Configure Fluent Bit

Verify Log Collection

Configurate Fluent Bit

Verify log collection

Configure Fluent Bit

Verify log collection

Configure Vector

Configure Vector

Configure OpenTelemetry

Verify Log Collection

Configure OpenTelemetry

Verify Log Collection

Configure OpenTelemetry

Verify Log Collection

Configure OpenTelemetry

Verify Log Collection

Configure OpenTelemetry

Verify Log Collection

Configurate FluentD

Verify Log Collection

Configure Datadog Agent

Verify Log Collection

Configurate Logstash

Verify Log Collection