Fluent Bit

How to Set Up Fluent Bit to Ship Logs to Parseable | Parseable

Fluent Bit is a lightweight, and scalable logging and metrics processor and forwarder. It is one of the most popular choices in cloud and container environments. Fluent Bit agents can be configured to send logs to Parseable with HTTP output plugin and JSON output format.

This document explains how to set up Fluent Bit to ship logs to Parseable. We use Fluent Bit's Memory Metrics Input plugin as the source of logs and send the data to Parseable using the HTTP output plugin.

In this document we'll cover a simple Docker Compose based setup and a Kubernetes based deployment with Helm.

Docker Compose

Please ensure Docker Compose installed on your machine. Then run the following commands to set up Parseable and Fluent Bit.

mkdir parseable
cd parseable
wget https://www.parseable.com/fluentbit/fluent-bit.conf
wget https://www.parseable.com/fluentbit/docker-compose.yaml
docker-compose up -d

You can now access the Parseable dashboard on http://localhost:8000. You should see a log stream called fluentbitdemo populated with log data generated by the Memory Metrics Input plugin.

Kubernetes

Install Fluent Bit

We use the official Fluent Bit Helm chart. But, we'll use a modified values.yaml file, that contains the configuration for Fluent Bit to send logs to Parseable.

wget https://www.parseable.com/fluentbit/values.yaml
helm repo add fluent https://fluent.github.io/helm-charts
helm install fluent-bit fluent/fluent-bit --values values.yaml -n fluentbit --create-namespace

Let's take a deeper look at the Fluent Bit configuration in values.yaml. Here we use the kubernetes filter to enrich the logs with Kubernetes metadata. We then use the http output plugin to send logs to Parseable. Notice the Match section in the http output plugin. We use kube.* to match all logs from Kubernetes filter. With the header X-P-Stream fluentbitdemo, we tell Parseable to send the logs to the fluentbitdemo stream.

  filters: |
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Merge_Log           On
        Keep_Log            Off
        K8S-Logging.Parser  On
        K8S-Logging.Exclude On

  outputs: |
    [OUTPUT]
        Name http
        Match kube.*
        host parseable.parseable.svc.cluster.local
        http_User admin
        http_Passwd admin
        format json
        compress gzip
        port 80
        header Content-Type application/json
        header X-P-META-meta1 value1
        header X-P-TAG-tag1 value1
        header X-P-Stream fluentbitdemo
        uri /api/v1/ingest
        json_date_key timestamp
        json_date_format iso8601

Batching and Compression

Parseable supports batching and compressing the log data before sending it via HTTP POST. Fluent Bit supports this feature via the compress and buffer_max_size option. We recommend enabling both of these options to reduce the number of HTTP requests and to reduce the size of the HTTP payload.

Check logs in Parseable

If you've not already done so, port-forward Parseable service to access the dashboard with:

kubectl port-forward svc/parseable 8000:80 -n parseable

You can now check the Parseable server fluentbitdemo stream to see the logs from this setup.

Updated on
Log Agents
Vector