Fluentd

Fluentd is an open-source data collector that plays a crucial role in data processing. It is designed to handle large volumes of data from various sources, making it an ideal tool for integrating with platforms like Parseable. This reference document will guide you through the process of using Fluentd to ingest data into Parseable, covering key configurations, plugins, and best practices.

Docker Compose

Please ensure Docker Compose installed on your machine. Then run the following commands to set up Parseable and Fluent Bit.

Copymkdir parseable
cd parseable
wget https://www.parseable.com/fluentd/fluentd.conf
wget https://www.parseable.com/fluentd/docker-compose.yaml
docker-compose up -d

You can now access the Parseable dashboard on http://localhost:8000. You should see a log stream called fluentbitdemo populated with log data generated by the Memory Metrics Input plugin.

Kubernetes

• Please ensure kubectl and helm installed and configured to access your Kubernetes cluster.

• Parseable installed on your Kubernetes cluster. Refer the Parseable Kubernetes documentation here: https://www.parseable.com/docs/installation/kubernetes-helm.

Install Fluent Bit

We use the official Fluent Bit Helm chart. But, we'll use a modified values.yaml file, that contains the configuration for Fluent Bit to send logs to Parseable.

Copywget https://www.parseable.com/fluentd/values.yaml
helm repo add fluent https://fluent.github.io/helm-charts
helm install fluentd fluent/fluentd --values values.yaml -n f

Fluentd Configuration

Let's take a deeper look at the Fluent Bit configuration in values.yaml. Here we use the kubernetes filter to enrich the logs with Kubernetes metadata. We then use the http output plugin to send logs to Parseable. Notice the Match section in the http output plugin. We use kube.* to match all logs from Kubernetes filter. With the header X-P-Stream fluentbitdemo, we tell Parseable to send the logs to the fluentbitdemo stream.

Copy# Fluentd configuration
config:
  system: |
    <system>
      log_level debug
    </system>

  ## Input configuration
  inputs: |
    <source>
      @type monitor_agent
      tag memory
      emit_interval 5
    </source>

  ## Output configuration
  outputs: |
    <match *>
      @type http
      host parseable.parseable.svc.cluster.local
      port 80
      path /api/v1/ingest
      
      <headers>
        X-P-META-meta1 value1
        X-P-TAG-tag1 value1
        X-P-Stream fluentbitdemo
      </headers>
      
      <format>
        @type json
        time_key timestamp
        time_format iso8601
      </format>
      
      <buffer>
        @type memory
        flush_interval 5s
      </buffer>
      
      <auth>
        method basic
        username admin
        password admin
      </auth>
    </match>

Check logs in Parseable

Port forward Parseable service to access the dashboard with:

Copykubectl port-forward svc/parseable 8000:80 -n parseable

You can now check the Parseable server fluentbitdemo stream to see the logs from this setup.

Navigate to the Parseable dashboard and verify that logs are being ingested correctly.