Redpanda is a Kafka compatible streaming data platform. It is a drop-in replacement for Apache Kafka.
This document explains how to set up Redpanda, along with Redpanda Connect to send logs to Parseable using Docker Compose as the deployment platform. We assume here that you have producer applications that are sending logs/events to Redpanda.
Prerequisites
Please ensure Docker Compose is installed on your machine.
Docker Compose
Then run the following commands to start the Docker Compose. It will deploy Redpanda, Redpanda Console, and Parseable (standalone).
mkdir redpanda-parseable
wget https://www.parseable.com/docs/redpanda/docker-compose.yaml
docker-compose up -d
Using the Redpanda Connect
Exec into the Redpanda instance by using the command:
docker exec -u root -it redpanda-0 /bin/bash
Once you are logged in, download the connect file using:
curl https://www.parseable.com/docs/redpanda/connect.yaml > connect.yaml
The connect.yaml file contains input and output configurations. Here Kafka is the input, i.e. we listen to a Kafka topic using Redpanda connect. The output is an HTTP Connector that will send data to Parseable. Execute the configuration setup in the Redpanda instance now using the command below:
rpk connect run connect.yaml
If correctly implemented, your terminal should show that the http_client is active and running! Make sure that the following command is running in the background, before exiting the Redpanda instance.
Send data to Redpanda topic
This step helps you test if everything is working as expected. In a production scenario, you'll have a log agent sending logs to the Redpanda topic.
Run the following commands to start a bash shell in the Redpanda container and post some sample data.
docker exec -it redpanda-0 /bin/bash
rpk topic produce redpandatest
The rpk command will open up a console, ready to accept events. Then paste the below data in the terminal.
{"reporterId": 8824, "reportId": 10000, "content": "Was argued independent 2002 film, The Slaughter Rule.", "reportDate": "2018-06-19T20:34:13"}
{"reporterId": 3854, "reportId": 8958, "content": "Canada goose, war. Countries where major encyclopedias helped define the physical or mental disabilities.", "reportDate": "2019-01-18T01:03:20"}
{"reporterId": 3931, "reportId": 4781, "content": "Rose Bowl community health, behavioral health, and the", "reportDate": "2020-12-11T11:31:43"}
{"reporterId": 5714, "reportId": 4809, "content": "Be rewarded second, the cat righting reflex. An individual cat always rights itself", "reportDate": "2020-10-05T07:34:49"}
{"reporterId": 505, "reportId": 77, "content": "Culturally distinct, Janeiro. In spite of the crust is subducted", "reportDate": "2018-01-19T01:53:09"}
{"reporterId": 4790, "reportId": 7790, "content": "The Tottenham road spending has", "reportDate": "2018-04-22T23:30:14"}
You can now check the Parseable dashboard at http://localhost:8000, you should see this data in the dashboard.