Ingest OTEL Logs with Grafana Alloy & Parseable

Logs are critical components of an application. They help maintain system reliability, performance and most of the time also help understand user behaviors, uncover patterns in customer actions and much more. Problem however in managing logs is manifold - missing structure, ad-hoc formats and the sheer volumes making it difficult to manage and analyze.

OpenTelemetry (aka OTEL) is a community initiative to help standardize collection, querying and visualization of logs, traces and metrics. With a well defined standard for logs, users can choose the log processing system that best fits their needs.

For example, in earlier stages of their company's lifecycle the customer can choose SaaS based systems so they can move fast. As the company grows and the volume of logs increases, they can move to a solution that let's them keep their data longer and offers much better access and control over analysis. With OTEL - this becomes much easier because there is not format change needed as and when companies move from one system to another.

In this blog post, we'll see how to use the latest entrant in the OTEL ecosystem - Grafana Alloy to forwards logs from our log generator application and export them to Parseable.

Parseable helps you store, query, and analyze large volumes of log data. By using Grafana Alloy as an OTEL Collector and Ingesting Logs to Parseable, you can create a robust log data stack that enhances your ability to monitor and troubleshoot your application. Let's start with brief overview of the OTEL ecosystem.

Understanding OTEL ecosystem

OTEL is a collection of specifications, and corresponding implementations as collectors, SDKs and exporters (available in all popular programming languages). One of the key goals of the OTEL ecosystem is to ensure the specification remains open to implementation by vendors - ensuring vendor neutrality.

OpenTelemetry Collectors are the deployed and used to receive and export this telemetry data. They collect, process, and export data from multiple sources which makes it super easy to integrate with multiple backend application.

Architecture overview

Let's Dive into the components we'll be using in this blog.

• Log Generator Application will generate random Opentelemetry Logs in every 3 seconds and send it forward to the Grafana Alloy.

• Grafana Alloy is a vendor-neutral distribution of the OpenTelemetry (OTEL) Collector. Alloy uniquely combines the very best OSS observability signals in the community. Note: Grafana Alloy is still in experimental stages.

• Parseable is a cloud native, log analytics platform, with a focus on performance & resource efficiency. Parseable is useful for use cases where complete data ownership, security and privacy are paramount.

Pre-requisites

• python 3.11 is installed.
• virtualenv is installed.
• Docker installed in your system. If not, follow this guide.

Log generating application

The first step in this whole journey is an application whose logs we're interested to capture. Ideally this would be a business critical application whose uptime, reliability and performance are all very important for success of the business. For demonstration purposes in this blog, we'll create a simple Python application that generates random logs every 5 seconds. We are using OTEL SDK to generates traces and Logs and send it to the OTELEXPORTER.

Follow the steps below to Create a log generator application,

Setup the log generator application

• If you don't have a Virtual environment, you can create one using the below command:

#Installing the virtualenv package
pip install virtualenv

# Using venv
python -m venv myenv

#Activate the Virtual Environment
source myenv/bin/activate

• Install the required libraries, open your terminal and type the command:

pip install opentelemetry-api==1.25.0 opentelemetry-sdk==1.25.0 opentelemetry-exporter-otlp==1.25.0 opentelemetry-exporter-otlp-proto-http==1.25.0

• Download the demo application code in this python file named app.py:

wget https://raw.githubusercontent.com/thecoderpanda/kubernetes-application-log-generator/main/application/app.py

Run the Sample log generator application

To Run this log generator application, open your terminal and type the command:

python app.py

Once you run the application, you should see logs being generated on your terminal.

The application should also be generating similar logs:

{
    "name": "foo",
    "context": {
        "trace_id": "0xfadd99f4c23680862b7dd9edcf51ba3a",
        "span_id": "0x39e290736b719d3a",
        "trace_state": "[]"
    },
    "kind": "SpanKind.INTERNAL",
    "parent_id": null,
    "start_time": "2024-05-20T23:17:42.117262Z",
    "end_time": "2024-05-20T23:17:42.117291Z",
    "status": {
        "status_code": "UNSET"
    },
    "attributes": {},
    "events": [],
    "links": [],
    "resource": {
        "attributes": {
            "telemetry.sdk.language": "python",
            "telemetry.sdk.name": "opentelemetry",
            "telemetry.sdk.version": "1.24.0",
            "service.name": "unknown_service"
        },
        "schema_url": ""
    }
}

Now that we have our logging application running, Let's now deploy grafana alloy!

Installing Grafana Alloy

Grafana Alloy is a vendor-agnostic OpenTelemetry Collector that does the heavy lifting of collecting, transforming and exporting of the OTEL data. There are multiple ways to install Grafana alloy. In this tutorial, we'll be installing it through MacOS client. To install Alloy on MacOS using homebrew, run the following commands in a terminal window.

• Add the Grafana Homebrew tap:

brew tap grafana/grafana

• Install Alloy:

brew install grafana/grafana/alloy

Once you follow the above steps, you have successfully deployed Alloy on MacOS. You can install it through multiple ways, you can find the tutorials here: https://grafana.com/docs/alloy/latest/get-started/install/

Our next step is to configure Grafana alloy to connect with our Application and start collecting the OpenTelemetry logs.

Configuring Grafana Alloy

We'll configure Grafana alloy to receive logs in at the default GrPc server and then process the logs and export them to our parseable instance. Let's create our Alloy Config file to perform the above actions.

• Edit the default configuration file, open your terminal window and navigate to the alloy directory:

cd /opt/homebrew/etc/alloy/

• Open the file using the nano editor:

nano config.alloy

• Now change the config settings with the one listed below, copy and paste the below in your config.alloy file:

wget https://raw.githubusercontent.com/thecoderpanda/kubernetes-application-log-generator/main/alloy.config

In the above config file, change the following:

• "X-P-Stream" = "testotel" Change testotel with your stream name
• endpoint = "http://localhost:8000" Change with your Parseable URL

note

Ingesting the OTEL Logs to parseable requires a flag to be passed in the header.

"X-P-log-Source" : "Otel"

• Run the following command in a terminal to restart your Grafana Alloy:

brew services restart alloy

To learn more about the configuration and installation, please refer to the Grafana alloy documentation.

We have now successfully deployed and configured grafana alloy on a MacOS system. Now, let's deploy Parseable!

Installing Parseable

We'll use the Quick Start guide to install Parseable locally. Before you begin, ensure that Docker is installed on your system. There are various methods to install Parseable, and you can explore them in the documentation here.

Open your terminal and type the command to deploy parseable locally:

docker run -p 8000:8000 \
-v /tmp/parseable/data:/parseable/data \
-v /tmp/parseable/staging:/parseable/staging \
-e P_FS_DIR=/parseable/data \
-e P_STAGING_DIR=/parseable/staging \
containers.parseable.com/parseable/parseable:latest \
parseable local-store

Once you run the above command, Parseable gets successfully installed and you can access it at http://localhost:8000. The default username is admin and password is admin. You can change these values in the by passing the credentials with P_USERNAME and P_PASSWORD values in the above command.

Finally, we need to create a log stream before we can send events. It is like a project which will essentially store all your logs that will be ingested. For this tutorial, we'll create a log stream named otelalloydemo. To create a log stream, login to Parseable instance and you'll find an button on the right-hand top side.

Once the log stream is created, let's check the status of our log ingestion in Parseable.

Final Configuration and Troubleshooting

Since our Log generator application will be generating random logs and Grafana Alloy will be receiving them, the final step is to ensure the ingestion is working.

The Parseable Dashboard should have the log ingested. To check, open Parseable instance - http://localhost:8000 and check the respective log stream.

note

Grafana Alloy is the latest release in the grafana product suite and is still experimental. Some of the features might not be operational. You can follow their documentation for more development.

Conclusion

This is how the Grafana Alloy <> Parseable integration works. This tutorial aims to create a simple example of this integration. We'll update this article as Grafana Alloy and Parseable progress. You can find all the configuration and code used in this tutorial on GitHub: https://github.com/thecoderpanda/kubernetes-application-log-generator. If you encounter any issues, please feel free to open an issue on the GitHub Repository. In addition to Grafana Alloy, Parseable allows you to ingest logs from multiple sources. We will soon release more tutorials on Parseable data ingestion.If you encounter any issues, please reach out to us on our Slack channel.

Happy Logging!