Parseable as logging target for MinIO Audit & Debug logs

As new architecture patterns like separation of compute and storage evolve, object storage is increasingly the first choice for applications to store primary (non-archival) data. This is in addition to the old use cases of archival and large blob storage.

MinIO is an open-source, AWS S3-compatible object storage platform, allowing developers to use one of the most common object-based storage options without vendor lock-in and with flexibility.

As production workloads move to MinIO, it’s critical for businesses to audit and track each and every access to their MinIO cluster. Additionally, debugging issues in a production cluster is critical for business continuity.

MinIO exposes both audit logs and its internal server logs via a simple REST API, that you can point to your logging infrastructure. This is perfectly suited to writing to external services optimized for logging, such as Parseable.

This post walks through a live setup of MinIO and Parseable on Kubernetes using helm charts, configuring the MinIO cluster to output audit and server logs to Parseable. Finally, it shows you how to build dashboards on Parseable using the MinIO audit logs. We use an example application that generates random text files of various sizes and saves them to MinIO while also writing logs to a Parseable instance. This allows you to benefit from features such as alerts and comprehensive analytics.

Start Parseable

The preferred way to start Parseable under Kubernetes is using the Helm chart. The Parseable documentation provides more detailed instructions.

Once the Parseable service is running, open the web UI http://localhost:8000, log in with the username and password defined in the parseable-env-secret file (admin/admin unless you changed anything), and create two new log streams: minio_log and minio_audit using the + Create Stream button.

Log streams are logical (and physical) collections of related log events. This MinIO example posts audit data to one log stream and error log data to another. While Parseable supports dynamic schema, splitting two fundamentally different data sets into different streams for access control, search, and other reasons makes sense.

Start MinIO

Download this Kubernetes YAML file that defines a namespace and pod for MinIO. There are a few values you need to change to suit your setup:

• Update kubernetes.io/hostname to match the node running your Kubernetes cluster. For example, docker-desktop.
• Create a base64-encoded string of the Parseable username and password for use as a basic authentication header and add it to the values for the MINIO_LOGGER_WEBHOOK_AUTH_TOKEN_PARSEABLE and MINIO_AUDIT_WEBHOOK_AUTH_TOKEN_PARSEABLE environment variables. If you haven’t changed the username and password, this is "Basic YWRtaW46YWRtaW4=".

The MINIO_LOGGER_WEBHOOK_ENDPOINT_PARSEABLE and MINIO_AUDIT_WEBHOOK_ENDPOINT_PARSEABLE variables use the two respective log streams you created in the last step, directing the two sources of MinIO log data between the two log streams.

With those changes made, send the configuration to Kubernetes to start MinIO:

kubectl apply -f minio-parseable.yaml
``

And forward the local 9000 port to port 9090 of the MinIO pod:

```shell
kubectl port-forward pod/minio 9000 9090 -n minio-parseable

Open the MinIO Web UI http://localhost:9090 and log in with minioadmin for the user name and password.

Then, open the Access Keys section on the right-hand User menu, and click Create access key +. Note the Access Key and Secret Key values and click the Create button.

Demo Application

As MinIO acts as Amazon S3-compatible storage, you need an application that generates files to connect to it.

Download this JavaScript file that uses the faker library to generate a random number of text files on each run and PUTs them into a MinIO bucket called js-test-bucket (which is created on the first run), deleting the local copy of the file.

After this, it GETs the same file and saves it elsewhere on the local file system.

These two operations simulate a more typical production application so that MinIO sends a reasonable mix of API data to Parseable

Before running the code in the Minio.Client object, change the endpoint property to match yours and add the access key values created earlier to the accessKey and secretKeyProperties.

Run the application several times and open the minio_audit stream in the Parseable Web UI. Unless the MinIO pod encounters any issues, the minio_log log stream is probably empty. You can see the JSON structure of each audit entry in the MinIO documentation.

The JSON structure has several objects, which Parseable flattens, for example, api > name as api_name.

Dashboards

The Parseable log stream view offers some ways of analyzing log data, but for quick and convenient views, Parseable also offers user-created dashboards and tiles in the dashboard that can use SQL queries and a variety of visualization options.

Find and create dashboards from the Dashboards tab. Click the + New Dashboard button, give it a name, and set the time range to the last 24 hours.

This example creates four dashboards, showing the following:

• The largest files uploaded.
• Operation count.
• Bucket activity.
• Data transferred.

You can find a JSON export of all the tiles and the dashboard in the dashboards folder of the example repository that you can import instead of creating a dashboard from scratch.

You can export dashboards and tiles using the share button on each tile or dashboard. For tiles you can also export PNGs, CSVs, and JSON files of the rendered data.

Largest Files Uploaded

Click the + Add Tile button, give it a name, and select the minio_audit stream. Add the following SQL into the Query field, which queries the fields that contain the file name and size, filtering to any that are larger than 20kb, and sorting the result by the size.

Select api_object, api_rx  from minio_audit where api_rx > 20000 order by api_rx desc LIMIT 10

Click the Edit Visualization button. Set the Type to Bar Chart, the X-Axis to api_object, and the Y-Axis to api_rx. Click Done and then Save Changes.

Operation Count

Click the + Add Tile button, give it a name, and select the minio_audit stream.

Add the following SQL into the Query field, which queries the field that contains the API call, groups them, and returns the total of each group.

select count(*) as count,api_name from minio_audit group by api_name LIMIT 100

Click the Edit Visualization button. Set the Type to Area Chart, the X-Axis to api_name and the Y-Axis to count. Click Done and then Save Changes.

Bucket Activity

Click the + Add Tile button, give it a name, and select the minio_audit stream.

Add the following SQL into the Query field, which queries the fields that contain the bucket name and the API operation, returning the count for each operation on the bucket.

select count(*) as count,api_bucket,api_name from minio_audit group by api_bucket,api_name LIMIT 100

Click the Edit Visualization button. Set the Type to Table. Click Done and then Save Changes.

Data Transferred

Click the + Add Tile button, give it a name, and select the minio_audit stream.

Add the following SQL into the Query field, which queries the field that contains the API name and the size for each operation, returning the total size for each operation.

select sum(api_rx) as sum,api_name from minio_audit group by api_name LIMIT 100

Click the Edit Visualization button. Set the Type to Line Chart, the X-Axis to api_name and the Y-Axis to sum. Click Done and then Save Changes.

Summary

MinIO is a powerful and flexible option for object storage that avoids vendor lock-in. As object stores become a more common and important component of a modern application stack, it’s important to ensure you have comprehensive audit logs of the storage as well as quick to see window into the logs. This post covered outputting MinIO audit and error logs to Parseable and creating four dashboards to highlight key information on the log data.