Build Clickstream Data Infrastructure with Winston and Parseable

Clickstream data are a series of individual events. These individual events are sequenced to do analytics and draw a variety of insights. To build a dependable analytics system, you need to build reliable infrastructure and a pipeline to record each event. You can build a robust clickstream data infrastructure using React, Node.js, Winston, and Parseable.

Server side logging

Clickstream data can be logged either from the client side or server side. Is there a preferred mode for sending logs? Client side logging sends the data directly from the browser or from within the app. But companies have reported event loss up to 30% when sent from the frontend. This happens because different browsers, ad blockers, manufacturer settings, connectivity issues, etc hinder sending event logs consistently.

Server side logging is reliable. Servers comfortably handle a large number of requests per second and log them all in storage. Everytime a user interacts on the web page or on the app screen, it triggers an API to the backend. We use each API call to log user interaction to the clickstream database. Hence, you get comprehensive, reliable, and accurate clickstream and user behavior analytics when we log from the backend server.

But there is a category of events that can only be captured from the frontend. For example, mouse hover, scroll, and other user interactions. So, like everything else in life, there is a trade-off. Refer to our other article in this series to see how to log events from the frontend using React and Parseable.

In this article, we will focus on logging events from the backend. We will use Node.js, Winston, and Parseable to create a robust clickstream data infrastructure.

Auto capture clickstream

First, we need to capture user interaction. Auto capture script or library can capture all user events on the frontend. We will capture every event, fetch data, and create a log object to send it to the server.

Along with event data, we can also collect metadata. For example if you have more than one signup button, one in the header and another in the footer. In our example, we add custom attribute “data-component-name=“HeaderSignUpButton” to the button. This hook will also fetch custom-attributes and send it to the logger service along with event data.

Our logger service sends the log object to the backend. To keep it simple, we will send logs through the HTTP request. You can use HTTP request safely, but at a high volume we prefer to use a bi-directional API.

Getting started

We'll use Node.js, Winston, and Parseable to create a customer data infrastructure. Winston is the most popular Node.js logging library. It’s highly adaptable and offers flexible log formatting and transportation options. We use Winston to collect logs and store it in Parseable.

Parseable is a developer-friendly log storage and analytics tool. Like Winston, Parseable is adaptable and user-friendly. It supports dynamic schema so that you can send log objects without any restriction. It comes with a powerful SQL-based query engine, live tail, table data with filter, alerting and more features.

At a high level, these are the steps to capture clicks and send the log to the backend:

1. Create globalEventListener Hook to auto capture all events
2. Add global event listener at the root
3. Create a button with custom attributes
4. Send logs to the backend

Auto capture all events

Create globalEventListener Hook to auto capture all events

//useGlobalEventTracker.js

import { useEffect } from "react";
import { useLocation } from "react-router-dom";

import packageJson from "../../package.json";

const useGlobalEventListener = (loggerService) => {
  const logger = new loggerService();

  const location = useLocation();
  useEffect(() => {
    const captureClickEvent = (event) => {
      let element = event.target;
      // Traverse up the DOM tree until we find an element with the data-component-name attribute
      while (element && !element.dataset.componentName) {
        element = element.parentElement;
      }
      let componentName = "NA";
      if (element) {
        componentName = element.dataset.componentName;
      }

      const { innerText, className } = event.target;

      const metadata = {
        event: "click",
        component: componentName,
        class: className,
        content: innerText,
        appVersion: packageJson.version,
        url: location.pathname,
      };

      logger.trackEvent("click", metadata);
    };

    document.addEventListener("click", captureClickEvent);

    return () => {
      document.removeEventListener("click", captureClickEvent);
    };
  }, []);
};

export default useGlobalEventListener;

Add global event listener at the root

import Home from "../src/pages/Home";
import { LoggerService } from "./logger/logger-service";
import useGlobalEventListener from "./hooks/useGlobalEventTracker";

function App() {
  useGlobalEventListener(LoggerService);
  return <Home />;
}

export default App;

Create a button

Set custom attribute on the button to pass additional metadata. For example, we have a send event button in the header. Set the component name to the HeaderSendButton.

<button data-component-name="“HeaderSignUpButton”">Click to Send Event</button>

Send logs to the backend

Use Axios to make http request.

// logger-service.js

import axios from "axios";

const backendURL = "Your Backend URL"; // Replace with your backend

export class LoggerService {
  trackEvent(eventType, payload) {
    axios({
      method: "POST",
      baseURL: backendURL,
      url: `/logger`,
      headers: {
        "x-source": "ReactApp", // Your custom data (optional)
        "x-event-type": eventType, // Custom header params (optional)
      },
      data: payload,
    });
  }
}

Events from Node.js backend service to Parseable

Now that the backend server has the events, let's see how to send these event streams to Parseable. We will use Winston to log events to Parseable.

Create a Parseable transport service

As a pre-requisite, you'll need to have Parseable installed and running. Refer to the documentation here to install Parseable. Then, configure the transport:

• URL of your Parseable instance
• Username and password to access Parseable
• Stream name where you want to send these events

// parseable-transport.js

const axios = require("axios");
const Transport = require("winston-transport");

const BASE_URL = "https://demo.parseable.com"; //your parseable instance

const auth = Buffer.from("username:password").toString("base64"); //
const streamName = "clickstream"; 

class WinstonParseableTransport extends Transport {
  constructor(opts) {
    super(opts);
    this.name = opts.name || "customTransport";
  }

  log(info, callback) {
    console.log("info", info);
    const config = {
      method: "POST",
      baseURL: BASE_URL,
      url: `/api/v1/logstream/${streamName}`,
      headers: {
        Authorization: `Basic ${auth}`,
        "Content-Type": "application/json",
      },
      data: {},
    };

    axios(config)
      .then(function (response) {
        console.log(
          `Custom Transport:sent with status with code ${response.status}`
        );
      })
      .catch(function (error) {
        console.log("axios error", error);
      });
    callback();
  }
}

module.exports = WinstonParseableTransport;

Create a Winston logger instance to handle events and logs

// packages/logger.js

const { createLogger, format, transports } = require("winston");
const WinstonParseableTransport = require("./parseable-transport");

const logger = createLogger({
  level: "info",
  format: format.combine(
    format.timestamp({
      format: "YYYY-MM-DD HH:mm:ss",
    }),
    format.errors({ stack: true }),
    format.splat(),
    format.json()
  ),
  transports: [
    new transports.File({ filename: "error.log", level: "error" }),
    new transports.File({ filename: "combined.log" }),
    new WinstonParseableTransport({ name: "customTransport" }),
  ],
});

if (process.env.NODE_ENV !== "production") {
  logger.add(
    new transports.Console({
      format: format.combine(format.colorize(), format.simple()),
    })
  );
}

module.exports = logger;

Add an API Endpoint

Use this to process frontend events to the Parseable backend.

// Example: Logging events using Winston

const express = require("express");
const path = require("path");
const logger = require("./logger-utils/logger");

const app = express();
const port = 4444;

// Middleware
app.use(express.json());
app.use((req, res, next) => {
  // To handle global events on request (Optional)
  // logger.info(`${req.method} ${req.url}`);
  next();
});

// Handle React Frontend App rendering
app.use(express.static(path.join(__dirname, "../frontend/build")));

// API to handle log process
app.post("/logger", (req, res) => {
  const { body = {} } = req;
  const ip = req.ip;
  const userAgent = req.get("User-Agent");
  logger.info({
    ...body,
    userAgent,
    host: ip,
  });
  res.json({
    message: "Log Triggered successfully!",
  });
});

app.listen(port, () => {
  console.log(`Express app listening at http://localhost:${port}`);
});

Summary

We have implemented a react hook on frontend to auto-capture all user interactions and a logger service on the backend to send all these events to the server. At the server we collect and store all data using Winston and Parseable.

Use the Parseable console to your query data and visualize as a report.

The code is available on GitHub.