This document outlines the overall architecture of the Parseable Observability Platform, detailing the flow of MELT data from ingestion to storage and querying.
This document is organised into specific section for each sub-system like ingestion, query, search and index. To understand the specific decisions and trade-offs, refer the design choices document.
Overview
Parseable is shipped as a single unified binary (or container image if you prefer). This includes the Prism UI and Parseable DB. There is no additional dependency to run Parseable.
The binary can be run in different modes. You’d generally run standalone mode to test and experience Parseable on your laptop, or a small testing server.
As you move to a production setup, we recommend running the distributed mode. Here each node has a specific role, i.e. ingestion, query or search.
Ingestion
Parseable ingestion nodes are based on shared nothing architecture. This means each node is independently capable of ingesting events end to end. In a production environment, you’d set up a load balancer in front of two or more ingestion nodes. Ingestion request can then be routed to any of the ingestion nodes.
Once a node receives an ingestion request, the request is validated first. The node then converts the HTTP or Kafka protocol payload to Apache Arrow based file format and keeps it in a dedicated staging area. In this process it also validates the event schema. As the disk write is completed, the ingestion node responds with success.
We recommend a small, reliable disk (NFS or similar) attached for staging area to ensure data protection against disk failures.
A background job then reads the ingested Arrow files, converts those files to heavily compress Parquet files, and pushes these files to S3 or other object store as configured. In this process, the ingestion node also generates the metadata for this Parquet that helps during querying.
Query
Query node is primarily responsible responding to query API. This node also serves as the leader node in a Parseable cluster and hence it also responds to other API.
The query workflow starts when someone calls the query API with (a PostgreSQL compatible) SQL query, and a start and end timestamp. The query node looks up the metadata locally first, falling back to object store only if not found.
Based on metadata, the node identifies the relevant parquet files and uses the object store API to get these files. Here again, this only happens if the files are not already present locally. If the files are to be downloaded from object storage - this adds to latency and hence the occasional cold queries.
As the leader, query node also responds to all the role, user management, dataset management API.
