Key Concepts

Authentication

Parseable API calls require Basic Authentication. You can find the username and password for your Parseable server in the environment variables P_USERNAME and P_PASSWORD. If not set, the default username and password is admin and admin. HTTP clients generate basic auth headers from the username and password.

In case you want to manually add the basic auth header, use the following command.

Copyecho -n '<username>:<password>' | base64

Then add the following HTTP header to the API call.

CopyAuthorization: Basic <output-from-previous-command>

You can also use OAuth2 tokens for authentication. Refer to the OIDC section for more details.

High Availability

Parseable can be deployed in a high-availability configuration. In this configuration, Parseable runs in a cluster of multiple ingestors and a query node, also serving as the main node. The ingestors can be load balanced in a standard round robin configuration. In case an ingestor fails, the load balancer will automatically route the requests to the healthy ingestors.

The query node is responsible for executing the queries and fetching the data from the storage. The query node can be scaled vertically to handle more queries. Refer to the High Availability section for more details.

Storage

Once the JSON payload data reaches the server, it is validated and parsed to a columnar Apache Arrow format in memory. Subsequent events are appended to the Arrow record batch in memory, and a copy is kept on disk (to prevent data loss). Finally, after a configurable duration, the Arrow record batch is converted to Parquet and then pushed to the S3 (or compatible) bucket.

Parquet on object storage is organized into prefixes based on stream name, date, and time. This ensures the server fetches specific dataset(s) based on the query time range. We're working on a compaction approach that will further compress and optimize the storage while ensuring queryable data at all times.

Modes

Parseable can use a drive (mount points) or S3 (and compatible) bucket as the backend storage. The storage mode can be configured while starting the Parseable server with the sub-commands local-store or s3-store, for drive or object storage, respectively.

Based on the storage mode, the server requires certain environment variables to be set. Refer to the environment variables section for more details.

We recommend the local-store mode for only development and testing purposes. Note that once the server is started, the storage mode can't be changed.

Compression

Parseable uses lz4 as the default compression algorithm for Parquet files. To choose a different compression algorithm, set the environment variable P_PARQUET_COMPRESSION_ALGO to one of the following - uncompressed, snappy, gzip, lzo, brotli, zstd.

You can even change the compression algorithm in run time. For example, to change the compression algorithm in an existing Parseable instance, stop the instance first. Next, set the environment variable P_PARQUET_COMPRESSION_ALGO to the desired compression algorithm and start the instance. The Parquet files created after the change will use the new compression algorithm. You can, in the meantime, continue to query the existing data seamlessly.

Storage manifest

For each day, Parseable creates a manifest file that contains the metadata of the Parquet files ingested for that day. The manifest file is stored in the same S3 bucket as the Parquet files. The query server uses the manifest file to filter out the relevant Parquet files based on the query filters and the time range.

Deep Dive