Over the last few months, we got the opportunity to speak (and work) with several observability practitioners at large Enterprises. One of the common challenges across the table was the RoI on their observability investment.
Second most common challenge was tool sprawl. We saw that OSS is widely adopted in the observability ecosystem, but it led to wide variety of tools running for different use cases and hence tool sprawl. The challenge manifests as added overhead and complexity in terms of day 2 operations, it also adds friction in getting the larger picture while debugging an issue.
For several high performing SRE teams this is a big problem. Fragmented logs, metrics, and traces scattered across multiple tools, dashboards, with DIY stacks just don’t cut it when systems fail at 3 AM.
Our research led to Prism, a new take on unified observability interface. Prism is built with the view that observability data as a whole needs to be seen from several lenses - in some cases you need a microscopic, needle in haystack view to debug, in other cases you need a bird’s eye view to understand the ongoing patterns, in other cases you need a to understand performance characteristics over a short period of time and so on.
Prism is the UI component of Parseable Observability Platform. It integrates natively with the Parseable high-performance observability database. It is built for SREs and developers who demand speed, precision, and efficiency.
Let’s zoom in
A huge chunk of the observability deployments are run on private infrastructure, within the organisation’s firewalls. This is generally because of security & privacy concerns or because of data volumes & related cost concerns.
With time, requirements evolve, tools evolve and of course new tools get adopted. All this leads to a mix of different tools, different views and added pressure of managing all these tools. It eventually looks like:
Logs: A traditional logging system, based on full or partial text search. This is the go to place for all logs. The interface looks like walls of text.
Metrics: Generally based on Prometheus or a compatible system, integrated with visualization system like Grafana. This is a well known, stable stack. But several teams struggle with the plain overhead of running this with GBs of data, leave alone today’s TBs worth data ingested in a day.
Traces: Traces have the least adoption, with almost always an in-house implementation
At Parseable, we’re tackling the tool sprawl problem with a unified observability platform, which is extremely simple to run and can be run on infrastructure of your choice.
At the core of this platform is ParseableDB - fast, diskless, S3 first observability database offering multi-modal interface for text search and SQL query. This enables efficient storage and effective retrieval of different types of telemetry signals.
Refer the ClickBench benchmarks here
But an observability platform is incomplete without a purpose built UI. This is where Prism comes in. While building Prism we’ve kept our focus on engineers who need to query, correlate, and act fast.
In this post we’ll give you an overview of what is Prism, its highlights, and what is next in the roadmap.
Overview
Prism provides a real-time snapshot of all ingested telemetry data, including data compression ratios, active alerts, total number of datasets and simple one click way to navigate to signals you’re interested in.
Let’s dive deeper into specific pages, their features and upcoming features.
Log explorer
The Log Explore page provides a intuitive, interactive interface for navigating ingested logs. Based on the selected dataset and time range, you get a real-time overview of ingestion activity. This helps identifying spikes and drops immediately.
Features
One click filters: You can filter log data in a dataset, based on well known log formats in a single click. In near future we’ll also add predefined filters based on log sources, hostnames etc.
Custom filters: For specific use cases you can always add filters for exactly the column
Save filters: Workflows are very important for SREs. With Prism you can save filters and load them as needed.
Auto detected log formats: The detected log format is displayed upfront, ensuring clarity on how logs are structured during ingestion.
Interactive rows: Each log entry in the table isn’t just a static record. You can
Dynamically add columns to the table based on any log column.
Include/Exclude key-value pairs directly into the active query.
Copy fields instantly for debugging or deeper analysis.
Log volumes visualization:
A time-based graph offers a live view of log ingestion trends.
You can zoom into specific time ranges by dragging the graph, making it easy to correlate spikes with system events.
Time-Synced Exploration: Jump between logs and metrics for the same time frame instantly.
On-demand text and search: ParseableDB is now multi-modal and supports a separate text search interface. The Search View in Prism utilizes the search API and is designed for sub second text search on indexed observability data. We’ll soon publish more about the search API and its design.
The indexing mechanism is designed to be on-demand. This means you can index certain columns of a specific dataset, that you can then search. Unlike traditional full-text search, which scans entire log datasets, Prism leverages structured indexing to return results in milliseconds, even when querying massive log volumes stored on S3.
Alerts
Prism offers real-time alerting on incoming events with flexible configuration. Each dataset can have multiple independently evaluated alerts that operate as dataset processors internally maintain state to track how often a rule evaluates to true.
Once a defined threshold is met, the alert fires and notifies the configured targets; when conditions are no longer met, it resolves and sends a resolution message.
You can send alerts to Webhooks, Slack, and Alert manager, with optional repeat settings and TLS configurations per target.
SQL Query Editor
For deep analysis, where you want to slice and dice the observability data as needed, we added the SQL editor to Prism. Here you get a clean, simple interface to run ad-hoc SQL queries as needed. Results are immediately visible in a table format.
Features
Smart Suggestions & Auto-Completion: As you type, key field suggestions appear automatically, reducing manual effort and query errors. Autocomplete helps construct precise searches without needing deep knowledge of the dataset schema.
Optimized Query Execution: Indexed fields are prioritized for ultra-fast look ups. Search operates over defined time ranges, ensuring focused queries and reducing unnecessary scans.
Near-Instant Results, Even at Scale: The query in the image (method: PUT) searched 900K+ records in just 370ms, demonstrating how Prism UI optimizes search performance. By focusing on structured queries rather than full-text scans, you get fast, relevant results, without the usual observability lag.
Datasets
The Datasets Page in Prism provides a structured view of all ingested telemetry data, offering insights into storage efficiency, real-time ingestion trends, and retention policies. Data formats for events sent to a dataset is auto-detected and categorized, streamlining observability management.
The three-dot menu at the end of each row allows users to:
View Dataset Info: Get detailed metadata and ingestion statistics.
Configure Indices: Optimize queries by defining structured indexing.
Manage Retention: Set data lifecycle policies for cost and compliance control.
Delete Dataset: Remove datasets when no longer needed.
Features
The Dataset Info Page provides a detailed breakdown of dataset properties, statistics, and source details, ensuring complete visibility into ingested telemetry data.
The Statistics Section offers a granular view of:
Total Events: The total number of logs or telemetry records ingested.
Ingested Size vs. Storage Size: Compares the raw data size with its compressed storage footprint, giving insights into compression efficiency.
Data Life Cycle: Tracks how much data is stored, deleted, or retained over time.
Prism automatically detects and displays source metadata, providing valuable context about where the data originates:
Distinct IPs: Lists the originating IP addresses sending data.
User Agents: Identifies the source applications or services generating logs, helping in debugging and analysis.
RBAC in Prism
Prism’s Role-Based Access Control (RBAC) system is designed for fine-grained access management, ensuring that users only see and interact with the data relevant to them.
You can create custom roles for fine grained access. Read more on Parseable RBAC here.
Users Management: The Users Page in Prism provides a list of all users with access to the platform, along with their authentication method and assigned roles.
Role Management: The Roles Page is where admins define custom roles with granular privileges for access control. Read more here.
Creating a New Role:
Enter Role Name: Define a new role name (e.g., new-user).
Select Privileges: Choose from available permissions:
Admin: Full control over all features.
Editor: Can modify datasets and configurations.
Writer: Can ingest data but not modify settings.
Reader: Read-only access to datasets and logs.
Ingestor: Only allowed to send telemetry data.
Conclusion
Whether you're debugging a microservice or analyzing system-wide trends during peak traffic hours, Prism is your go-to tool.
With Parseable, you get high-performance observability on S3 without the usual trade-offs. Because when systems misbehave at 3 AM, you deserve tools that work as fast as your brain does under pressure and maybe even faster!
What's Next?
We're actively working on exciting roadmap features, including correlation and dynamic dashboards! 🚀
If you love what we're building, show us some ❤️ by starring our repository, it keeps our team motivated to keep pushing for fast and seamless observability on S3!