Decoupling Observability: A BI Blueprint for Agility

The following blog post is an analysis of the SE Radio episode "SE Radio 717: Eric Tschetter on Decoupling Observability," drawing insights from the conversation between host Amay Ambade and guest Eric Tschetter.

The hidden consequence of tightly coupled observability systems is a slow, inevitable slide into vendor lock-in, escalating costs, and organizational friction. This conversation reveals that the future of effective observability lies not in monolithic solutions, but in a decoupled, layered architecture mirroring the evolution of business intelligence. Teams that embrace this shift now gain a significant advantage: the agility to adapt, integrate new tools, and access a broader pool of data without costly migrations or learning new, proprietary systems. This post is for engineering leaders, architects, and developers grappling with the escalating complexity and cost of their current observability stacks, offering a roadmap to a more flexible and cost-effective future.

The Unseen Chains: Why Monolithic Observability Stifles Innovation

The allure of a single, integrated observability platform--one where logs, metrics, and traces live harmoniously within a vendor's ecosystem--is powerful. It promises simplicity: a unified interface, a single point of contact for support, and a seemingly streamlined data ingestion process. However, as Eric Tschetter explains, this apparent simplicity is a trap. This tightly coupled, vertically integrated approach, prevalent in solutions like Splunk, the ELK stack, Datadog, and Dynatrace, creates a "walled garden" that, over time, becomes a significant source of pain. The immediate benefit of ease of use quickly erodes, replaced by the hidden costs of vendor lock-in, prohibitive scaling expenses, and organizational gridlock.

When companies grow, acquire others, or simply have multiple teams with diverse needs, these monolithic stacks become bottlenecks. A new team might want to adopt a different, more suitable tool, but the existing infrastructure makes this nearly impossible without a massive, disruptive migration. This leads to a constant struggle between centralized mandates and team autonomy, fostering political battles and the creation of data silos.

"And so at especially large organizations, there might be some acquisitions. They might buy a company. That company knew nothing about the ivory, that company knew nothing about your new product. They just made decisions all on their own. And now you end up with this sprawl where each independent vertically integrated stack has some data in it. But when you want to know what's going on across all of them, it's hard."

This fragmentation prevents a holistic view of system health, forcing engineers to jump between disparate tools, each with its own query language and data model. The promise of observability--to understand and act upon system behavior--is undermined by the very tools designed to enable it. The core issue, Tschetter argues, is not a lack of technological evolution, but a timeline difference. Observability and security markets are where business intelligence was 20-30 years ago, still largely mired in vertically integrated solutions.

The BI Blueprint: Decoupling as the Path to Agility

The business intelligence (BI) world offers a compelling blueprint for how observability can evolve. Historically, BI systems were also monolithic. However, driven by the need for data portability and flexibility, they naturally decoupled into distinct layers: data acquisition (ETL), data storage, and visualization. This separation allowed for independent innovation and choice at each layer. For instance, an organization could swap out its ETL tool without impacting its visualization platform, or migrate its data warehouse to a new provider while retaining existing dashboards.

Tschetter posits that observability is on the same evolutionary trajectory. The proposed architecture mirrors the BI model with four key layers:

1. Ingest/Route: This layer is responsible for collecting data from various sources and directing it to the appropriate storage. OpenTelemetry is highlighted as a prime example of an open-source solution that can fulfill this role, acting as a standardized way to get data into a system and route it to multiple back-ends.
2. Data Storage: This is where the raw observability data (logs, metrics, traces) resides. The key here is that this storage should be accessible by multiple query engines. Cloud object stores like S3 are often used due to their cost-effectiveness and accessibility.
3. Query/Compute: This layer is where the data is processed and analyzed. Crucially, in a decoupled model, this layer supports multiple query languages (e.g., PromQL for metrics, LogQL for logs, SQL for structured data, and proprietary languages like Splunk's SPL or KQL). This allows different teams to use the tools they are most familiar with.
4. Visualization: This is the user-facing layer, where dashboards and alerts are presented. This can be any tool, from Grafana and Kibana to Tableau or custom applications, all querying the same underlying data.

The critical shift is from vertically integrated stacks to a horizontal, layered approach. This decoupling is primarily enabled by standardizing the interface to the data--the query language--rather than trying to enforce a single, universal schema for the data itself.

"The real standard is not trying to figure out how that data should be shaped, but figuring out how that data should be interfaced with. And that's really the query language. And so to me, that's the actual key to the standardization so that you can decouple is supporting the query languages, because that's the thing."

This approach allows for data portability, enabling organizations to move data between storage solutions or query engines without massive re-engineering. It also fosters a more cost-effective model by allowing for better compression and more efficient utilization of compute resources, especially by enabling ephemeral compute for investigation workflows.

The Trade-offs and Triumphs of Decoupling

While decoupling offers significant advantages, it's not without its challenges. The most immediate concern is potential latency. When storage and compute are separated, there can be a "cold start" problem where data needs to be fetched from storage before it can be queried. However, Tschetter distinguishes between two primary use cases: detections and investigations.

For detections (e.g., real-time alerting), low latency is paramount, and continuous compute might be justified. For investigations, however, where data is queried ephemerally and iteratively, the cold start latency is less critical, especially when balanced against the cost savings of decoupled storage. Caching strategies, often implemented on the ephemeral compute instances themselves, become crucial here to speed up iterative queries.

Another challenge is ensuring data portability and interoperability. While a universal schema is deemed unlikely and impractical, standardization around query languages is the key. Technologies like OpenTelemetry help by providing a common way to collect and route data. Apache Druid, as implemented by Imply with its Lumi product, plays a crucial role by acting as a powerful analytical data store that can ingest data and expose it through multiple query languages and interfaces. This allows diverse tools--from log-specific query languages to SQL-based BI tools--to interact with the same data set.

Governance is also a consideration. With multiple query tools accessing the same data lake, robust access control is essential. Furthermore, separating compute resources for different use cases (e.g., business analytics vs. security investigations) can prevent "noisy neighbor" issues where one team's heavy queries impact another's performance.

Ultimately, the success of decoupling hinges on whether teams can continue using their preferred tools without learning new ones or migrating data. If putting data into a central location allows different teams to access it using their existing workflows, the decoupling has succeeded. It's about enabling access and flexibility, not forcing uniformity.

Key Action Items

• Immediate Actions (0-3 Months):

  • Audit Current Stack: Identify all observability tools in use, their data sources, and their underlying storage. Map out existing vendor lock-ins and data silos.
  • Pilot OpenTelemetry: Begin instrumenting new services or a small subset of existing services with OpenTelemetry to standardize data collection and routing.
  • Explore Cloud Object Storage: Investigate using cloud object storage (S3, GCS, Azure Blob) as a cost-effective, accessible data lake for raw observability data for a pilot project.
  • Evaluate Druid/Imply: Conduct a proof-of-concept with Apache Druid or Imply's Lumi to understand its capabilities in querying diverse data sources and supporting multiple query languages.
  • Define Critical Use Cases: Identify your most critical detection and investigation workflows to inform infrastructure and query language priorities.

• Longer-Term Investments (6-18+ Months):

  • Phased Migration to Decoupled Architecture: Gradually migrate data from monolithic observability solutions to the decoupled data lake, starting with less critical data sets or new initiatives.
  • Standardize Query Interfaces: Focus on supporting a defined set of query languages (e.g., LogQL, PromQL, SQL) on your central data store, enabling teams to use their preferred tools.
  • Implement Robust Access Control: Establish granular access control policies for the data lake, ensuring data security and compliance.
  • Isolate Compute Resources: Plan for and implement separate compute pools for different teams and use cases (e.g., business analytics vs. security) to manage performance and costs.
  • Invest in Compression and Efficient Indexing: Prioritize storage and indexing strategies that maximize data retention and query performance without resorting to excessive sampling.
  • Develop Observability for Observability: Implement monitoring and alerting for your observability pipeline itself to ensure its health and reliability.

• Items Requiring Present Discomfort for Future Advantage:

  • Adopting OpenTelemetry: This requires initial effort in instrumenting services but pays off by providing a unified data collection standard.
  • Migrating Data from Walled Gardens: This is a significant undertaking that can be painful in the short term but unlocks long-term flexibility and cost savings.
  • Learning New Query Patterns (if necessary): While the goal is to avoid this, some teams might need to adapt to new query languages or interfaces to access broader data sets, a discomfort that yields greater data access.
  • Investing in Compression over Sampling: This requires upfront engineering effort to optimize storage but preserves valuable data signals that sampling would discard, preventing future investigative blind spots.