Observability: what it is, pillars and how to implement it

Observability is the ability to infer a system's internal state from the data it emits externally — logs, metrics and traces. In distributed architectures, it lets engineering teams understand not only that something failed, but why it failed, accelerating diagnosis and optimization.

• What is observability and why is it crucial?
• How does observability work in practice?
• What are the three pillars of observability?
• What is the difference between monitoring and observability?
• How to implement observability in your system?

What is observability and why is it crucial?

Observability is the ability to infer a system's internal state from the telemetry it generates externally, and it is crucial because it reveals the root cause of failures in distributed applications that traditional monitoring cannot explain.

In a scenario where microservices, cloud computing and distributed architectures are the norm, just monitoring basic resources is not enough. Observability goes further: instead of only signaling that something is wrong, it lets you understand why the failure happened, what caused it and how the problem propagates through the system. It relies on collecting and correlating telemetry to provide a holistic view of the application's health, reducing MTTR (Mean Time to Resolution), optimizing resources and sustaining innovation.

The numbers explain the urgency of the topic. According to the New Relic State of Observability 2024, 82% of respondents take more than an hour to recover from incidents (up from 74% in 2023) and 62% say high-impact outages cost at least US$1 million per hour of unavailability. The same report shows that organizations with full-stack observability reduced the cost of these failures by 48$2.1 million to US$1.1 million per hour. Without observability, diagnosing problems in modern systems can become nearly impossible.

Practical example: an e-commerce application starts showing slowness at checkout. A traditional tool only points out that the database has high CPU. With observability and distributed traces, the team discovers that the slowness comes from a query executed multiple times by a recommendation microservice, which in turn calls a slow external service — allowing a surgical fix.

How does observability work in practice?

In practice, observability works by instrumenting the code and infrastructure to collect telemetry — logs, metrics and traces — which is then centralized, correlated and visualized on a single platform.

The process starts with instrumentation: the addition of code or configuration so that systems generate the necessary data, via logging libraries, metrics frameworks or tracing SDKs (Software Development Kits). Once generated, this telemetry is sent to a centralized platform responsible for ingesting it, storing it and, crucially, correlating it. Correlation turns raw data into actionable intelligence: it joins logs to a service's metrics and connects the steps of a transaction through a trace. Finally, this data is presented in dashboards and search interfaces that reveal anomalies and behavior patterns. Tools like Prometheus (metrics), Grafana (visualization), Jaeger or Zipkin (tracing) and the ELK Stack — Elasticsearch, Logstash and Kibana — for logs make up a typical observability architecture, often running on Docker containers.

Practical example: when developing an authentication microservice, the team integrates slf4j for logs, Micrometer for custom metrics (failed login attempts, external call latency) and OpenTelemetry for traces. If a user reports a login failure, the team searches for that request's trace, sees the error logs and the latency metrics of each component, identifying whether the failure happened in credential validation, in the database communication or elsewhere.

What are the three pillars of observability?

The three pillars of observability are logs, metrics and traces — the fundamental types of telemetry that, combined, provide a comprehensive view of the system. Metrics detect that there is a problem, logs diagnose what happened and traces show where and how it happened.

What are logs?

Logs are records of discrete events that occur within a system, providing a textual and chronological trail of what happened at each point in time.

Each log line represents an event — the start of a request, an error, a completed transaction or a state change. Logs are invaluable for debugging and auditing because they capture contextual details that are hard to obtain otherwise. To be effective, they should be structured (for example, in JSON — JavaScript Object Notation), contain relevant information (timestamp, severity level, request ID, service name) and be centralized in a tool like Elasticsearch, Splunk or Loki. Volume is a central consideration: too many logs generate cost and noise, while too few leave visibility gaps.

Practical example: in a payments service, a log might record {"timestamp": "2026-03-15T10:30:00Z", "level": "INFO", "service": "payment-gateway", "event": "transaction_started", "transaction_id": "tx12345"} and, seconds later, {"level": "ERROR", "event": "payment_failed", "transaction_id": "tx12345", "reason": "insufficient_funds"}. Centralized, these logs allow searching for all events of tx12345 and reconstructing the exact sequence of the failure.

What are metrics?

Metrics are numeric aggregations of data collected over time, representing a system's performance and state in a quantifiable and lightweight way.

Unlike logs, which are discrete events, metrics are continuous numeric data: counters (total requests), gauges (current CPU or memory usage) or histograms (latency distribution). They are ideal for monitoring trends, identifying anomalies and triggering alerts, as they are cheap to store and query. It is common to combine system metrics (CPU, RAM, network) with application metrics (API latency, error rate, throughput).

Practical example: a web service can expose http_requests_total (counter), http_request_duration_seconds (duration histogram) and cpu_usage_percentage (gauge). In Grafana, the team observes a spike in request duration correlated with an increase in total requests and CPU usage, indicating a bottleneck under higher demand.

What are traces (distributed tracing)?

Traces are representations of the full path of a single request as it propagates across multiple services in a distributed architecture, exposing the latency of each step in the transaction.

In a microservices system, a single user request can pass through dozens of different services. Distributed tracing connects these events into a single timeline, called a trace, composed of spans — each span represents a unit of work (a function call, a database query, an HTTP request) with a start, duration and metadata. A context identifier is propagated between services so that all spans of the same request are stitched into the same trace. This is the pillar that answers the hardest question in distributed systems: in which service, exactly, was the time spent?

Practical example: a purchase request generates a trace with spans for the API gateway, the inventory service, the payment service and the notification service. When visualizing the trace, the team notices that 90% of the total latency is concentrated in the payment span, directing the investigation straight to the responsible service — without trial and error.

What is the difference between monitoring and observability?

Monitoring and observability complement each other, but they are not synonyms: monitoring tracks already-known problems with predefined dashboards, while observability lets you investigate unforeseen problems by exploring high-cardinality telemetry. The table below summarizes the contrasts.

In practice, monitoring is a subset of observability. You monitor the metrics you already know matter (CPU, error rate) and use observability to investigate everything you did not anticipate. In a mature DevOps culture, the two go together.

How to implement observability in your system?

Implementing observability involves instrumenting the application, adopting an open telemetry standard and centralizing the data in a platform that correlates the three pillars. A practical roadmap:

1. Instrument with an open standard. Adopt OpenTelemetry, a framework maintained by the CNCF (Cloud Native Computing Foundation) that unifies logs, metrics and traces. Being a standard, the same instrumentation sends data to any compatible backend, avoiding vendor lock-in.
2. Structure your logs. Standardize output in JSON with consistent fields (timestamp, level, correlation ID) to allow reliable searches and aggregations.
3. Define application metrics. Go beyond CPU and memory: instrument latency, error rate and throughput of critical endpoints, aligned with your SLOs (Service Level Objectives).
4. Propagate the trace context. Ensure the trace identifier travels through every service, especially in cloud environments like those described in the guide on servers and the AWS universe in cloud programming.
5. Centralize and correlate. Send everything to a platform (Grafana, Datadog, Elastic or similar) that unifies the three pillars into a single investigation and triggers actionable alerts.

It is worth noting that the CNCF itself already treats profiling — CPU usage snapshots — as an emerging fourth signal, which broadens the analysis surface beyond the three classic pillars. Here at CodeCrush, we recommend starting simple: instrument the critical flows first and expand coverage as the team's maturity grows.

Conclusion

Observability has stopped being a luxury for large companies and has become a requirement for any serious distributed architecture. The point that often goes unnoticed is that it is not a tool you buy, but a property you design: instrumenting well, structuring logs and propagating trace context are engineering decisions made in the code, not on the dashboard. If your team still spends hours hunting for the root cause of incidents, investing in observability — starting with OpenTelemetry and the most critical flows — pays for itself on the first on-call night it shortens.