Understanding observability metrics: Types, golden signals, and best practices

The foundation of observability is often described in terms of three pillars: metrics, logs, and traces. Together, they provide essential visibility into system performance and behavior. As technology continues to advance and observability needs increase, a fourth pillar is emerging: profiles.

Metrics are raw numerical data points collected from hardware, software, and websites. In measuring known knowns, metrics are used for monitoring resource usage, performance, and user behavior. In other words, metrics tell monitoring and observability teams what is happening in their systems.

Core types of observability metrics
Observability is a practice that gives organizations a 360-degree view of their environments and operations. To do so, observability relies on these core types of metrics:

• Application metrics: Application metrics are the telemetry data generated by and related to applications within a technology stack. Some examples of commonly used metrics include response times, throughput, request rates, and error counts. These metrics allow engineers to monitor application performance and availability. Application metrics are also used in application performance monitoring (APM). 

• System metrics: System metrics, also referred to as infrastructure metrics, reflect the health of hardware and operating systems including key components like Kubernetes. Examples include CPU utilization, disk I/O, network throughput, memory usage, instance uptime, container resource utilization, and service availability. These metrics provide insights into the performance of cloud resources, virtual machines, containers, and other underlying components. 

• Business metrics: Business metrics tie technical and operational performance to business outcomes. For example, metrics like conversion rates, average transaction value, and user retention help correlate system performance with organizational objectives.

An effective observability solution ensures reliability, effective resource allocation, compliance, and security. It also helps plan capacity, optimize performance, improve user experiences, and control costs. Core metrics enable effective observability, and ultimately, data-driven decision-making that translates to better business outcomes. These metrics are typically aggregated and visualized in dashboards for real-time performance monitoring. 

Logs are timestamped entries of specific events generated by systems, applications, networks, and infrastructure. They provide event details and context, allowing engineers to understand why issues occur. 

Network devices, applications, operating systems, applications, IoT devices, and third-party applications emit different types of logs, including (but not limited to): 

• System logs: These include events like connection attempts, errors, and configuration changes.

• Application logs: They record software changes, CRUD operations, application authentication, and other events to help diagnose issues. 

• Network logs: Network logs record data from events that take place on a network or device, including network traffic, security events, and user activity.

Logs are recorded in structured and unstructured formats, which represents a storage challenge. They can also be hard to categorize since log data is often siloed in a variety of systems and not automatically correlated.

Traces are telemetry signals that let engineers see applications and services from a user-session perspective. Distributed tracing collects traces of requests that make their way through a distributed architecture. 

Traces allow engineers to monitor and debug applications, discovering bottlenecks. In other words, traces tell DevOps teams where issues are occurring in their environments. They’re the foundation of proactive monitoring. By analyzing traces, engineers can discover which metrics or logs are related to a particular issue, mitigating future issues.  

For example, traces that help identify slow processes include API queries, front-end API traffic, server-to-server workloads, and internal API calls. 

While metrics, logs, and traces offer users valuable application and system performance data, these signals don’t always provide the details required for troubleshooting code and performance tuning. This is where profiles come in.

Profiling is the gathering and analysis of profiles — stack traces that help identify issues related to data structures, code visibility, and memory allocation at the kernel and user levels. 

Profiling helps uncover bottlenecks across your system at the code level, another key benefit of modern observability. OpenTelemetry is also adopting profiling as a signal. As a result, profiling is emerging as the fourth and newest pillar of observability.

While every organization’s monitoring needs are unique, certain observability metrics are universally important. These metrics are sometimes referred to as the four golden signals within the site reliability engineering (SRE) community.

Latency measures the time it takes for data to travel from one point to another. Latency will signal underlying performance issues. High latency can degrade user experiences by increasing load times, causing application errors, and challenging user expectations.

Traffic metrics track the volume of requests or transactions an application processes. They help teams understand user behavior and anticipate scaling needs.

Error metrics provide visibility into failed requests or operations. Monitoring error rates and identifying patterns can help address recurring issues.

Saturation metrics indicate how close a system is to its capacity limits. Monitoring resource utilization ensures that engineers can proactively address bottlenecks before they impact performance.

These four golden signals are key to effective observability practices as they provide insights into the health and performance of IT systems. When monitored, correlated, and analyzed, these metrics help IT teams get actionable insights that enable them to take on a more proactive stance around site reliability and performance monitoring.

Elastic Observability provides a unified solution for collecting, monitoring, and analyzing observability metrics across your technology stack. With Elastic Observability, you can collect, store, and visualize observability metrics from any source and speed up problem resolution with our Search AI Platform. 

Elastic Observability prevents outages and accelerates problem resolution with search-based relevance, no-compromise data retention, improved operational efficiency and cost, and a future-proofed investment. Get fast, contextual, and unified insights across the broadest data sources with an open, OTel-first solution that seamlessly integrates with your evolving technology ecosystem.

Learn more about observability with Elastic.