Metrics

Metrics are numerical measurements of system behavior sampled over time. Unlike logs (which record individual events) and traces (which track request journeys), metrics give you aggregate, real-time visibility into system health — the foundation of SRE practice at Google, Microsoft, and every high-availability operation.

The Four Golden Signals (Google SRE Book): Latency (how long requests take), Traffic (how many requests per second), Errors (rate of failed requests), and Saturation (how full the system is). These four signals, monitored together, surface virtually every category of production problem. If latency is rising, traffic is stable, errors are low, and CPU saturation is increasing — a CPU-bound performance regression has been deployed. If errors are rising and traffic is flat — a bug was deployed. If saturation is approaching 100% while traffic is rising — you need to scale.

The USE Method (Brendan Gregg) applies to resource-level metrics: Utilization (% of time the resource is busy), Saturation (amount of work the resource can't service, queued), Errors (count of error events). Apply USE to CPU, memory, network, disk I/O, and message queue depth. A DBA notices that database CPU is at 95% utilization (USE: Utilization) and query queue is growing (USE: Saturation) — a missing index is the likely cause.

The RED Method (Tom Wilkie) applies to service-level metrics: Rate (requests per second), Errors (failed request count or percentage), Duration (latency distribution). RED metrics answer "is this service healthy right now?" for every service in your mesh. The combination of USE (for infrastructure health) and RED (for service health) gives a complete operational picture.

Metric Types in Prometheus/OpenTelemetry: Counter (monotonically increasing: total requests served, total bytes written — never decreases, useful for rates), Gauge (current snapshot: memory usage, queue depth, active connections — can go up or down), Histogram (distribution of observations: request latency bucketed into ranges, enabling p50/p95/p99 calculation), Summary (pre-computed quantiles: p50, p95, p99 latency calculated client-side). For SLO-based alerting, histograms are preferred over summaries because they can be aggregated across multiple instances.

Cardinality is the most common cause of metrics platform cost explosions. Cardinality = the number of unique label value combinations. A metric `http_requests_total` with labels `{service, endpoint, method, status}` — if you have 50 services × 100 endpoints × 5 methods × 20 status codes = 500,000 unique time series. Prometheus struggles above 10 million active series. Solutions: limit high-cardinality labels (avoid user_id, request_id as metric labels), use logs for high-cardinality data, use recording rules to pre-aggregate expensive queries.

SLI/SLO Alignment: Metrics must be designed in alignment with your Service Level Indicators. If your SLO is "99.9% of payment API requests complete in under 2 seconds," your metrics must give you the data to calculate: (requests completing under 2s) / (total requests) × 100%. This requires a latency histogram with buckets that include the 2-second boundary. Design metrics for SLOs, not for debugging convenience.

OpenTelemetry is the CNCF standard for metrics (and traces and logs) instrumentation. It provides vendor-neutral SDKs that emit signals to any compatible backend (Prometheus, Datadog, New Relic, Grafana Cloud). Adopting OpenTelemetry from the start eliminates vendor lock-in in observability tooling — you can switch from Datadog to Grafana without re-instrumenting your services.