Four Golden Signals, RED, USE: Monitoring Methodologies

The instinct for new engineers is to collect every metric a tool offers. The result is noisy dashboards and alert fatigue. Three structured methodologies answer 'what should I actually monitor?'

Four Golden Signals — from Google SRE

• Latency: time to serve requests; split successful vs failed
• Traffic: rate of requests (RPS, QPS)
• Errors: rate of failed requests
• Saturation: how full the service is; pending queue, utilization approaching capacity
• Applies to any user-facing service

RED — Tom Wilkie's simplification for request-driven services

• Rate: requests per second
• Errors: rate of failed requests
• Duration: distribution of request latencies (p50, p95, p99)
• Easier to remember, maps 1:1 with Golden Signals except saturation

USE — Brendan Gregg for resources, not services

• Utilization: % time busy (CPU, memory, disk, network)
• Saturation: extra work queued waiting (run queue, swap, I/O queue)
• Errors: system errors (disk errors, packet drops, OOM kills)
• Applies to infrastructure, not application logic

How I use them together

• RED for services: every Spring Boot app gets Rate, Errors, Duration automatically from Micrometer's http_server_requests
• USE for resources: node_exporter gives CPU/memory/disk USE; JVM metrics give JVM USE
• Golden Signals as the unified mental model bridging service-level and resource-level

The utilization vs saturation distinction is critical

• A CPU can be 100% utilized with 0 saturation — one task using all CPU, nothing waiting
• Or 50% utilized with high saturation — tasks alternating, lots of queue waiting
• Monitoring only utilization misses the latency signal that saturation gives
• Both matter; always instrument both

Dashboard layout per service

• Four panels in order: Latency, Traffic, Errors, Saturation
• Same layout for every service; consistency beats creativity
• Every engineer finds the answer fast in any service dashboard

SLO/SLI design flows from RED

• SLO: p99 latency less than 500ms, 99% of the time over 28 days
• SLI: the metric measuring that SLO
• RED metrics (rate, errors, duration) are the natural SLIs
• Define SLOs using RED-style metrics; alert on SLO burn rate, not raw thresholds

Common pitfalls

• Saturation metrics often missing; teams instrument latency and errors but forget thread pool queues, connection pool usage, message queue depth
• Cardinality trap: tagging metrics with per-user or per-request IDs explodes time series
• Histogram buckets not tuned; default Prometheus buckets don't fit most services
• Over-alerting on raw thresholds instead of SLO burn rate

The strategic answer

• Don't collect 'everything'; collect RED for every service, USE for every resource
• Build consistent dashboards; same panel layout per service
• Derive SLOs from RED; alert on SLO burn rate
• Saturation is the metric most teams forget; add it explicitly