Metrics Backends: Prometheus vs CloudWatch vs Datadog
Fast recall
Quick revision
Core answer
Understanding of metrics-collection trade-offs, pull vs push, and cost/cardinality concerns
Treating them as interchangeable; not understanding pull vs push model implications; underestimating custom-metric costs on CloudWatch and Datadog
Discuss pull vs push philosophy, scrape intervals and resolution, cardinality limits per tool, long-term storage (Thanos/Mimir), and the RED/USE methodologies
Interview-ready guide
Interview answer
Metrics are the cheapest telemetry to collect and the most valuable for real-time alerting, but picking where to store them affects cost, operability, and lock-in. Three fundamentally different systems:
Prometheus — the OSS standard
- Pull-based: services expose /metrics endpoints; Prometheus scrapes at 15-second intervals
- PromQL is powerful and designed for time-series math; industry-standard skill
- Free software; you operate HA, long-term storage, scaling yourself
- Single node by default; HA via federation or Thanos/Mimir
- Cardinality limits around 1M active series per node; design label schemas carefully
- Best for Kubernetes, microservices, OSS-first orgs
CloudWatch — AWS-native
- Push-based: services call PutMetricData; AWS services publish automatically
- Every AWS service emits hundreds of metrics for free; best infra coverage on AWS
- Metric Math is weaker than PromQL; 1-minute resolution default
- $0.30 per custom metric per month; gets expensive past ~10K custom metrics
- Use embedded metric format to batch metrics into log lines; cheaper at scale
- Best for AWS-native workloads, small to medium teams, simpler ops
Datadog — premium and unified
- Push via Agent (DogStatsD protocol); integrates with everything
- Best-in-class dashboards; smart alerting; unified with APM, logs, RUM
- Expensive at scale; custom metrics charged per unique tag combination
- High cardinality (user IDs, session IDs in tags) explodes cost fast
- Best when time-to-value > cost, premium UX matters
Pull vs push has real implications
- Pull (Prometheus): service down = target down = explicit alert signal
- Push (CloudWatch, Datadog): service down = metrics stop = ambiguous (network? service?)
- Push systems need explicit heartbeat metrics to detect 'quiet death'
The cardinality trap applies to all three
- Each unique label/tag combination creates a new time series
- Metric tagged with user_id = millions of time series = Prometheus OOM or huge Datadog bill
- Strip high-cardinality labels or use columnar stores like Honeycomb for those cases
The real-world pattern for mature orgs
- Prometheus for Kubernetes and microservice app metrics — free, powerful, OSS
- CloudWatch for AWS-native infrastructure — automatic, zero-config
- Datadog or Grafana as unified visualization layer pulling from both
- Alerting split: Alertmanager for Prometheus, CloudWatch Alarms for AWS, Datadog Monitors for app-level alerts
Long-term storage solutions
- Prometheus local TSDB: 15 days typical
- For months/years: Thanos or Grafana Mimir; S3-backed; adds operational complexity
- CloudWatch: up to 15 months retention built in
- Datadog: up to 15 months; higher tiers get rollups for years
OpenMetrics and OpenTelemetry are the portability bets
- Instrument once with OpenMetrics (Prometheus format) or OpenTelemetry
- Swap backends without re-instrumenting
- Datadog, New Relic, Grafana, CloudWatch all consume these standards
The decision rules
- Kubernetes + microservices + cost-conscious: Prometheus + Grafana
- AWS-first + small team: CloudWatch, add Prometheus only where needed
- Unified UX + willing to pay: Datadog
- Most mature orgs: hybrid of all three with different primary use cases
Metrics Backends: Prometheus vs CloudWatch vs Datadog
Direct Answer
Prometheus is the OSS default for Kubernetes and microservices — pull-based, powerful PromQL, free; but you own scaling, long-term storage (Thanos/Mimir), and HA. CloudWatch is AWS-native, push-based, integrated with every AWS service — great for AWS infra metrics; weaker for app metrics and expensive for custom metrics at scale. Datadog is push-based, hosted, integrates with everything, amazing dashboards — premium pricing especially for custom metrics and high cardinality. Most mature orgs use Prometheus for app metrics + CloudWatch for AWS infra + Datadog (or Grafana) as the visualization layer or pick one of the three as primary.
Common Mistakes & Pitfalls
Treating them as interchangeable; not understanding pull vs push model implications; underestimating custom-metric costs on CloudWatch and Datadog
Check your understanding
Answer these without looking back at the article.
- What is Thanos or Grafana Mimir and how do they solve Prometheus long-term storage?
- What is CloudWatch Embedded Metric Format (EMF) and why does it reduce cost?
- How does Prometheus handle high cardinality and what are the symptoms of cardinality overload?
- What are the RED and USE methodologies for metrics?
Related interview prep categories
1Looking for a structured roadmap?
Every module on this page is also part of the full Java backend interview track.