Deployment Strategies: Rolling, Blue/Green, Canary, Shadow, Feature Flags

Q: Deployment Strategies: Rolling, Blue/Green, Canary, Shadow, Feature Flags

**Rolling** is Kubernetes' default: replace pods N-at-a-time. Zero downtime if app handles graceful shutdown; risk evenly spread over the rollout window. **Blue/green** keeps two full environments; switch the router to 'green' for instant promote and rollback. Great for high-stakes cutovers and heavy state migrations but doubles infrastructure cost. **Canary** routes a small % of traffic (1–5%) to the new version, watches metrics, then ramps up. Best for detecting regressions cheaply. **Shadow** mirrors production traffic to the new version without serving its responses — great for testing non-idempotent changes (ML inference, new DB queries) without user impact. **Feature flags** decouple deploy from release: code ships dark, then is toggled on for cohorts. Production-grade systems **combine strategies**: canary for the deploy + feature flags for behavior + rolling as the underlying mechanic. Tooling: Argo Rollouts, Flagger, LaunchDarkly, Istio/Linkerd, AWS CodeDeploy.

Intermediate·7 min read·

KubernetesArgoFlagger

Fast recall

Quick revision

Core answer

Rolling is Kubernetes' default: replace pods N-at-a-time. Zero downtime if app handles graceful shutdown; risk evenly spread over the rollout window. Blue/green keeps two full environments; switch the router to 'green' for instant promote and rollback. Great for high-stakes cutovers and heavy state migrations but doubles infrastructure cost. Canary routes a small % of traffic (1–5%) to the new version, watches metrics, then ramps up. Best for detecting regressions cheaply. Shadow mirrors production traffic to the new version without serving its responses — great for testing non-idempotent changes (ML inference, new DB queries) without user impact. Feature flags decouple deploy from release: code ships dark, then is toggled on for cohorts. Production-grade systems combine strategies: canary for the deploy + feature flags for behavior + rolling as the underlying mechanic. Tooling: Argo Rollouts, Flagger, LaunchDarkly, Istio/Linkerd, AWS CodeDeploy.

Interview Answer7 min

Rolling is Kubernetes' default: replace pods N-at-a-time. Zero downtime if app handles graceful shutdown; risk evenly spread. Blue/green keeps two full environments; switch router to green for instant promote and rollback. Canary routes small % of traffic (1-5%) to new version, watches metrics, ramps up. Shadow mirrors production traffic to new version without serving its responses. Feature flags decouple deploy from release.

Strategy matters because deployment is where code meets users

Bad deploy = availability loss, data corruption, trust erosion
Differ in: blast radius, rollback speed, cost, complexity, traffic handling
Risk/cost trade-off matched to what you're deploying

Six common strategies

1
Recreate: stop all, start new; downtime; dev only
2
Rolling update: replace pods gradually
3
Blue/Green: parallel env, switch router
4
Canary: small % traffic to new version
5
Shadow (traffic mirroring): copy traffic, don't serve responses
6
Feature flags: code ships disabled, toggled per-cohort

Progressive delivery = canary + feature flags + auto-rollback on SLO breach

Rolling update

Deployment strategy RollingUpdate: maxSurge 25%, maxUnavailable 0
Spawns new pods before killing old, waits for readiness, terminates old
Requires good readiness probe (DB, cache warmed)
Gotcha: during rollout traffic split; API must be backward-compatible; no breaking schema

Blue/Green

Two deployments with color labels; service selector swap
Argo Rollouts: blueGreen with activeService, previewService, autoPromotionEnabled, prePromotionAnalysis
Test green via preview; promote = service switch; rollback = flip back
Great for large schema migrations (but not when versions must coexist)

Canary

Argo Rollouts canary with steps: setWeight 5 → pause 5m → analysis → setWeight 25 → ... → 100
Analysis templates query Prometheus/Datadog for SLOs (error rate, p99)
Breach = auto-rollback
Routing via service mesh (Istio, Linkerd) or ingress
Best ROI for most teams — catches 90% of regressions with 5% risk

Shadow (mirroring)

Istio VirtualService: route to v1 100%, mirror v2 at mirrorPercentage 10%
10% of requests copied to v2; responses discarded
User sees only v1's response
Perfect for: new DB queries (N+1?), new ML model (latency), refactored serialization
Gotcha: non-idempotent side effects (double-writes); point v2 at read replica or sandbox DB

Feature flags

if (featureFlags.isEnabled("new-checkout-flow", userId)) ...
Tools: LaunchDarkly, Unleash, Flagsmith, OpenFeature (standard)
Deploy disabled; enable for internal testers, then %, then full
Combines with canary: canary = runtime path; FF = behavior
Decouples deploy from release; marketing launches without code change

Schema migrations - hidden hard part

Both v1 and v2 run against same DB during rolling; breaking changes break one
Expand-contract:
1. 1
  Expand: add nullable column; code writes to both old + new
2. 2
  Backfill: fill new from old
3. 3
  Deploy v2: read new, still write both
4. 4
  Contract: drop old after v2 fully rolled out
Tools: Flyway, Liquibase, gh-ost, pg_squeeze
Never break schema during single deploy

Graceful shutdown

Spring: server.shutdown=graceful + timeout-per-shutdown-phase=30s
K8s: terminationGracePeriodSeconds + lifecycle.preStop (sleep 10 for LB drain)
Sequence: SIGTERM → preStop sleep → Spring stops new requests, completes in-flight → connections drain → JVM exits
Without: old pod terminates mid-request → errors

Readiness probe

/actuator/health/readiness + /actuator/health/liveness separate (Spring Boot 2.3+)
Readiness checks app + deps; liveness just 'JVM alive'
Don't confuse: dependency checks in liveness → DB hiccup kills pods → cascade
startupProbe for slow apps (>30s) prevents liveness from killing

Progressive delivery automation (Flagger)

Canary: interval 1m, threshold 10, maxWeight 50, stepWeight 5
Metrics: success-rate >= 99%, duration <= 500ms
Incrementally shifts 5%/min; breach = auto-rollback
Modern progressive delivery: humans set guardrails, automation drives rollout

Cost vs safety

Rolling: cheapest, most common, good enough
Blue/Green: doubles infra during deploy; critical systems
Canary: small extra replica cost; highest confidence with automation
Shadow: doubles load for mirrored; specific test scenarios
FF: minimal infra; cognitive/process overhead

Data and state issues

Stateful (DBs, brokers): rolling with StatefulSets; blue/green hard due to migration
Cache invalidation: new version's keys may differ; plan warming
In-flight messages: Kafka offsets, queue acks; versions handle both formats
Long requests: blue/green 'instant' not instant if 30s uploads; drain time

Anti-patterns

1
Rolling with breaking schema
2
Canary without metrics-based promotion
3
Blue/green without prepared 'green'
4
Feature flag sprawl (500 flags, nobody knows what's safe to remove)
5
No rollback plan
6
Graceful shutdown neglected
7
Liveness probe hits DB — DB hiccup kills all pods
8
Shadow against prod DB — duplicate writes

Progressive delivery - 2020s default

1
Feature flags for behavioral toggles (instant on/off)
2
Canary for new code paths (risk-limited exposure)
3
Rolling as underlying mechanism
4
Automated rollback on SLO violations
5
Chaos engineering for resilience

Stack: Argo Rollouts + Flagger + LaunchDarkly + Istio/Linkerd Simpler: K8s rolling + feature flags

Observability during deploys

Dashboard: error rate, p99, RPS by version (v1 vs v2)
Alert on SLO breach during rollout
Annotate metrics: deploy markers in Grafana
Traces with version tag; compare latencies

Mature deploy flow

1
PR merged → CI builds image, pushes registry
2
CD (Argo CD) detects manifest; starts rollout
3
Flagger/Argo shifts 5% traffic
4
Analysis queries Prometheus for SLOs
5
Every 1 min, step +5% if healthy
6
50% or 100% → rollout complete
7
Any step: auto-rollback on SLO breach
8
Human pause/resume; automation is default

Strategic take

Deployment strategy IS risk management
Depends on: blast radius, rollback speed, safety spend, state involvement
Don't use blue/green for everything (expensive)
Don't use rolling for everything (risky for high-stakes)
Combining strategies = modern default
Automation with auto-rollback on SLO pays off when catching 3 AM regressions

Bottom line

Rolling = default; canary = safer; blue/green = instant rollback; shadow = validate; FF = decouple deploy/release
Schema migrations expand-contract; graceful shutdown mandatory
Progressive delivery (Flagger/Argo) is the 2020s standard

Interview-ready guide

Interview answer