Pipeline Suddenly Slow: Profile, Cache, Parallelize

Pipeline went from 8 minutes to 35 minutes. The temptation is to throw bigger runners at it, but the right first step is always profiling — every CI tool surfaces per-stage timing, and the answer is usually one specific stage that grew, not the whole pipeline degrading uniformly.

Step 1 — profile

• Open the run summary; compare per-stage timing today vs a recent fast run
• Identify which stage grew — dependency install, test stage, Docker build, or queue wait
• This narrows the search from "the whole pipeline is slow" to one specific bottleneck

The four most common culprits, in order of frequency

Cache miss on dependencies

• Symptom: dependency install jumped from 30 seconds to 5 minutes
• Diagnose: cache step logs say "cache not found"; the cache key probably hashes a lockfile that changed
• Fix: add restore-keys for fallback (partial hit beats nothing); cache the right artifacts — npm cache directory, Maven .m2, Gradle caches, not the build output

Test bloat or flaky retries

• Symptom: tests went from 4 minutes to 20 minutes
• Diagnose: surface slowest tests via JUnit reports; check retry counts — a flaky test retried 3 times costs 3x
• Fix: quarantine flakes immediately into a non-blocking job; parallelize with 2–4 workers; split slow integration tests into a separate stage that runs only on main or labeled PRs; the 80/20 rule applies — 5% of tests usually take 50% of the time

Docker layer cache invalidated

• Symptom: Docker build went from 2 to 10 minutes
• Diagnose: build log shows the first "not cached" layer; everything after rebuilds; most common cause is copying source before installing dependencies
• Fix: reorder the Dockerfile so dependency files are copied and installed before source; use multi-stage builds; use BuildKit cache mounts for Maven or npm caches; use registry cache exporter for cross-pipeline cache

Runner queue time

• Symptom: jobs wait 5+ minutes before starting
• Diagnose: gap between queued and started timestamps
• Fix: add concurrent runner capacity; switch to self-hosted runners on autoscaling Kubernetes or EC2 spot; add path filters so unrelated changes (docs) don't trigger full builds

The traps to avoid

• Bigger runners aren't always faster — many workloads are single-threaded; doubling CPU does nothing
• Test parallelism has overhead — spinning up 8 workers each with database fixtures costs 30 seconds per worker; sweet spot is usually 2–4
• Cache eviction — GitHub Actions evicts at 10GB per repo; if you cache everything, frequently-used caches get evicted
• Flaky tests destroy velocity — a 1% flaky rate across 100 tests means 63% chance of failure per run; quarantine and fix or delete within a week

The bigger picture

• The target is PR feedback under 10 minutes and full deployment under 30; longer kills the developer build-test-fix tight loop
• Anything over an hour signals a CI architecture problem — monolithic test suite, missing test impact analysis, no caching strategy — not just a tuning problem