Horizontal vs Vertical Scaling: When to Scale Out vs Scale Up

Two mental models

Vertical: bigger machine. Same JAR, more CPU/RAM. System is still 'one machine' from the app's perspective. Horizontal: more machines behind a load balancer. System is 'many machines coordinating.'

It's not either-or; modern systems do both — vertical to a comfortable size, then horizontal for growth and resilience.

Vertical strengths

• No code changes. Same JAR, bigger box. Critical for legacy and stateful workloads where 'just shard it' isn't an option
• Lower latency for in-process workloads — bigger CPU caches, fewer hops
• Simpler operations — one instance, one set of logs
• Default for most relational databases

Vertical limits

• Hardware ceiling — largest cloud instance is ~32TB RAM, 224 vCPU; specialty hardware higher but extreme cost
• Single point of failure — one disk dies, you're down
• Superlinear cost — doubling vCPU often more than doubles price; 16xlarge is ~3-4x a 4xlarge, not 4x
• Resize requires downtime — stop-modify-start, 3-15 minutes
• No elasticity — can't scale for a 2h spike

Horizontal strengths

• Effectively unbounded — webscale companies run 100K+ instances of the same service
• Built-in fault tolerance — one dies, LB routes around
• Elastic — autoscale up for spikes, down for quiet; cloud cost optimization
• Roughly linear cost — no big-box premium
• Independent deploys and isolation

Horizontal challenges

• Requires stateless or shardable design
• Distributed system problems appear: CAP trade-offs, partitions, leader election, clock skew
• Operational complexity — service discovery, LB, certificates, logs, distributed tracing
• Cold start and reactive autoscaling — sudden spikes outpace it
• Coordination overhead — distributed locks, consensus, config sync
• Some workloads (heavy joins, in-memory analytics) are hostile to horizontal split

The pragmatic order — diagonal scaling

1. 1
   Make it work on one box — ship monolith, profile bottlenecks
2. 2
   Scale up first — cheap, zero-effort, gets you 10-100x
3. 3
   Make stateless tiers horizontal — push session/state to Redis or DB; LB + autoscale
4. 4
   Keep stateful tiers vertical as long as possible — beefy Postgres + standby + read replicas handle a lot
5. 5
   Shard stateful tier only when you must — hot-shard problems, rebalancing, cross-shard queries are real costs; pick shard key carefully (user_id, tenant_id)

Diagonal scaling: combine vertical (bigger nodes) with horizontal (more nodes). 10-node Cassandra of 16xlarges, not 100 smalls.

Statelessness — the key enabler

Stateless service: any instance can serve any request.

• Store sessions in Redis/DynamoDB, not JVM memory
• Files in S3, not local disk
• In-memory caches are best-effort
• WebSocket needs sticky sessions or sharded handling

LB can use round-robin or P2C. Autoscale freely. Rolling deploys, blue-green, canary all trivial.

Deeper truth: 'horizontal scaling' is mostly 'pushing state down to the next tier.' Eventually you hit a tier (database) that's hard to horizontally scale — sharding becomes the question.

Database scaling patterns

• Vertical first: Postgres/MySQL beefy instance handles tens of thousands of QPS
• Read replicas for read-heavy: leader vertical, N replicas
• Sharding for write throughput: Vitess (MySQL), Citus (Postgres); cross-shard queries become app-level fan-out
• Distributed-native: Cassandra, DynamoDB, Spanner, CockroachDB — horizontal out of the box
• NewSQL: Spanner, CockroachDB, TiDB — SQL semantics with horizontal scaling

Recommendation

Diagonal scaling is the real-world default. Start vertical because it's cheap and zero-effort; push to horizontal once you hit hardware ceilings, single-failure cost is unacceptable, or per-instance cost climbs superlinearly. Stateless services scale out trivially via load balancer + autoscaling — push state down to Redis or the database. Keep relational databases vertical with read replicas for as long as possible; shard only when write throughput truly demands it. For new systems with global scale needs, consider distributed-native (Cassandra) or NewSQL (Spanner/CockroachDB) from day one to avoid a painful sharding migration later.