Scenario — WebFlux Event Loop Stalling Under Load

Q: Scenario — WebFlux Event Loop Stalling Under Load

Event-loop stalls in WebFlux mean **blocking work on the wrong scheduler** — event-loop threads are sacred and must return immediately. The investigation: capture a **thread dump** during the spike to see what `reactor-http-epoll-N` threads are doing; enable **BlockHound** in non-prod to catch the culprit with a stack trace; check `reactor.netty.eventloop.pending.tasks` plus p99 latency; and audit code paths for blocking JDBC, synchronous SDKs, `Thread.sleep`, `.block()`, or CPU-heavy work on the event loop. Mitigation: wrap blocking calls with `Schedulers.boundedElastic()`, replace blocking clients with `WebClient` or R2DBC, and add circuit breakers. Long-term: consider migrating to virtual threads if the service is mostly blocking anyway.

Intermediate·3 min read·

SpringNetflixVMwarePivotalStripe

Fast recall

Quick revision

Core answer

Event-loop stalls in WebFlux mean blocking work on the wrong scheduler — event-loop threads are sacred and must return immediately. The investigation: capture a thread dump during the spike to see what reactor-http-epoll-N threads are doing; enable BlockHound in non-prod to catch the culprit with a stack trace; check reactor.netty.eventloop.pending.tasks plus p99 latency; and audit code paths for blocking JDBC, synchronous SDKs, Thread.sleep, .block(), or CPU-heavy work on the event loop. Mitigation: wrap blocking calls with Schedulers.boundedElastic(), replace blocking clients with WebClient or R2DBC, and add circuit breakers. Long-term: consider migrating to virtual threads if the service is mostly blocking anyway.

Interview Answer3 min

Event-loop stalls in WebFlux mean blocking work on the wrong scheduler — event-loop threads must return immediately.

Symptom pattern

p99 spikes to seconds; p50 maybe OK
Random timeouts; CPU modest but requests queue
Thread dump shows reactor-http-epoll-N threads RUNNABLE in user code, not epollWait
reactor.netty.eventloop.pending.tasks high

Step 1 — Diagnose

Three thread dumps 10s apart
Identify event-loop threads (reactor-http-epoll-N)
Read the frames — JDBC / SocketInputStream.read = blocking I/O; JSON / crypto = CPU; LockSupport.park = contention
Capture metrics — pending tasks, p99, pool saturation, GC pauses

Step 2 — Confirm with BlockHound in non-prod

JVM agent throws on blocking calls on non-blocking schedulers
Install only on dev / staging profile
Reproduce under load with k6 / Gatling; correlate spikes with BlockHound errors

Step 3 — Fix categories

Blocking JDBC — Mono.fromCallable(...).subscribeOn(boundedElastic()) or migrate to R2DBC
Synchronous HTTP — replace RestTemplate with WebClient
Synchronous SDKs — AWS v2 async + Mono.fromFuture
CPU-heavy — publishOn(Schedulers.parallel())
.block() in chain — replace with flatMap
synchronized with I/O — fetch outside lock; use ReentrantLock or putIfAbsent
Bulk work on one thread — flatMap(fn, 32) or parallel().runOn(parallel())

Step 4 — Prevent and monitor

Metrics — reactor.netty.eventloop.pending.tasks, p99 per endpoint
Alerts — pending greater than 100; p99 2x baseline; BlockHound errors
BlockHound mandatory in staging CI
ArchUnit lint banning .block() in reactive packages
Degrade gracefully — rate limiter, bulkhead, circuit breaker, timeouts, fallbacks

Hard-earned lessons

Reactive is not self-protecting; tooling and discipline are required
Mixing blocking and reactive is fragile
More threads is not the fix
boundedElastic has limits — don't push everything blocking there
Timeouts on every outbound call
Consider MVC + virtual threads if you keep wrapping blocking calls

The one-line summary

Event-loop stalls are blocking code on the wrong scheduler
Find with BlockHound and thread dumps
Move to boundedElastic or replace with reactive alternatives
Prevent with tooling
Consider virtual threads if reactive isn't paying off

Interview-ready guide

Interview answer