Remote Partitioning Design — Kafka, AMQP, Failure Modes, Restart
Fast recall
Quick revision
Core answer
StepExecutionRequest messages to a queue (Kafka, RabbitMQ, JMS); worker JVMs subscribe, execute the worker step against their slice's ExecutionContext, and reply with StepExecution status. The master polls or listens for replies and aggregates outcomes. Critical design choices are at-least-once message delivery (workers must be idempotent), per-partition timeout (master must declare a partition failed if no reply arrives), dead-letter handling (poison messages from worker JVM crashes), and JobRepository sharing so the master can read worker step status from the BATCH_* tables.Whether you can architect a distributed batch system and reason about messaging semantics, restart, and failure modes
Assuming Kafka delivery is exactly-once or that workers can be stateful across messages
Discuss JobRepository as the source of truth, worker idempotency requirements, partition timeout handling, and the failover model when worker JVMs crash mid-step
Interview-ready guide
Interview answer
Remote partitioning scales Spring Batch beyond one JVM by treating workers as independent processes coordinated by messaging.
The components:
- Master step runs the
PartitionerandMessageChannelPartitionHandler; publishesStepExecutionRequestper slice to the messaging channel; waits for replies - Worker JVMs subscribe to the request channel; execute the worker step against their slice's
ExecutionContext; reply withStepExecutionstatus - Shared JobRepository (BATCH_* tables) is the source of truth — master falls back to it when reply messages are lost
Critical design choices:
- At-least-once delivery — Kafka, RabbitMQ both deliver at-least-once; workers must be idempotent (UPSERT, idempotency keys, outbox)
- Partition timeout — master must declare a partition failed if no reply arrives; set on p99 duration not p50
- Dead-letter topic for poison partitions; operator replays after fixing
- JobRepository sharing so master can read worker step status from the BATCH_* tables when messaging fails
Failure modes:
- Worker crashes after accepting request — partition timeout fires; marked FAILED; restart re-runs only this partition
- Reply message lost — master falls back to JobRepository; if StepExecution is COMPLETED, partition succeeds
- Duplicate delivery — repeat worker creates new StepExecution; idempotent business logic prevents corruption
- Master JVM crash — separate cleanup job marks abandoned executions FAILED after threshold
Restart semantics:
- Failed
JobExecutionretains all StepExecutions in the repo - Restart launches new
JobExecutionfor sameJobInstance - Only FAILED partition StepExecutions get re-published
- Workers process re-delivered slices from scratch — idempotency mandatory
When to use remote vs local:
- Datasets that fit in one JVM — local partitioning is simpler
- More than 10M rows or heterogeneous worker hosts — remote partitioning earns its complexity
- Teams without Kafka/AMQP expertise — local partitioning first
In an interview I'd emphasize the JobRepository as source of truth and the at-least-once idempotency requirement — those are the two pieces that distinguish a working topology from a broken one.
Remote Partitioning Design — Kafka, AMQP, Failure Modes, Restart
Direct Answer
Remote partitioning uses Spring Batch Integration's MessageChannelPartitionHandler to publish StepExecutionRequest messages to a queue (Kafka, RabbitMQ, JMS); worker JVMs subscribe, execute the worker step against their slice's ExecutionContext, and reply with StepExecution status. The master polls or listens for replies and aggregates outcomes. Critical design choices are at-least-once message delivery (workers must be idempotent), per-partition timeout (master must declare a partition failed if no reply arrives), dead-letter handling (poison messages from worker JVM crashes), and JobRepository sharing so the master can read worker step status from the BATCH_* tables.
Common Mistakes & Pitfalls
Assuming Kafka delivery is exactly-once or that workers can be stateful across messages
Check your understanding
Answer these without looking back at the article.
- How do you handle the case where a worker JVM crashes after writing some chunks but before replying to the master?
- Why is the JobRepository the source of truth even when reply messages succeed?
- How would you implement a cleanup job that marks abandoned StepExecutions as FAILED?
- What are the trade-offs between Kafka, RabbitMQ, and JMS for the partition channel?
Related interview prep categories
1Looking for a structured roadmap?
Every module on this page is also part of the full Java backend interview track.