Scenario — Restarting a Crashed Batch Job Without Duplicates

Q: Scenario — Restarting a Crashed Batch Job Without Duplicates

First, **stop the bleeding** — confirm no other process is still writing to the target table; mark the abandoned `JobExecution` as `FAILED` so restart can proceed. Then **assess what got committed** — query the BATCH_STEP_EXECUTION row for the failed run and compare `read_count`/`write_count` against the target table to identify which slices completed cleanly. **Restart strategy** depends on idempotency: if writers are **UPSERT-based or use idempotency keys**, a plain restart is safe (committed chunks will be re-read but not duplicated); if writers are **non-idempotent** (`INSERT`-only), you must either delete the partial rows for the failed step or restart with parameters that target only the unprocessed range. Always **document the recovery action** as a post-incident runbook and add idempotency to the writer for the next time.

Intermediate·3 min read·

Goldman SachsJP MorganCapital OneWells FargoStripe

Fast recall

Quick revision

Core answer

First, stop the bleeding — confirm no other process is still writing to the target table; mark the abandoned JobExecution as FAILED so restart can proceed. Then assess what got committed — query the BATCH_STEP_EXECUTION row for the failed run and compare read_count/write_count against the target table to identify which slices completed cleanly. Restart strategy depends on idempotency: if writers are UPSERT-based or use idempotency keys, a plain restart is safe (committed chunks will be re-read but not duplicated); if writers are non-idempotent (INSERT-only), you must either delete the partial rows for the failed step or restart with parameters that target only the unprocessed range. Always document the recovery action as a post-incident runbook and add idempotency to the writer for the next time.

Interview Answer3 min

When a batch job crashes mid-run, restart safety hinges on writer idempotency — not on the framework alone.

Phase 1 — stop the bleeding and assess:

Stop the scheduler so nothing else triggers the job
Confirm no concurrent writers on the target table
Mark the abandoned JobExecution as FAILED with JobOperator.abandon(id) so restart can proceed
Inspect BATCH_STEP_EXECUTION to see what got committed — write_count, commit_count, exception text
Cross-reference with the target table — query rows by batch_run_id if you tagged them

Phase 2 — pick a restart strategy by writer idempotency:

Idempotent writer (UPSERT or idempotency keys) — plain restart with same parameters; framework resumes from checkpoint; no duplicates
Non-idempotent writer with clean partition boundary — delete partial rows for failed partition; restart
Non-idempotent writer without boundary — delete all run rows and relaunch with new parameters, OR run a manual reconciliation step
Side effects already fired (emails, API calls, Kafka) — these are NOT rolled back; assess business impact and issue compensating actions

Triggering the restart:

JobOperator.restart(failedJobExecutionId) — canonical API; creates a new JobExecution for the same JobInstance
Resumes from the failed step using ExecutionContext checkpoint

Phase 3 — forward-looking fixes:

Retrofit idempotency to the writer — UPSERT semantics is the most direct fix
Add a cleanup tasklet that abandons stuck executions automatically
Move side effects out of the chunk transaction — outbox pattern or final tasklet
Add observability — Micrometer counter for partial failures, dashboard for STARTED-status executions
Document the recovery runbook with exact SQL, restart commands, and compensating actions
Test restart in CI by injecting deliberate mid-job failure

Common pitfalls:

Assuming Spring Batch restart magically prevents duplicates — only true with idempotent writers
Using RunIdIncrementer thinking it restarts — it actually creates a new JobInstance
Multi-threaded step restart — ExecutionContext may be imprecise
Side effects fired before crash — emails, API calls, Kafka events not rolled back

In an interview I'd emphasize that the operational answer to 'how do I restart?' is always 'how idempotent is the writer?' — that determines whether the framework can do the work or whether you need manual cleanup.

Interview-ready guide

Interview answer