Designing an Idempotent Payment Microservice — Idempotency Key + Atomic Insert + Outbox

Why this matters for payments specifically

• Double charging is a trust event worse than failure — customer calls support while their balance is double-debited
• Stripe, Adyen, Square all make idempotency a required header, not optional
• The mechanism: client generates a UUID per logical charge attempt, reuses on retry
• Server deduplicates by storing key mapped to response

The race-safe primitive

• Not 'check if key exists, then insert if not' — classic TOCTOU race
• Atomic INSERT with UNIQUE constraint on the key
• DB guarantees exactly one succeeds; the loser catches DataIntegrityViolation and re-reads
• Without this, two simultaneous retries both pass the check and both process

The DB schema

• idempotency_records: key UUID primary key, request_hash SHA-256, status IN_PROGRESS / COMPLETED / FAILED, response_status SMALLINT, response_body JSONB, created_at, completed_at
• charges table: own table; foreign key to idempotency_records
• outbox_events: id, aggregate, event_type, payload, published, created_at

The endpoint flow

• Read Idempotency-Key header; hash the canonical request body
• Look up key; four cases:

  • Found COMPLETED with matching hash: return cached response
  • Found any status with mismatched hash: return 422
  • Found IN_PROGRESS: return 409
  • Not found: INSERT IN_PROGRESS; on unique-violation re-enter lookup
• Call payment gateway with our key (defense in depth)
• Single DB transaction: insert charges row, update idempotency record to COMPLETED, write outbox event
• Return 200 with response body

Outbox pattern for downstream events

• Fraud, accounting, notifications subscribe to ChargeCompleted
• Outbox ensures charge commit and event publish are atomic
• Separate publisher drains outbox, marks published, sends to Kafka
• Consumers dedupe on their own using the charge ID

Key race conditions to walk through

Two simultaneous retries with same key

• First INSERT wins; second gets unique-violation and retries lookup
• Finds IN_PROGRESS; returns 409
• First completes; third retry from client sees COMPLETED; returns cached response
• No double charge

Server crash after gateway call, before DB commit

• DB has IN_PROGRESS but no charges row
• Customer charged at gateway; our records don't reflect it
• Retry with same key sees IN_PROGRESS and returns 409
• Solution: pass our key to gateway; next retry's gateway call with same key returns original response

Payload mismatch on key reuse

• First request K1 with amount 1000
• Second request K1 with amount 10000
• Payload hashes don't match; return 422
• Client learns their key logic is broken

TTL expires during long retry window

• TTL 24 hours; client retries at T plus 25 hours
• Key gone; server processes again; double charge
• TTL must exceed maximum client retry window

Gateway-side idempotency

• Upstream gateways accept an idempotency key too
• Pass your key or a derived one to them
• Defense in depth: if our service crashes mid-flow, gateway's idempotency prevents double charge at their layer

FAILED status policy

• Stripe default: return cached failure; client decides next step
• Alternative: retry FAILED; risk of masking real problems
• Policy-driven is best: retry on gateway timeouts, return cached on validation errors

Metrics to expose

• idempotency_hits — seen-before count; spike signals retry storm or clock sync issue
• idempotency_payload_mismatch — client-bug signal
• idempotency_in_progress_returns — concurrent retries; possibly misbehaving client

Testing discipline

• Testcontainers with real Postgres — H2 hides unique-constraint concurrency
• Test concurrent retries converge to exactly one charge
• Test payload mismatch returns 422
• Test gateway failure doesn't corrupt state

Client SDK pattern

• Ship an SDK that generates one key per logical operation and reuses on retry
• Reduces client bugs 10x vs documenting a protocol

Common pitfalls to name

• Check-then-insert race (not atomic)
• No payload hash (different request same key succeeds silently)
• Gateway call outside transaction with no gateway-side key (can double charge)
• TTL shorter than retry window
• Client regenerates UUID per HTTP retry (not logical operation)
• In-memory test DB that hides the race

The strategic answer

• Idempotency key required; atomic INSERT with UNIQUE constraint; payload hash for mismatch detection
• Single transaction for charge, idempotency, outbox
• Gateway-side idempotency for defense in depth
• 24-hour TTL; metrics on hits and mismatches; SDK for clients
• Testcontainers Postgres tests for race safety — H2 lies about concurrency