Idempotent Payment API Design

The problem

A user clicks Pay, the request times out, the client retries. If the server processes it again, the customer is charged twice. Idempotency means making the same request multiple times has the same effect as making it once. For payments it's non-negotiable.

Clarifying questions I'd ask first

• Who generates the idempotency key — client only or server fallback?
• Idempotency key namespace — global, per merchant, per API key?
• Behavior on payload mismatch — 409 (Stripe-style) or silent replay?
• TTL on stored responses — 24 hours typical?
• Long-running flows like 3D Secure — in scope?

Assumptions for this design

Client generates a UUID v4 per request attempt, scoped to (api_key, key) so merchants can't collide. 24-hour TTL on stored responses. Payload mismatch returns 409. 5,000 QPS at peak. Sub-100ms p99 server-side. Single primary region with cross-region async replication.

Quick capacity check

• 5K QPS, sub-100ms latency budget
• Idempotency table grows ~430M rows over 24-hour window at peak — needs daily cleanup
• Redis caches last 24 hours of responses; ~50 GB at average response size

Storage — two layers, both essential

• Redis for the fast path: cached response by idem:{api_key_hash}:{key} with 24h TTL, plus a lock key with 60s TTL
• PostgreSQL idempotency_records table with PRIMARY KEY (api_key_hash, idempotency_key) — the airtight backstop
• A duplicate INSERT raises a unique-constraint violation, which we catch and convert into a replay
• Cache alone isn't enough — Redis restart shouldn't permit duplicates

The dedup algorithm — step by step

1. 1
   Hash the API key, fingerprint the canonicalized request body
2. 2
   Fast path: check Redis for cached response
3. 3
   If found and fingerprint matches, replay; if fingerprint differs, return 409
4. 4
   Acquire a Redis SETNX lock with 60-second TTL
5. 5
   Try INSERT into the idempotency table; on UNIQUE violation, fetch the existing row and replay
6. 6
   Process the payment in a DB transaction that also writes to the outbox
7. 7
   UPDATE the idempotency record with the response, write to Redis cache, release lock, return

Why both lock AND DB constraint

The lock makes the common case (concurrent retries) fast — they wait briefly and replay rather than racing through the slow path. The UNIQUE constraint is the airtight backstop if Redis is down or the lock TTL expires mid-processing.

Request fingerprinting

• Canonicalize the JSON: sort keys, normalize whitespace, lowercase known case-insensitive fields, then sha256
• If a duplicate key arrives with a different fingerprint, return 409 — surfaces the client bug instead of silently doing the wrong thing

Outbox for gateway-DB consistency

• We never want to call Stripe and then fail to record it, or record and then fail to charge
• Same DB transaction inserts the payment row AND a payment_outbox row
• An outbox worker drains pending rows, calls the gateway with our payment_id as the gateway's idempotency key
• If the worker crashes mid-charge, the next worker retries; the gateway's own idempotency returns the original charge result

Long-running flows — 202 Accepted pattern

• 3D Secure, Apple Pay, ACH can take 30 seconds; holding the HTTP connection open guarantees client timeouts and retry storms
• Return 202 Accepted within 2 seconds with payment_id, status: requires_action, and the next action payload (3DS redirect URL)
• Client polls GET /payments/{id} or subscribes to a webhook for completion
• Same idempotency key still identifies the whole flow — a retry while requires_action returns the same next_action

LLD — clean abstractions

• IdempotencyMiddleware wraps the controller action — dedup is orthogonal to payment logic
• IdempotencyStore interface with a two-layer impl (Redis + Postgres); could swap to DynamoDB for multi-region
• RequestFingerprinter for canonicalization rules
• OutboxWriter happens inside the same DB transaction as the payment row

Failure handling

• Redis down: fast path skipped, every request goes through the DB; latency up but correctness preserved
• DB violation race: catch and replay
• Worker crash: lease expires, watchdog re-marks pending, next worker retries
• Gateway success but our DB write fails after: outbox row stays processing, watchdog re-runs, gateway returns idempotent success, DB updates
• Client retries with different body: fingerprint mismatch, 409
• Old records: daily cleanup of expires_at < NOW()

Production extras

• API key hashed (sha256) before storage so leaked DB doesn't expose secrets
• Stored response bodies encrypted at rest
• Audit log of every payment attempt for regulatory retention
• Distributed tracing through middleware → service → outbox worker → gateway

At 10× scale

• DynamoDB Global Tables for multi-region active-active dedup with conditional writes
• CDC (Debezium) replacing outbox polling
• Per-merchant TTL config

Recommendation

Client-generated UUID Idempotency-Key, two-layer dedup (Redis fast path + PostgreSQL UNIQUE backstop), request-body fingerprint check returning 409 on mismatch, transactional outbox for gateway consistency forwarding our key as the gateway's idempotency header, and 202 Accepted for any flow longer than 2 seconds — the same pattern Stripe, Adyen, and Square run in production.