🔒 Paid

Idempotency Failure

A background worker checks an idempotency store before calling an external side-effect API (e.g. a fulfillment or payment call) on behalf of queued requests. That store is degrading. Each external side effect must run at most once per logical request — a side effect applying more than once is a critical correctness incident regardless of how healthy request success rate and throughput otherwise look, and dropping a legitimate request's side effect entirely is an equally real integrity failure, not a cosmetic one. Diagnose why the store is failing to prevent duplication and bring the system to a state that is genuinely safe to hand back.

Difficulty
Staff
Expected Time
~20 min
Category
Distributed Systems

Skills

  • Distinguishing availability metrics from correctness guarantees
  • Reasoning about idempotency-key TTL versus realistic retry horizons
  • Evaluating key-scope tradeoffs between false duplicates and missed duplicates
  • Weighing fail-open against fail-closed behavior under a dependency outage

System Architecture

  • Request Source
  • Service A
  • Queue / Worker
  • Idempotency Store
  • External Side Effect

This is a Pro scenario. Upgrade to Pro to unlock it.

Upgrade to ProBack to Incidents