The Error Message Is Lying To You
An intermittent production failure with an error message that points at the wrong thing. It takes 75 minutes, the full rubric is published below before you start, and AI tools are allowed.
The brief
The situation. A background job syncs orders from a partner API into your database. It runs every 15 minutes. Roughly twice a day it fails with TimeoutError: connection pool exhausted. On failure, some orders are written twice.
The obvious fix already tried. The pool size was raised from 10 to 50. Failure frequency dropped to once a day. It did not stop, and the duplicate writes continued at the same rate.
What you have. The sync function, the partner API docs, and logs from three failures.
What the logs show. Failures correlate with partner API responses that took over 8 seconds. The sync opens a database transaction, then calls the partner API inside it, then writes rows. There is a retry wrapper around the whole sync function with 3 attempts and no backoff. The partner's docs note that POST /orders/ack is not idempotent and that rate limiting returns a 429 with a Retry-After header.
Your task. Diagnose the real cause, explain why raising the pool size helped without fixing it, and specify the fix. Address the duplicate writes explicitly — they are a separate defect with the same root, and finding both is the difference between a Level 2 and a Level 3 submission.
What you hand in
A written root-cause analysis: the actual cause, why the error message is misleading, why the pool increase partially masked it, and a specified fix covering both the timeouts and the duplicate writes.
Any tools including AI assistance. An assistant given the error message alone will suggest raising the pool size — which has already been tried, and is the trap.
How is this challenge scored?
Every criterion, with its weight and the skill it produces evidence for. Nothing else is measured — not writing quality, not grammar, not length. You see this before you start, and there are no hidden criteria.
Names holding a database transaction open across a slow external API call as the cause of pool exhaustion. Stopping at 'the partner API is slow' scores 1.
Articulates that a larger pool raises the threshold without removing the coupling between external latency and connection hold time.
Connects retry-without-idempotency plus a non-idempotent partner endpoint to the double writes. Missing this entirely caps the overall level at 2.
Moves the API call outside the transaction, adds idempotency keys or a dedupe check, and honours `Retry-After` with backoff. Partial fixes score proportionally.
Describes how you would confirm the fix in production — what to instrument and what would indicate the problem persists.
Which skills does this prove?
Diagnoses failures where the error is misleading, tests the cheapest hypothesis first, and verifies the root cause before fixing.
Designs for idempotency and partial failure, and makes state recoverable when the remote system misbehaves.
Ranks findings by severity, catches security and data-integrity issues, and writes feedback the author acts on.
Questions people ask
- Do I need to write the fixed code?
- No. A precise written specification of the fix scores identically to code. Reasoning is what is measured.
- What if I only find one of the two defects?
- You can still reach Level 2. Level 3 requires connecting both to the same root, which is the judgment being tested.
- Is the stack relevant?
- No. The failure pattern is language-agnostic and the code is presented in pseudocode-adjacent form for exactly that reason.