Ship an AI Agent That Won't Embarrass You
Design an agent that processes refund requests — then prove you know where it will fail and what stops it. It takes 90 minutes, the full rubric is published below before you start, and AI tools are allowed.
The brief
The situation. You have joined a 20-person e-commerce company. Support handles roughly 400 refund requests a week. Each one means reading a customer email, checking the order in the admin system, applying a refund policy with several exceptions, and either issuing the refund or writing a rejection.
The CEO has asked for "an AI agent that just handles refunds." Support is nervous. Finance is very nervous — last quarter a scripted automation double-refunded 60 orders before anyone noticed.
Available tools. You have API access to: get_order(order_id), get_customer_history(email), issue_refund(order_id, amount), send_email(to, body), and escalate_to_human(reason).
The refund policy. Full refund within 30 days if unopened. 50% within 60 days if opened. No refund after 60 days, *except* for a documented defect, which is always a full refund at any age. Customers with more than 3 refunds in 12 months go to manual review.
Your task. Produce a design, not code. Cover: which tools the agent gets and which it does not; how the loop is bounded; what happens on a partial failure halfway through; where a human must approve; and how you would know, before launch, whether it is good enough.
The uncomfortable question you must answer explicitly: is an agent the right solution here at all, or would a narrower system be safer? Argue your position either way — both can score full marks.
What you hand in
A written design covering tool surface, loop bounds, failure handling, human approval points, and your pre-launch evaluation plan. Diagrams optional. 800–1,500 words is typical.
Use any AI assistant you like — we assume you will. The rubric targets judgment calls an assistant will not make for you: what you refuse to automate, where you put the approval gate, and how you would know it works.
How is this challenge scored?
Every criterion, with its weight and the skill it produces evidence for. Nothing else is measured — not writing quality, not grammar, not length. You see this before you start, and there are no hidden criteria.
Each tool granted is justified; at least one available tool is deliberately withheld with a stated reason. Awarding `issue_refund` without an approval boundary caps this criterion.
Names what happens when a tool call fails mid-task, how a double-refund is prevented (idempotency, ledger, or approval), and how the system recovers rather than retrying blindly.
Specific test cases targeting real failure modes — policy exceptions, the 3-refund threshold, ambiguous defect claims — plus a stated bar for launch. Vague 'we'd test it thoroughly' scores 0.
Policy is encoded so it applies consistently; ambiguous inputs have a defined behaviour rather than being left to the model's discretion.
Explicitly separates deterministic logic (policy arithmetic, thresholds) from model judgment (intent, defect claims). Trusting the model with the arithmetic caps this criterion at 1.
Takes an explicit position with reasoning tied to the risk profile. Both answers can score full marks; not addressing it scores 0.
Which skills does this prove?
Designs the tool surface deliberately, gates irreversible actions behind approval, and can explain when a workflow beats an agent.
Decomposes unreliable tasks into verifiable steps, chooses between prompting, tool use, and retrieval on the merits, and builds a small evaluation set before changing anything.
Designs evals that target real failure modes, chooses metrics that resist gaming, and knows the limits of what the eval measures.
Makes justified trade-offs between model tiers, caching, and retrieval, and designs explicit fallback behaviour for model failure.
Questions people ask
- Do I need to write working code?
- No. This is a design challenge — code earns no extra credit. The rubric scores design decisions and reasoning, which is why 90 minutes is enough.
- Can I argue that an agent is the wrong solution?
- Yes, and it is a full-marks answer if the reasoning is sound. The criterion rewards taking a defended position, not picking a particular side.
- Which roles does this challenge unlock?
- AI Builder primarily, and it contributes proofs toward AI Engineer and AI Product Manager specs, since agent design and evaluation appear in all three.