Experiment Design
Experiment design is the skill of building a test that can actually answer the question. It is assessed on whether the design isolates the variable and whether the candidate knows in advance what result would change their mind.
Also called: A/B testing, causal inference, test design
What is experiment design?
Experiment design covers hypothesis formation, choosing a metric that reflects the real goal, sizing and duration, controlling for confounds, and pre-committing to a decision rule before results arrive.
Badly designed experiments produce confident wrong answers, which are worse than no answer. Pre-registration of the decision rule is the single practice that most improves organisational decision quality, and almost nobody does it.
What do the levels of experiment design mean?
Four levels, each defined by observable behaviour rather than by years. This is the definition employers set their bar against, and the one every proof is scored to.
- L1Aware
Can set up a simple A/B test and read the result.
Evidence: A running test with a defined metric.
- L2Working
Chooses a sensible primary metric and understands why sample size and duration matter.
Evidence: A test design with sizing reasoning.
- L3Independent
Isolates the variable, controls for confounds, and pre-commits to a decision rule before seeing results.
Evidence: A pre-registered design with a stated decision rule.
- L4Leading
Builds experimentation practice, prevents common inference errors across teams, and knows when an experiment is the wrong instrument.
Evidence: An experimentation standard in use across an organisation.
| Level | Label | People proven here |
|---|---|---|
| L1 | Aware | 0 |
| L2 | Working | 0 |
| L3 | Independent | 1 |
| L4 | Leading | 0 |
How is experiment design assessed?
Candidates design a test for a proposed change. Scoring rewards a falsifiable hypothesis, a metric that resists gaming, and an explicit statement of which result would cause them to abandon the idea.
Which roles require experiment design?
| Role | Level required | Status |
|---|---|---|
| Product Manager | L2 Working | Nice to have |
| Growth Analyst | L3 Independent | Required |
How do you prove experiment design?
Experiment Design: questions people ask
- Do I need statistics training?
- Level 3 requires understanding why sizing and confounds matter, not the ability to derive a formula. Candidates from ops and product backgrounds regularly reach Level 3.
- What is 'pre-committing to a decision rule'?
- Writing down before the test what result would make you ship, iterate, or abandon. It is what prevents reading whatever you hoped for into an ambiguous result, and the rubric weights it heavily.
- Is this only for growth roles?
- No. AI teams need it for model comparison, product teams for feature decisions. It appears in the skill spec for more roles than any other Data skill.