How to Hire for AI Roles That Have No Job Title Yet

Because every proxy is broken at once. The field is about three years old, so "5 years of experience" selects for people who renamed existing work. There is no degree, so education filters nothing. There is no standard title, so keyword matching returns noise. And the strongest candidates frequently learned on side projects while employed under an unrelated title.

The result is a screen that systematically deletes your best applicants and forwards your most fluent ones.

Four, in rough order of how often they are the deciding factor:

AI evaluation. Can they define what "good" means for a fuzzy task and build a test set that catches real failures? This is the bottleneck skill on almost every AI team and the one candidates most often lack. It is also the one that never appears on a CV, because no job title contains it.

Agent design. Can they design a tool surface, bound a loop, handle partial failure — and say when an agent is the wrong answer? The last part is the senior signal.

Prompt engineering. Not clever phrasings. Can they diagnose why a prompt fails on input 600 and fix it structurally?

LLM application architecture. Where does the model sit, what is it trusted with, and what happens when it is wrong?

Notice what is not on the list: knowledge of any specific framework. Framework knowledge is a week of ramp-up. These four are not.

Give a realistic design problem with genuine risk in it and score the judgment. Our AI Builder challenge asks candidates to design an agent that issues refunds — a task where mistakes cost real money — and the highest-weighted criteria are failure handling, the pre-launch evaluation plan, and whether the candidate explicitly separates deterministic policy arithmetic from model judgment.

The most informative criterion is one that seems odd until you use it: *did they answer whether this should be an agent at all?* Candidates who reflexively build the agent they were asked for score below candidates who argue, with reasons, that a narrower system is safer.

Yes, and they should — using AI well is the job. Design for it. An assistant will produce a competent generic agent design; it will not decide what your company refuses to automate, where the approval gate goes, or what the launch bar is. Those are the weighted criteria.

In practice, unedited assistant output is easy to spot: it is comprehensive, balanced, and commits to nothing. That pattern reliably scores in the middle of the scale, which is the correct outcome.

Overwhelmingly from adjacent roles. Backend engineers who shipped an AI feature. Support engineers who found the failure patterns and then built the fix. Data analysts who moved into evaluation. Founders of startups that did not work out.

Every one of those has a CV that fails an AI-role keyword screen. Assessing skills directly is how you reach them, and it is why teams that switch usually report the same thing: their strongest hire was someone the old screen had already rejected.