How to interview AI engineers
Interview AI engineers on production LLM work: fix a broken retrieval pipeline, design a service with evals and fallbacks, then defend the trade-offs.
- 1✓AI screening interviewLLM features taken to productionARIARunningStrong · 88
- 2✓Live codingA retrieval bug in a RAG service we builtMAXRunningBug fixed
- 3✓System designA support assistant with evals and fallbacksMAXRunning1 gap: fallback
- 4✓Follow-upCost, latency and failures in the designNIXRunningHire · 85
The AI engineer interview loop
Four rounds, each run by an AI interviewer with its own report. Candidates take every round on their own schedule.
Tell me about an LLM feature you took to production. What broke after launch?
Round 1 · AI screening interview
LLM features they have taken to production
A real two-way conversation that follows up on the answers the candidate actually gives.
- 1✓RetrieveHybrid search over the docs, filtered by productSearchRunningSolid
- 2✓AnswerThe model writes only from cited chunksLLMRunningCites sources
- 3✓Evaluate300 graded questions run on every changeEvalsRunningGates deploys
- 4✓FallbackModel down: retry the same providerFailoverRunningNo second provider
What if retrieval finds nothing relevant?
Round 3 · System design
An LLM service with evals and fallbacks
The candidate designs on a shared whiteboard while the AI probes trade-offs and failure modes.
Answers cite the wrong documents. Find the bug and fix it.
| Step | What the candidate did | Check |
|---|---|---|
| Root cause | Queries embedded with a newer model than the index | CheckingFound |
| Fix | Read the model from the index metadata | CheckingTests pass |
| Guard | Fail at startup on a model mismatch | CheckingAdded |
Round 2 · Live coding
A retrieval bug in a RAG service we built
The candidate solves a live problem while the AI scores the result and the reasoning behind it.
Your design costs $0.12 per answer. How would you cut it in half without losing quality?
Round 4 · Follow-up interview
Cost, latency and failures in their design
The AI questions the candidate on their own earlier work, so you see whether they understand it.
What to assess in a AI engineer interview
Score every candidate on the same 6 skills, rated against evidence from the rounds, so two reviewers reach the same call.
- LLM application engineering
- Retrieval and RAG
- Evaluation and testing
- System design for AI services
- Cost and latency trade-offs
- Production reliability
AI engineer interview questions
Use them in your own loop, or let ARIA, MAX and NIX ask them, follow up on the answers and score them.
Tell me about an LLM feature you took to production. What broke after launch?
This RAG service returns answers from the wrong documents. Find the bug and fix it.
Design a support assistant that answers from our docs. How do you test it, and what happens when the model is down?
Your design costs $0.12 per answer. How would you cut it in half without losing quality?
How would you know a model upgrade made answers worse before customers did?
Integrity in every round
A second voice or screen, pasted work and claims that contradict earlier rounds are flagged with timestamps.
No scheduling between rounds
AI engineers take each round when they are ready, in any time zone.
Your team makes the call
Hello Recruiter evaluates and organises the evidence. People decide who moves forward.
Frequently asked questions
How do you interview AI engineers?
Interview AI engineers on production LLM work: fix a broken retrieval pipeline, design a service with evals and fallbacks, then defend the trade-offs. A strong loop has four rounds: AI screening interview (LLM features they have taken to production), live coding (a retrieval bug in a RAG service we built), system design (an LLM service with evals and fallbacks) and follow-up interview (cost, latency and failures in their design).
What skills should you assess in a AI engineer interview?
LLM application engineering, Retrieval and RAG, Evaluation and testing, System design for AI services, Cost and latency trade-offs and Production reliability. Rate each one Very strong, Strong, Weak or Very weak against evidence from the rounds, not impressions.
How does Hello Recruiter detect cheating in AI engineer interviews?
Every round is monitored for integrity signals such as a second voice or screen, work pasted in from outside the workspace, and claims that contradict earlier rounds. Each signal is timestamped in the candidate's report.
What does the hiring team get after each round?
A report with an AI score out of 100, criteria marked Met, Not met or Not yet evaluated, skills rated Very strong to Very weak, a risk level and timestamped integrity signals. Hello Recruiter evaluates; your team decides.
More go-to-market interview guides
How to interview sales engineers
AI screening interview · Take-home assignment · Presentation · Role-play
How to interview sales engineers
AI screening interview · Take-home assignment · Presentation · Role-play
How to interview sales engineers
AI screening interview · Take-home assignment · Presentation · Role-play
Run this loop on your next AI engineer hire
Start from this loop or generate one from your job description. Your team makes the call.
