Giving Your AI a Job Interview
Source: One Useful Thing
Author: Ethan Mollick
Original source: https://www.oneusefulthing.org/p/giving-your-ai-a-job-interview
Source type: essay
Private backup: the full article text is archived in the private repository at archives/articles/oneusefulthing-org-giving-your-ai-a-job-interview.source.md. It is not published on the public Quartz site.
Summary
Ethan Mollick argues that public AI benchmarks are useful for tracking broad model progress but insufficient for deciding which AI system is best for a specific person or organization. Benchmarks can be contaminated, poorly calibrated, narrow, or disconnected from local work. Individuals may rely on personal “vibes” tests, but organizations need rigorous evaluations built from realistic tasks, repeated trials, and expert assessment. The core metaphor is that AI systems should be evaluated less like generic software and more like candidates interviewing for the actual work and judgments they will perform.
Pull quotes
Interview the AI
“To figure this out for yourself, you are going to need to interview your AI.”
— Mollick, arguing that model choice depends on local work rather than generic rankings.
Benchmark the real work
“You shouldn’t just pick a model for your company, you need to conduct a rigorous job interview.”
— Mollick, framing AI selection as evidence-based hiring rather than software shopping.
Test actual judgments
“You need to systematically test your AI on the actual work it will do and the actual judgments it will make.”
— Mollick, on designing realistic evaluations.
Not SAT scores
“You wouldn’t hire a VP based solely on their SAT scores.”
— Mollick, warning against overreliance on generic benchmark scores.
Big ideas
- AI is changing what knowledge work asks people to do
- Students need to check AI answers against real evidence
- AI tools should be judged by the work they will actually do
Claims
Key evidence and examples
- The article lists benchmark problems including training-data contamination, unclear measurement, calibration issues, and question errors.
- Mollick contrasts personal tests such as image prompts or small coding/writing challenges with organization-scale evaluations.
- OpenAI’s GDPval is used as an example of workplace-task evaluation with expert-created tasks and blind expert grading.
- The “GuacaDrone” example shows that models can have different optimism, skepticism, and risk-tolerance patterns that shape recommendations.
Education relevance
Useful for schools and universities choosing AI tools or teaching AI literacy: educators should test tools against real instructional, assessment, advising, administrative, and policy tasks rather than relying only on vendor claims or leaderboard scores.
Durability note
Benchmarks and model names will change, but the article’s core practice—testing AI on real local work before trusting it—should stay relevant for procurement and classroom tool selection.