Quality

AI Evaluation Engineer

Write the cases that catch fluent lies. Gold sets, red teams, and a gate that can fail the build.

2 weeks · gold set, red team, a gate

0/11 lessons on this path

Mastery syllabus

1. 1 · Gold cases

   Easy, hard, empty, fluent lie.

   • Measure before you brag11 min
   • Proof over confidence11 min
   • Closed book vs open notes12 min

2. 2 · The gate

   Fail the build. Show the miss.

   • Crush the take-home10 min
   • Where AI must not decide9 min
   • Integrity is the grade9 min
   • Teach the human who scores12 min

3. 3 · Gold and judge

   Twenty cases. A judge is a sieve, not a court.

   • Twenty cases, four kinds12 min
   • LLM-as-judge is a tool, not a court12 min

4. 4 · Catch the lie

   The invented citation goes first in the file.

   • The user is not the boss of the rule12 min
   • Closed book vs open notes12 min

Simulation labs

• EvalSuite

  Twenty cases. One must refuse.

  Open the lab

• Ground or refuse

  No chunk, no decoration.

  Open the lab

• Twenty cases

  Five of four kinds.

  Open the lab

• Rater guide

  Two humans, one lie.

  Open the lab

The project that hires

Twenty cases and a gate that can fail the demo

1. Easy, hard, empty, lie — five each.
2. A rater guide of one page.
3. One case that caught a real invented citation.

Interview room

• Write one fluent-lie trap for a handbook bot.
• How do you score groundedness with a human rater guide?
• Empty retrieval: what must the system do, and how do you test it?
• LLM-as-judge disagrees with two humans. Who wins?

Resume lines that hire

• Wrote a 20-case eval that caught invented citations before a client demo.