← Videos

Does Your AI Work - Evaluation
One impressive answer is encouraging. What happens when the source is missing, two documents disagree, or a tool fails?
From the episode description
One impressive answer is encouraging. What happens when the source is missing, two documents disagree, or a tool fails?
We turn the series’ fictional Workshop Notes into a small evaluation plan. Before looking at an answer, we decide what success means—and when a useful response should express uncertainty or stop.
In this episode:
• Five cases covering supported answers, missing evidence, conflicting sources, prompt injection and tool failure.
• Separate checks for facts, citation support, permitted actions and actual execution.
• Where simple code checks help, and where judgment is needed.
• Why reviewers and model-based judges can also make mistakes.
• How repeated trials, recorded configurations and additional test cases help reveal regressions.
These are designed teaching cases, not reported benchmark results. The unresolved-source case deliberately changes the documents’ authority information; it is different from choosing a clearly current note over an explicitly outdated one.
Start with a few cases that represent the job you need done. Keep the actual responses and tool results, then trace one failure to the stage where it began. The aim is a repeatable way to examine changes, with clear limits on what the results establish.
ResearchNode: AI from the ground up • Episode 10 of 10.
Previous: Can a Document Trick an AI Assistant?
Presented by Michelle, an AI-generated presenter.
Sources and further reading:
Demystifying evals for AI agents — Anthropic
https://www.anthropic.com/research/demystifying-evals-for-ai-agents
Inspect AI documentation
https://inspect.aisi.org.uk/
Inspect AI scorers
https://inspect.aisi.org.uk/scorers.html
Descriptions and chapter markers are supplied by the channel. They are not a transcript or an independent check of every claim in the video.