The easy question does not test trust
An agent correctly finds business hours. Then someone asks whether an undocumented exception applies to their case. If an invented answer sounds as confident as the first one, the user can hardly distinguish knowledge from assumption. Evaluation must include questions where acknowledging missing information is the appropriate answer.
Define expected behavior: cite an available source, ask for missing detail, or hand off to a person. “I don’t know” should open a useful next step. Refusing everything also fails: the goal is to help within clear boundaries, not maximize answers or refusals.
Build a test set that challenges the agent
Include common cases, ambiguous questions, outdated information, out-of-scope requests, and users with different permissions. For each case, define an acceptable response and a serious error. Include users’ languages: translating a question does not guarantee equal accuracy in every version.
Repeat evaluation after changing instructions, documents, or tools. Compare by case type, not just the average. A 95% overall result may hide the five questions with the most serious consequences. Test size and representativeness limit what you can infer about real use.
Answering and acting carry different risks
A retrieved document may contain malicious instructions aimed at the agent. Treat that content as data, not authorization. Limit tools and permissions; the system executing an action should check access independently of generated text. An instruction to “be careful” does not replace these controls.
For consequential actions, such as sending communications or changing important records, show the destination and exact change before confirmation. Record the result and support investigation. Keep oversight proportionate to risk: evaluation and controls reduce problems; they do not prove failures are impossible.
Start with twelve well-chosen cases
Write three normal questions, three ambiguous ones, three unsupported by the sources, and three that challenge permission boundaries. Save expected responses, agent version, and results. Fix the highest-impact errors first and repeat. Twelve cases are a learning starting point, not a safety certificate.
Sources & further reading
Sources accessed October 24, 2026. Numerical examples and scenarios are illustrative.
Ready to put this knowledge into practice?
We can work together on your team’s challenge or your next project.
Talk to Calembe ↗