Is your AI summary reliable? Build a five-case evaluation pack

The AI summary is beautifully written. It has also turned “we need to discuss the deadline” into “the deadline is Friday”. A tidy paragraph can hide a change that matters.

An AI evaluation, often shortened to an eval, checks an output against a stated expectation. For a product or delivery team, a small evaluation pack can make those expectations concrete before a summary becomes a decision record.

This guide uses fictional project notes to build a five-case practice pack. It is a starting exercise for an internal summarising workflow, not evidence that an AI tool is ready for unrestricted use.

Choose one job for the summary

Imagine a team wants a short handover containing agreed actions, named owners and unresolved questions. Its job is to help a colleague continue the work. It should not choose priorities, assign people or fill gaps in a meeting.

Write that scope above your test cases. If the team also wants recommendations, evaluate them separately and label them as suggestions. Otherwise, a useful-sounding recommendation can quietly become an instruction.

1. Write the expected facts first

Purpose: Give reviewers an independent reference.

Create a short note yourself and mark the facts a summary must preserve. Example: “Maya will check the export issue. No delivery date agreed. Accessibility question remains open.” A correct summary may use different wording, but it must preserve those meanings.

Include a prohibited claim for each case. Here, “Maya will fix the issue by Friday” adds both a commitment and a deadline. Make that a failure even if the rest of the summary reads well.

2. Build five different cases

Purpose: Challenge different ways a summary could mislead.

Use an ordinary handover, a missing owner, a changed decision, an unresolved disagreement and an explicit exclusion. Keep the inputs small enough that a reviewer can inspect every sentence. Use invented or approved sanitised material rather than pasting confidential team notes into an unapproved service.

For a changed decision, include an early suggestion and a later correction. For an exclusion, write that a particular detail must not appear in the handover. These cases test the task you defined, rather than the model’s ability to sound fluent.

Practise judging the output

Fictional input: “Aisha suggested a Thursday launch. Leon said the support rota was not confirmed. The team agreed to decide after the rota check. Leon will check it. No launch date was agreed.”

Proposed AI summary: “The team agreed to launch on Thursday. Leon will confirm support.” Decide whether you would accept this before opening the explanation.

Reveal the review

Reject it for this handover task. Thursday was a suggestion, not an agreed date. A more faithful summary is: “Leon will check the support rota. The launch date remains undecided until that check is complete.” The reviewer can accept different phrasing if the same facts and uncertainty remain intact.

3. Record failures by type

Purpose: Keep one serious mistake visible.

For each output, mark factual fidelity, action ownership, uncertainty and exclusions as pass, fail or not applicable. Add the exact input and output sentence that explains a failure. A single overall score can hide an invented deadline among several correct details.

Run the same cases again to look for variation. Keep the prompt, tool or model version where available, date and relevant settings with the results. Repeated success on five cases is limited evidence; it does not estimate reliability across all your team’s notes.

Agree what happens next

Before wider use, agree who reviews summaries and what blocks sharing. For this practice workflow, invented owners, deadlines or decisions should stop the handover until a person corrects it. Add each new failure to the pack and recheck when the workflow changes.

Use fresh cases as well as the ones used to improve the prompt. Otherwise, you may only show that your changes fit the examples you already knew. A larger representative set and appropriate operational checks are needed for broader deployment.