Your team disagrees about a good AI answer. Build a scoring sheet together.
One colleague calls the AI answer helpful. Another says it is too long. A third notices that the instructions describe a button the product does not have. Before changing the prompt, check whether everyone is judging the same thing.
An evaluation is a test of an AI system against an expected result. A scoring rubric describes what reviewers look for and how they assign scores. You can start with a shared document rather than an automated testing platform.
This exercise helps a team turn different opinions into explicit rules. You will compare three answers, examine a disagreement and build a scoring sheet that keeps serious errors visible.
Agree the task before judging the answer
In Product Talk’s 27 October discussion about defining good AI results, Teresa Torres and Petra Wille argue for judging value in the user’s context. The practical question here is smaller: can colleagues apply the same expectations to one answer?
Example: imagine your team is testing an AI help assistant in a project-management app. A person using the app for the first time asks:
How do I export only the completed tasks from my project?
You and two colleagues are the reviewers: the people checking whether the AI gives a useful, correct answer. You will assess three possible answers to that same question. First, everyone reads the help note below, which explains how this example app works.
Help note for this example app
- Open the project you want to export from, then select its Tasks tab.
- Set the Status filter to Completed.
- Choose Export, then CSV. The downloaded file should contain only the tasks matching the filter.
- Open the file and check that every task row is marked Completed.
CSV stands for comma-separated values. It is a file format for rows and columns that you can open in a spreadsheet.
Who can export? People with an editor or owner role. People with a viewer role cannot export.
What if Export is missing? Ask the project owner to check your role. Do not assume access will be granted.
The app and help note are made up for this exercise. In your own project, use the current product documentation instead. Everyone should receive the same version.
Your task: compare each AI answer with this help note. Check whether the answer gives the right steps, includes the information a first-time user needs and explains who can export. You are assessing the AI’s answer, not the user’s ability to follow instructions.
Compare three answers
Answer A: polished but unsupported
Purpose: Notice claims that conflict with the reference.
“Open Quick Export, select Completed and download your file. Anyone on the project can do this.”
Answer B: correct steps, missing support
Purpose: Separate an omission from an invented instruction.
“Open Tasks, filter Status to Completed, then choose Export and CSV.”
Answer C: complete for this practice task
Purpose: Help the first-time user act and check the result.
“Open your project and select Tasks. Set Status to Completed, then choose Export and CSV. Open the downloaded file and check that its task rows are marked Completed. You need an editor or owner role to export. If Export is missing, check your role with the project owner.”
Use the disagreement to improve the rule
Start with a deliberately loose criterion: “The answer is complete.” Two reviewers give Answer B different scores. Ask each to identify the evidence behind the score before discussing who is right.
| Reviewer | Initial score and reason |
|---|---|
| Reviewer 1 | 2 out of 2: the filter and export sequence is there. |
| Reviewer 2 | 1 out of 2: a first-time user is not told to choose the project or check the file. |
Do not average these scores to 1.5 and move on. The team has discovered two interpretations of “complete”. Rewrite the rule around observable evidence: name the project, give the filter and export steps, and explain how to check the downloaded rows. Score permission guidance separately under handling limits.
Write a small rubric with visible boundaries
Use 0, 1 or 2 for each criterion below. The numbers are ordered judgement labels, not precise measurements of quality. Keep the rubric small enough that reviewers can explain every score.
| Criterion | Score descriptions |
|---|---|
| Accuracy | 0: an instruction contradicts the reference or invents a feature. 1: a factual claim cannot be checked against it, without a direct contradiction. 2: every stated product claim is supported. |
| Completeness | 0: the documented Status filter or Export then CSV sequence is absent. 1: those actions are present, but choosing the project or checking the file is omitted. 2: all three parts are present. |
| Clarity | 0: wording leaves the action or order unclear. 1: understandable, but some navigation terms need interpreting. 2: explicit navigation labels and a clearly stated action order. |
| Handling limits | 0: permitted roles are missing or wrong. 1: permitted roles are stated, but no next step is given if Export is absent. 2: roles and the project-owner check are both explained. |
Add a separate must-fix check: invented controls or incorrect access claims block acceptance, regardless of the total. For this exercise, accept an answer only when no must-fix issue exists and every criterion scores 2. This is a proposed rule for this task, not a universal release standard.
| Answer | Scores and decision |
|---|---|
| A | Accuracy 0; completeness 0; clarity 2; limits 0. Total 2/8. Blocked: invented Quick Export and incorrect access claim. |
| B | Accuracy 2; completeness 1; clarity 2; limits 0. Total 5/8. Revise: add project selection, file check and permission guidance. |
| C | Accuracy 2; completeness 2; clarity 2; limits 2. Total 8/8. Meets this example’s rule. |
Run a 20-minute calibration session
Calibration means checking that reviewers interpret the scoring rules consistently. Anthropic’s evaluation guidance, published in January 2026 discusses human grading and calibration within a wider evaluation process. The short session below is a suggested starting exercise, not a validated measurement method.
- Minutes 0–3: agree the reader, task and reference version.
- Minutes 3–8: independently score the same three answers and mark supporting phrases or omissions.
- Minutes 8–15: discuss differences, clarify boundaries and record unresolved questions.
- Minutes 15–20: independently re-score one answer, compare again and assign an owner to test fresh examples.
Keep original and revised scores. If disagreement remains, record it rather than requiring consensus. An unclear reference may need a product decision before any answer can be judged fairly.
Copy this into your next review
Try new examples after revising the sheet, including a viewer asking to export and a user whose file contains unexpected rows. Change the expected answer with the question; do not reward identical wording across different needs.
Three constructed answers cannot establish reliability, fairness or performance across users. They help expose assumptions. Keep checking current documentation and real task outcomes before making a broader release decision.
Start with one answer colleagues already disagree about. Replace one vague judgement with a rule another person can actually apply.
You May Also Like
Inherited a messy product backlog? Start with the next decision
What is Product Discovery? A Friendly Dive into the World of Innovation
