How to measure whether an AI feature is actually helping users

People are opening your new AI assistant. They are submitting questions and trying different prompts. Your dashboard is busy, but one question remains: is the feature helping them finish the work they came to do?

Activity measures can tell you whether people are trying something. They do not, on their own, explain whether the answer was useful, correct or worth the time spent checking it.

Start with a specific user task. Then build a small scorecard that connects usage with results, quality and effort. You can expand it as you learn; you do not need thirty measures before making a useful decision.

Define success before choosing the numbers

Consider a fictional assistant that drafts project handover summaries. Its intended benefit is helping a colleague understand decisions, open questions and next actions from approved source notes.

A successful task might require a summary that includes all agreed decisions, introduces no unsupported claims and can be approved after review. Define those checks with the people doing the work.

The measures below are a suggested starter scorecard, not an industry benchmark. Agree the task boundaries, measurement period and people included before comparing results.

Five questions for your scorecard

1. Are people completing the intended task?

Task completion rate compares successful tasks with eligible tasks started. In the handover example, opening the assistant is not a completed handover.

Define what counts as a start, how retries are grouped and how completion is confirmed. If one person submits five prompts for a single handover, counting five separate successes could give a misleading picture.

Include abandoned attempts. A completion rate based only on finished tasks hides the people who gave up.

2. Is the result good enough to use?

Check output quality against agreed criteria. For a handover, these might include factual accuracy, coverage of decisions and correct handling of missing information.

Review a sample of ordinary and difficult tasks. Record the types of errors, not just a single average score. An invented approval is more consequential than a clumsy sentence, even if both require one edit.

Keep confirmed defects separate from users’ preferences. A satisfaction rating can add context, but it does not establish factual accuracy.

3. What happens to total effort?

Measure the full workflow: preparing the input, generating the draft, checking it, correcting it and producing the usable result. A short generation time can coexist with a long review.

Compare similar tasks using the existing process. Note differences in length, complexity and reviewer experience. If the easier tasks use AI and the harder tasks do not, the comparison will be difficult to interpret.

Ask users what checking work has become easier or harder. Their explanation can reveal effort that your timing misses.

4. Can people recover when it goes wrong?

Track tasks that require a restart, manual completion or escalation. These are different outcomes, so record them separately where practical.

Make sure users can reach the source material and continue without the assistant. For the fictional handover, a reviewer should be able to correct a misleading draft rather than feel forced to accept it.

Investigate repeated failure patterns. A particular document type or missing piece of context may explain more than the overall failure rate.

5. What does a successful task cost?

Compare the costs within a defined boundary with the number of successfully completed tasks. Say whether the cost includes only the AI service or also review time, supporting software and maintenance.

Include failed attempts and retries in the cost total. Otherwise, an unreliable feature can appear cheaper than it is. Keep service cost and total operating cost clearly distinguished if you report both.

A worked example

These figures are illustrative. In a small pilot, 40 eligible handovers are started and 30 meet the agreed success criteria. Task completion is 30 ÷ 40 × 100 = 75%.

The AI service costs £12 across all 40 attempts. Service cost per successful handover is £12 ÷ 30 = £0.40. That excludes staff time and maintenance, so it is not the total cost of the process.

The team also records review time and error types. The completion rate alone does not tell them whether the ten unsuccessful tasks are acceptable or whether successful tasks took too much effort.

This example is too small to establish a universal benchmark. It helps the team decide what to investigate next.