How to tell if an AI agent is working: five metrics for management

An AI agent in production generates plenty of activity: conversations, cases handled, records. Management needs something simpler: whether it resolves the work it was given, and whether it does it well. Five metrics answer that question, as long as they are measured against a starting point.

Before you start: the baseline

The baseline is a picture of how the process works today, without the agent. Without it, any figure looks good or bad depending on who presents it.

Before going live, spend a few weeks measuring what you will measure afterwards: how many requests arrive, how long the first answer and the resolution take, how many cases are corrected, and the business result the process affects. If the data does not exist, collect it by hand from a sample. A rough baseline is worth more than none.

That is how a result like Recu’s is measured: +38% recovery on overdue accounts against the previous period.

The five metrics

1. Requests resolved with no human intervention

The share of requests the agent closes from start to finish without a person touching them, out of the total received in the period. At Cootradecun, 85% of requests are resolved with no human intervention.

The point to watch is what “resolved” means. A conversation that ended is not always a resolved case: if the customer stops replying, or comes back the next day with the same request, the case is still open. Write that definition down before you start.

2. Response time and resolution time

These are two different clocks. Response time runs from the moment the request arrives to the first useful answer; resolution time runs until the case is closed, including the cases that went to a person. An agent answers almost instantly, so resolution time is the figure that tells you more.

Measure both from the same starting point and compare them with the baseline. Averages hide the long cases: look at the slowest ones too, since that is usually where the problem sits.

3. Escalations to a person, and why

Escalating means handing the case to someone on the team, and it is often the right call.

A case is well escalated when it needs human judgment (an exception, a complaint, a request outside the agent’s limits) and reaches the person with full context, so the customer does not have to tell the story again. It is poorly escalated when the agent could have solved it with the information it had, when it escalated late or when it arrived without context.

Count the total escalations and sort a sample by reason. The sensible goal is for poor escalations to fall week after week. Good ones can hold steady, and it is healthy that they exist.

4. Errors and corrections

An agent can answer with confidence and still be wrong. The only way to know is to review a sample of conversations or cases every week, with someone who knows the process, and grade each one: correct, incomplete or wrong. Add the corrections on top: cases where a person had to undo or fix what the agent did.

The sample should be random, since complaints only reveal the errors the customer noticed. If the agent answers from company documents, count any answer without a source as an error, even when it happened to be right (here is why).

5. Business result

The first four metrics tell you whether the agent works; this one tells you whether it is worth it. It depends on the process: in collections, recovery on overdue accounts; in internal processes, the team time freed up; in customer service, satisfaction when the case closes. At Legalnova, for example, time spent on document review fell 70%.

Pick one or two results before you start, the ones that justify the project, and measure them against the baseline.

Weekly at first

The first weeks in production call for the most adjustments, so review them every week with the process owner: the five metrics, the error sample and the poorly escalated cases. Each finding becomes a concrete adjustment, such as a new instruction, a missing document or a clearer limit. Once the figures settle, the review can move to monthly.

No metric on its own

Each metric can improve at the expense of another, which is why none should be celebrated in isolation:

• More requests resolved without a person can mean the agent stopped escalating cases that needed one.
• Faster answers can come with more errors.
• Fewer escalations can hide customers who gave up and left.

Read the five together. A real improvement moves the business result without making errors worse.

How we do it at Xplouse

We measure the baseline during the diagnostic, before building anything. Every action the agent takes is recorded (each answer and each escalation, with the information it consulted) and can be reviewed, so the weekly sample comes from a complete record. The metrics are in view for management from day one. The architecture shows how it works.