Your QA Score Is High. Your Customers Are Unhappy. Here's Why.

Last quarter, one of our agents scored 91 on a QA review. Clean communication, accurate information, policy followed to the letter. The customer left a one-star CSAT and sent a follow-up email asking to speak to a manager.

That gap, a technically perfect ticket and a genuinely frustrated customer, is what most QA programs cannot explain. And if your program cannot explain it, it cannot fix it.

The problem is almost never the rubric itself. It is what the rubric was built to measure.

The Two Jobs QA Is Supposed to Do

Most support QA programs are built for one job: catching mistakes. Did the agent follow the escalation policy? Did they use the right template? Did they close the ticket inside the SLA window? This is compliance QA, and it is genuinely useful. You need it. Agents who consistently miss procedure create risk for the team and the business.

But compliance QA, on its own, answers the wrong question. "Did the agent follow the process?" is not the same as "Did the customer get the help they needed?" An agent can check every box and still leave the customer feeling dismissed, confused, or unheard. That 91-point review captured exactly this: every protocol followed, zero empathy, no real resolution of the customer's underlying concern.

The second job of QA is development. Not just "did this agent comply," but "is this agent getting better?" Empathy, ownership, knowing when the script is the wrong tool, reading what the customer is actually asking underneath what they typed. These do not show up in a compliance checklist.

What to Sample, and How Much

Before fixing the rubric, fix the sample. Most teams review too few tickets to catch patterns, or pull a biased sample: escalations only, complaints only, the same two struggling agents while the rest go un-reviewed for weeks.

The goal is a representative slice: routine tickets and edge cases, spread across every agent, every cycle.

A workable baseline for a team of five to ten agents:

• Ten to fifteen tickets per agent per month minimum. New agents in their first 90 days warrant more. They are forming habits that are cheap to correct early and expensive to unlearn later.
• Stratify the sample. Roughly half from your high-volume types (billing questions, basic how-tos, account issues) and half from complex or unusual cases. The complex cases are where agents reveal what they do when there is no clear script.
• Include at least one low-CSAT ticket per agent per cycle. The gap between the agent's apparent confidence and the customer's reaction is where the most useful coaching material lives.

Tools like Zendesk QA (which absorbed the former Klaus product) can automate stratified sampling and reduce the manual overhead considerably. Help Scout's reporting surfaces CSAT-flagged conversations in one click. Either way, the sampling logic matters more than the tool.

The Rubric That Coaches Instead of Just Scores

A compliance rubric asks yes/no questions. Did the agent use the approved greeting? Did they include the ticket number? Did they offer a follow-up?

A coaching rubric asks quality questions. Did the agent acknowledge the customer's frustration before jumping to the fix? Did they take ownership of the problem, or deflect to policy? Did the explanation match the customer's technical level? When the first solution did not work, did they re-engage or close the ticket and hope for the best?

The dimensions I use, adapted for a blended human and AI support environment:

• Accuracy. Was the information correct? Non-negotiable, scored binary. A wrong answer that sounds confident is worse than an uncertain answer with a clear path to resolution.
• Ownership and tone. Did the agent treat this like their problem to solve, or a ticket to move out of their queue? Scored 1 to 5. This is the single dimension most correlated with CSAT in my experience.
• Customer-level communication. Did the reply match how the customer communicates? A five-paragraph technical reply to a two-sentence question is a mismatch, even if every sentence is accurate.
• Resolution quality. Not just "was the ticket closed," but does the customer have what they need to not come back with the same question next week? Partial resolutions that generate repeat contacts should score lower even when CSAT is neutral.
• Protocol adherence. The compliance layer. Follow the playbook, escalate correctly, meet the SLA.

Weighting matters. I weight accuracy and resolution quality higher than protocol adherence, because an agent who solves the problem without ticking every process checkbox is more valuable than one who ticks every box and leaves the customer confused. The weighting also signals to agents what the team actually values, not just what it tracks.

Calibration: Why Your QA Falls Apart Without It

The most common reason QA programs stop improving quality is not the rubric. It is calibration, or the lack of it.

Two reviewers scoring the same ticket differently undermines everything. One person scores "I understand your frustration" as good empathy. Another scores it as a hollow template phrase. Without a shared standard, your scores measure the reviewer's mood as much as the agent's quality.

Run a calibration session once a month. Pull three tickets that have already been scored. Have every reviewer score them independently, then compare. Where scores diverge by more than one point on the same dimension, talk through why and document the reasoning. Calibration is tedious. Teams skip it. It is also the practice most correlated with QA programs that actually improve quality, because it forces the team to articulate what "good" looks like rather than assume everyone agrees.

Closing the Loop: How to Turn a Score Into a Better Agent

The score is not the feedback. The score is the starting point.

A QA score emailed to an agent with no accompanying conversation will either be ignored or resented. The agent sees a number. They have no context, no path forward, and no reason to believe this cycle will be different from the last one.

Agents who can self-critique improve faster than agents who wait to be told what to fix. The QA conversation is where you build that capacity.

The feedback structure that works, and that I have used across multiple teams:

1. Start with what was done well. Specific, not generic. "Your second reply was clearer than your first because you dropped the technical terms after the customer said they were not technical" lands better than "good communication." Generic praise is forgotten in thirty seconds.
2. Name one thing to work on this cycle. One. If you give an agent three improvement areas in a single review, they work on none of them. Pick the one that, if fixed, will move their CSAT score the most and come back to the others in later cycles.
3. End with a question. "What would you have done differently looking back at this?" Agents who can name their own gap close it faster. The ones who can't yet are the ones who need the most coaching, and that is useful information too.

The Real Cost of QA That Only Scores

Support QA done right is not about catching bad agents. Most agents in a team are not bad. They are under-coached, reviewing their own work without useful feedback, or following a process that was never designed to teach them anything.

AI can now handle a meaningful slice of tier-one tickets: the routine lookups, the standard how-to responses, the acknowledgments that buy time while a human investigates. What that shift leaves on the human agent's desk is the harder work: the frustrated customer, the ambiguous situation, the case where the playbook does not quite fit. That is exactly the work where empathy, ownership, and judgment matter most. And it is exactly the work that a compliance-only QA rubric does not prepare agents for.

Build QA that develops those skills, not just one that tracks whether the agent clicked the right button. The customers who need a human are the ones who most deserve an agent at their best.

If you want to compare notes on your current QA setup, including what your rubric is actually measuring, drop me a message here.