When people hear "72% of support tickets auto-resolved by AI without human intervention," they usually ask one of two things: how do I get that number, or is that number real. The answer to both is the same: focus on the 28%, not the 72%.
Here's what I mean. Getting from zero to ~50% auto-resolution — password resets, status checks, standard troubleshooting steps, billing lookups — those resolve themselves almost by accident once you have decent intent classification and a connected knowledge base. They're high-volume, low-variance, and the LLM handles them confidently on the first try. Building that part took maybe three weeks.
The next six months were spent pushing from 50% to 72%, and understanding why the remaining 28% resists automation.
The three failure modes
After logging and analyzing every ticket that fell through to a human agent, the failures clustered into three categories:
Ambiguous intent. The user says something like "it's not working." No entity. No context. Low confidence classification across four possible intents. The naive approach is to ask a clarifying question — but ask the wrong one and you've just added a round-trip that frustrates a user who already wanted to talk to a human. The right approach: when confidence is below threshold, don't guess. Route immediately, but attach the classification attempt and the extracted entities so the human agent doesn't start from zero.
Emotional escalation. A user who's been dealing with the same problem for three days doesn't want to be auto-resolved. They want to feel heard. LLMs can detect sentiment reliably, but the product decision — "what do we do when we detect frustration?" — is not a technical decision. We route to human within two turns for any session where compound frustration signals are detected. The AI does the triage; the human does the relationship work.
Multi-step state. Some tickets reference previous interactions: "same issue as last week," "the fix you suggested didn't work." Single-turn classification misses these entirely. We added session context hydration — on every incoming ticket, the last three related sessions (matched by email + entity overlap) are loaded into the LLM context window. Resolution rate for this class went from 40% to 71% after that change.
The boundary is the product
The lesson I keep coming back to: the quality of an AI support system isn't measured by how many tickets it resolves — it's measured by how well it handles the ones it shouldn't resolve. A graceful, context-rich handoff to a human agent is a better product outcome than a failed auto-resolution that leaves the user confused about what just happened.
We now track two metrics side-by-side: auto-resolution rate (72%) and handoff quality score (based on agent feedback on whether the context provided was useful). The second metric is harder to improve and more meaningful than the first.
If you're building something similar, design the exit path first. The first 50% will mostly take care of itself.