Support triage and tagging: LLM ticket classification with a human in the loop

A support inbox that takes 60 new tickets a day turns every agent into a part-time dispatcher, and the tickets that suffer are the urgent ones buried between password resets. This is for support and operations leads who want classification and first drafts handled by a workflow, while the decisions that can annoy a customer stay with a person.

What support triage automation does

Triage is the work that happens before the work: reading a ticket, deciding what it is about, how urgent it is, which team should handle it, and whether the customer is already angry. In most teams it is done by whoever picks the ticket up, which means it is done sixty times a day, inconsistently, by people who would rather be solving the problem.

The automation reads each incoming ticket or email, returns a structured classification, applies tags and priority in the help desk, routes to the right queue and, where it makes sense, attaches a draft reply for the agent to edit. It does not close tickets or send anything on its own until the accuracy numbers say it can, and then only for a short list of intents.

The four labels and who consumes them

The model returns all four as a JSON object with a confidence per field and a one-sentence reason. The help desk gets the tags; the agent sees the reason. That reason is what makes the classification checkable in three seconds instead of forcing the agent to re-read the ticket, and it is the same pattern we use for lead routing when a sales inquiry lands in the wrong inbox.

Tag taxonomies that stay stable

Most triage automations decay not because the model gets worse but because the tags drift. Someone adds "billing-urgent" next to "billing" and "urgent", a campaign tag from last spring is still live, and by month six the model is choosing among 70 labels that overlap. Rules that prevent this:

• One dimension per label. Urgency is a field, not a suffix on the intent.
• At most twelve intents. If a new one is needed, an old one is merged or retired.
• Every label has a one-line definition and five example tickets, kept next to the prompt.
• An explicit "other" bucket, reviewed weekly. When it exceeds 10% of volume, the taxonomy is missing something.
• Changes are versioned. Classifications made under taxonomy v3 are not compared with v4 accuracy numbers.
• Temporary campaigns use a separate field, never the intent list.

Drafting replies with a human in the loop

For intents with a known answer — how-to questions, invoice copies, plan comparisons — the workflow drafts a reply from the ticket, the classification and a retrieval step over your help center and internal notes. The agent sees the draft, the sources it used and a send button. Editing takes 20 seconds; writing from scratch took four minutes.

What stays with a person: anything tagged cancellation, refund, legal, security or outage; sentiment beyond "frustrated"; confidence below threshold; and anything from an enterprise account. These are not the cases the model gets wrong more often — they are the ones where a wrong reply costs more than the four minutes saved. Auto-sending without review is a separate, later decision, taken per intent, only after a month of agents approving drafts unedited more than 90% of the time. That gate is a guardrail, and it keeps the automation from becoming a customer-facing incident.

A worked example: a scheduling software company

Consider a B2B scheduling product with about 1,200 tickets a month, five support agents across two time zones, a help desk tool and a legacy support address that still gets 200 emails a month. Every ticket was read by the on-shift agent and tagged by hand, when it was tagged at all. Priority was set by gut. Two agents handled billing because they knew the invoicing system, and the rest forwarded to them.

The workflow: help desk and mailbox feed one n8n flow. The model classifies with a nine-intent taxonomy the team wrote in a two-hour session. Billing intents route to the two specialists; bug reports get a draft that asks the customer for the three details engineering always needs; how-to tickets get a draft grounded in the help center. Cancellations, refunds and anything tagged "at risk" go to the team lead untouched. A weekly job samples 30 tickets, shows the agent's final tags next to the model's, and posts the disagreement rate.

After six weeks: triage accuracy at 91% on intent and 84% on urgency, measured against agent corrections. First response time on high-urgency tickets dropped from a median of five hours to 40 minutes, mostly because urgent tickets were no longer invisible. The "other" bucket sat at 6%, and two new intents came out of reviewing it.

How to measure triage accuracy

1. Label a baseline set. Two hundred historical tickets, tagged by two agents independently. Where they disagree, the definition is unclear; fix the definition before blaming the model.
2. Measure per label, not overall. A 90% average can hide a 60% score on "cancellation", which is the one label that matters. Track precision and recall per intent.
3. Watch the confusion pairs. Bug report versus how-to, billing versus cancellation. They tell you which two definitions need a sharper boundary, or a merge.
4. Treat agent overrides as ground truth. Every time an agent changes a tag, log the before and after. That is free labeled data and an early warning of drift.
5. Re-run on every change. New prompt, new model version, new taxonomy version: the baseline set runs again in the evaluation harness before it touches live tickets.

Escalation rules that do not depend on the model

Some escalations are too important to leave to classification. Keep them as plain rules running alongside the model: a ticket mentioning an outage, a data breach, a regulator or a lawyer escalates on keyword match; an enterprise account with negative sentiment goes to the account manager; any ticket past its SLA escalates regardless of tags. Rules are boring, and boring is what you want when a miss costs a churned account. In Hilluter's automation work the rules layer is written first and the model is added on top, not the other way around.

If you can share a month of anonymized tickets and your current tag list, send them over and we will come back with a proposed taxonomy and a realistic accuracy estimate before any build starts.