Sorting customer emails: when rules are enough and when AI is needed

Sorting customer emails with AI is justified when messages need to be classified by the meaning of free text and testing shows acceptable results compared with simple rules. If a category can be reliably determined by the recipient address or a trusted structured field, start with rules. Both approaches need a queue for ambiguous messages and someone responsible for handling it.

A shared support address receives billing questions, fault descriptions and new orders. An employee reads each message, chooses a department and forwards it. This seems straightforward until one message contains both a delivery complaint and a request to correct an invoice. The automation task first needs a clear answer: who owns that conversation?

It is worth starting the choice of method with a small, controlled comparison. For the same examples, obtain human-assigned categories, rule-based results and AI results. Then examine where errors create work or risk. The overall proportion of correct answers alone can conceal the most important misrouted messages.

Define the sorting task before choosing the technology

Classification needs a finite list of categories with clear boundaries. “Sales”, “billing” and “technical support” are a start, but names alone are insufficient. For each category, record what belongs there, what does not and who should receive the message. Otherwise, employees will label the same email differently too.

Separate subject from priority. A billing question may be urgent, but the word “urgent” must not itself authorise the sender to bypass company procedures. Likewise, a customer's name in an email is not reliable identity verification. Routing logic must use only data that can be trusted for the decision concerned.

Anthropic's technical explanation describes routing as classifying an input and directing it to the appropriate downstream task. A language model or a traditional algorithm can do this. Description of routing. This support task does not automatically require an autonomous agent authorised to write to customers or change orders.

The initial boundary can be simple: the system recommends a category and the responsible queue but does not issue refunds, change customer data or send replies. The safety of additional actions needs separate assessment. This boundary allows sorting quality to be tested without exposing other processes to new risks.

When rules are enough

Rules are useful when input is stable and a verifiable attribute determines the category. For example, a message may originate from a specific company form with an allowed request type. Check the origin of that attribute too: a freely editable email subject is not as trustworthy as a form field validated on the server.

Keywords can help, but they have limits. In “this is not an invoice question”, the word “invoice” appears to exclude that category. In another message, it appears in a quotation from an earlier conversation. Before introducing a rule, test these negative examples too, not just messages where the keyword matches the intended subject.

Rules have the advantage of an understandable rationale. You can show which condition was triggered and change that condition specifically. However, a large list of exceptions also needs maintenance. If every new phrase requires a separate adjustment, assess the maintenance burden and the proportion of emails still left for people to handle.

Where AI classification may help

AI may be a candidate when people express the same intention in different ways. Its suitability cannot, however, be established from a smooth demonstration using carefully selected messages. Test the company's actual variety of correspondence while respecting data-protection requirements.

Assess Latvian, English and Russian messages separately. Include mixed languages, typos, short questions and long conversations. A good EN result does not prove equally good LV classification. If a group is barely represented in the sample, state that clearly in the results.

The model must return a constrained result from the allowed list. If it invents a category or returns a free-form answer, the trusted part of the system must reject it or route it for review. The model's explanation may help an editor, but it is not proof of correctness and must not grant new execution permissions.

In practice, it is also worth comparing a combined workflow: rules handle clear cases, while a model or person assesses the rest. There is no need to complicate the entire incoming stream if the problem concerns only part of it.

How to build a manually labelled test sample

Select messages using a predefined period and set of channels. Keep difficult cases too: incomplete text, irrelevant messages and misdirected requests. Removing them from the pilot means the test will not reflect the work the team must handle after implementation.

Remove unnecessary personal and customer data before use. At the same time, check that anonymisation has not removed the meaning needed for classification. For example, the recipient's function can be retained as a role label without keeping a particular person's address. The contents of sensitive attachments are often unnecessary for this test.

Responsible team members label the expected categories and record ambiguous examples. Discuss disputed messages and refine the category definitions. If agreement is impossible, that signals an unclear company process, not automatically a model defect. An abstention class may be appropriate for such a message.

Use some examples to develop rules and instructions, but keep the test portion separate. Do not place the same conversation thread in both portions so that the system has effectively already seen the answers. After changing conditions, record the version and which data were used for tuning.

A confusion matrix shows where risk arises

The following is an illustrative matrix, not a measured company result. Rows represent human-assigned categories and columns represent classifier responses. The figures are invented to explain the calculation.

The direction of misrouting matters here. If a technical problem reaches accounting, establish how quickly it is noticed and forwarded. The human-review column is not automatically an error: it may be an intended safety choice. At the same time, assess whether almost all the work is accumulating in that queue.

Use the same labelled sample and category definitions for both methods. Compare correct routing, types of error, the volume of human handover and processing time. Show results by language separately. If a method has not yet been tested in one group, do not present that group as successful.

Choose acceptance limits according to the consequences of an error. A general question that can safely be forwarded and a message about a possible data breach cannot have the same risk weighting. Before testing, define which cases a person must see in all circumstances and how quickly.

An ambiguous message must not disappear between departments

Abstention must trigger a specific action: the message enters a designated queue with an owner and a handover procedure. An “other” status without an owner merely moves the problem elsewhere. The team also needs to see the waiting time and whether a person has already made a decision.

For a multi-topic email, decide who retains overall responsibility. Linked subtasks can be created, but the customer should not receive contradictory replies from different departments. Splitting is itself an action with its own error risks; a classifier recommendation should not automatically become new tasks without a check.

If you use a confidence score, test it against actual errors. Confident wording from the model is not a reliable quality measure. A threshold cannot be chosen simply because it looks high. What matters is the observed result for the relevant group of messages.

Instructions in an email do not authorise the system

Incoming email is untrusted content. It may instruct the system to ignore classification rules, assign special priority or send data elsewhere. OWASP describes the risk of such instructions influencing a system as prompt injection. Asking the model to ignore them is not sufficient. OWASP risk description.

Classification output must be constrained, while recipient selection and access decisions remain in trusted local logic. The model must not be allowed to construct an arbitrary address or command from email text. Block disallowed output before any forwarding. Attachments and quoted conversations must also be treated as input data, not system instructions.

Include harmless attack examples in testing and confirm that they do not expand permissions. This does not prove protection against every future attempt, but it helps test the specific execution boundary. Retain human handover and do not allow automation to silently delete the original message.

Start the pilot with measurement, not fully automatic forwarding

Initially, the system can run in observation mode, recommending a route while the team continues the real work. Record differences and the causes of errors. Only after an acceptance decision should specified low-risk actions be enabled. Keep the option to return to manual processing without losing the queue's contents.

CSP data on business use of AI helps explain the adoption context but does not prove the usefulness of email sorting in a particular company. The survey has a defined business population and does not test the return from this process. CSP business ICT (IKT) data. The decision needs your own labelled sample and observed cost of errors.

Will AI necessarily outperform rules?

No. For structured input, rules may be sufficient and easier to maintain. Test AI where meaning is difficult to capture through a stable attribute. Keep the simplest option that meets the company's acceptance requirements.

Can we start without sending customer data to a model?

Category definitions and the rule base can be organised first. The volume of model-test data, processing environment and permissions must be assessed before handing over real correspondence. Anonymised examples help, but the quality of anonymisation also needs checking.

To assess a business process automation pilot, prepare a category list and appropriately protected email examples. For broader process selection, see business automation examples; for limitations of model responses, see incorrect AI answers and hallucination risk.