Which step in your product actually needs AI?

"Let's add AI to support" does not describe a project yet. It might mean classifying a message, finding an order, or answering the customer. Each task needs different data and tolerates different mistakes.

Consider a support operation where someone reads a message, finds the purchase, and routes the case. You can test classification without giving the model permission to change the purchase. That scope helps establish whether AI improves the work before increasing its responsibility.

We prefer starting with the smallest step that has value on its own. If classification does not help anyone finish handling a case, a good demo does not justify the integration.

Choose a task with a checkable result

A first version could suggest a category and show the part of the message that supports it. The support worker accepts or corrects the suggestion. The application records both decisions.

Before building, collect a sample of messages you are authorized to use. Remove unnecessary data. Include routine requests, ambiguous messages, and cases that should go straight to a person. Hold back some examples to evaluate future changes without using them to tune the prompt.

Ask someone who knows the operation to classify the same cases. If two people disagree, the support policy may need work. Changing models will not fix a category the team cannot define.

A fixed workflow may be enough

Anthropic distinguishes workflows whose steps are defined in code from agents that choose their next steps. That distinction helps decide how much freedom the task needs.

For the support example, the sequence can be fixed. Receive the message, request classification, validate the output, and show the suggestion to the worker. Code chooses the order. The model interprets the text.

An agent becomes useful when later steps depend on intermediate discoveries. Even then, describe the actions it can take and when it must stop. "Resolve the support case" is too broad to become a permission.

Compare the cost of the whole case

Measuring only the percentage of correct categories misses part of the work. A misclassified case might pass through two teams before returning to the start. A correct suggestion can also arrive so late that nobody waits for it.

During a pilot, track time to correct routing, worker corrections, and cases without a useful response. Add the cost of model calls and the human time spent reviewing. Compare with the existing process using cases of similar difficulty.

Do not treat the model's stated confidence as evidence that it is right. Define observable triggers for human review, such as a missing category, insufficient information, or an out-of-scope request.

Set the condition for continuing before the pilot. For example, retain the suggestion only if it reduces triage time without increasing incorrect routing. Numerical limits depend on the operation and need a baseline measurement.

Permissions belong in code

A customer's message can contain instructions to ignore rules. The application must treat those words as received content. They cannot grant access to tools or another customer's data.

In a classification pilot, the model does not need credentials to edit orders. If a later version can act, each call must pass server authorization. Limit the resources it can access and record the action and its outcome.

Set a retry ceiling and a maximum duration as well. Once either limit is reached, hand over the case with the available context. A case stuck in automated attempts is still a case waiting for service.

Be ready to switch the suggestion off

Keep a way for the worker to do the job when the model is unavailable. Record the prompt and model version used in each evaluation without retaining unnecessary personal data. Before a change, rerun the held-back evaluation cases.

The first useful result might be modest. A suggestion that saves someone from searching for the right queue is worth measuring. Expand autonomy only when the records show which next step is worth automating.