Adding a model API call to an existing application can take an afternoon. Making the resulting feature dependable enough for daily work takes considerably more, because language models are probabilistic: the same input can produce different outputs, and some of those outputs will be wrong. Reliability comes from the engineering around the model, not from the model alone.
Start from a narrow, measurable job
“Add AI to our CRM” is not a feature. “Suggest a category and priority for each new support ticket, which a team member can accept or change” is. A narrow job lets you define what a good output looks like, measure it, and design what happens when the output is wrong. Anthropic’s documentation recommends defining specific, measurable success criteria before building.
A hypothetical example
A property-management company’s maintenance platform receives free-text repair requests from tenants. Staff read each request, choose a trade such as plumbing or electrical, set an urgency level and assign a contractor.
The AI feature reads each request and proposes a trade and urgency, with a one-line reason. Staff see the suggestion pre-filled and confirm or correct it. Emergencies such as a reported gas smell still trigger the existing rule-based alert, whatever the model says.
Nothing in the existing workflow is removed. The feature saves typing and speeds up triage, and every correction becomes data for evaluating the next version.
Engineering considerations
- Build an evaluation set before launch. Collect a representative set of past cases with the correct answers, including hard and ambiguous ones. Run it whenever the prompt, model or data changes. Both OpenAI and Anthropic document evaluation workflows for exactly this reason.
- Constrain the output. Ask for structured output with fixed fields and allowed values rather than free text, and validate it in code. Provider features such as structured outputs help, but your application should still reject anything outside the expected schema.
- Design the fallback. Decide what happens on a timeout, a provider outage, invalid output or an unclear case. Usually the feature quietly steps aside and the existing manual path works exactly as before.
- Keep people on consequential decisions. Suggestions that a person confirms are far safer than actions taken automatically. Move towards automation only where evaluation and production data show it is reliable enough, and keep the ability to switch it off.
- Handle data deliberately. Decide what is sent to which provider under which terms, and whether personal or confidential information should be removed first. Check the provider’s data retention and processing terms.
- Monitor in production. Track acceptance and correction rates, latency, errors and cost per request. A fall in acceptance is often the earliest sign that inputs or behaviour have drifted.
- Version prompts and models. Treat prompts as code: version them, review changes and pin model versions where the provider allows, so behaviour doesn’t change unnoticed.
For the wider risk-management picture, including governance, measurement and incident response, NIST’s Generative AI Profile of its AI Risk Management Framework is a useful reference.
Limitations and trade-offs
Even a well-evaluated feature will make mistakes. Evaluation tells you how often it fails on cases like the ones you tested, not that it won’t fail. Costs scale with usage, so estimate the per-request cost at expected volume early. Latency matters on interactive screens: a suggestion that takes several seconds to appear may simply be ignored. Providers also change and retire models, so keep the integration behind an internal interface that lets you switch and re-run your evaluations.
Sometimes evaluation shows that a simpler approach, such as a rules engine, better search or a redesigned form, performs as well. That is a useful result, not a failure.
Where to start
Choose one task with clear inputs and outputs, an existing manual process to fall back on, and people willing to review suggestions. Build the evaluation set first, then the feature.
If you have a specific workflow in mind for an AI feature, we’re glad to discuss how it could be scoped and evaluated.
