AI deployment and monitoring: staging, canary, and the metrics that matter

An AI workflow that passed every test can still go wrong in month three: the provider updates the model, a supplier changes its invoice layout, a worker stalls and the queue quietly grows. This is for teams about to put an LLM feature in front of real traffic — and for the ones who already did and hear about problems from users first.

Why AI workflows need their own deployment path

Ordinary software fails loudly. An LLM feature fails quietly — the workflow keeps running, the output is well-formed, and it is wrong in a way no health check sees. Add a dependency you do not control, a cost that scales with every request, and behavior that changes when the provider changes something, and uptime monitoring is not enough. The rollout has to be staged, and the monitoring has to measure outcomes, not just availability.

Staging, canary, full rollout

1. Staging

Same prompts, same pinned model, integrations pointed at sandboxes or read-only copies. Exit criteria: the evaluation harness is green on the exact version being deployed, and the red-team corpus has been run against the real pipeline with logging on.

2. Canary

A slice of real traffic: 5–10% of items, or one team, one mailbox, one document type. Where the workflow takes actions, the canary runs in shadow mode: it produces and logs the proposal, the human decision still stands, and the two are compared. Exit criteria are written before the canary starts: agreement with humans above a threshold, escalation rate in an expected band, no tool-call errors, cost per item under budget, over a fixed period such as two weeks. A canary without written exit criteria ends when someone gets impatient.

3. Full rollout

Switched on by a configuration flag, not by a deploy, so it can be switched off the same way. The previous path — manual process, rule-based classifier, template — stays available as a fallback for at least a month. Autopilot for low-risk actions is enabled here, per action type, with the guardrails deciding what still needs a person.

Environment, secrets and model pinning

Secrets live in the platform's secret store — n8n credentials, environment variables, a vault — never in prompts or in workflow exports that end up in a repository. Each environment has its own API keys, which gives cost attribution for free. The model is pinned by its exact version identifier, not by an alias the provider may quietly move to a new release. A model upgrade is a change like any other: it runs the harness, goes through a canary, and is logged alongside the prompt version on every call.

Two budgets are set before rollout and enforced after it: a cost budget per outcome — €0.05 per triaged ticket, €0.30 per extracted contract — and a latency budget, usually a p95 target per call type. The budgets drive design choices: caching, batching, a smaller model for the easy cases. When a budget is exceeded, that is an alert, not a surprise on the invoice.

Monitoring that matters

Five signals cover most of what goes wrong. Each has an owner and a runbook, and each alerts on both a threshold and a trend.

The sampled human review is the one teams try to skip. It is also the only signal that measures whether the output is right, not just whether the system is running; thirty documents a week takes about an hour. Alerts go to a named owner, not a channel nobody reads; an alert without a runbook step is noise.

Incident replay from logs

Every run is logged as one record: input reference, prompt version, model id, full output, every tool call and its result, the decision taken, latency and token counts. These live in Postgres with an agreed retention period. Replay means running the logged input through the current version and the previous one, then diffing. Most incidents are explained within the hour, and every one becomes a new harness case. Logging that stops at "request received, response sent" cannot do this.

A worked example: a contract and invoice extractor going live

A property management company processes around 900 tenant contracts and supplier invoices a month across eight buildings, each with its own mailbox. The extractor writes structured records to Postgres. Staging: harness at 98.1% per-field accuracy on 350 documents, red-team corpus clean. Canary: two of the eight mailboxes, shadow mode, two weeks. The comparison of 412 documents found one field under threshold — deposit amounts on an older contract template the sample had missed. Forty of those went into the harness, the prompt was adjusted, and the canary ran one more week.

Full rollout by flag. In month two the escalation rate rose from 6% to 14% inside one week. The logs showed a new supplier whose invoices put the total above the line items instead of below. Thirty cases added, prompt updated, harness green, escalation back to 7% within days. Cost ran at €0.03 per document against a €0.05 budget. The queue-age alert fired once, after an API key rotation missed one environment; the runbook step took ten minutes.

Operate means iterate and expand

In Hilluter's process the Operate phase is not a support contract with a phone number. It is a monthly review of harness score, sampled accuracy, escalation rate and cost per outcome, followed by a decision about what to expand next: another document type, another mailbox, the next action moved from suggestion to autopilot. Expansion is earned by stable numbers, not by the roadmap; that is how AI integration work stays in production.

Checklist before switching on real traffic

• Harness green and red-team corpus run on the exact version being deployed.
• Canary scope, written exit criteria and a fixed duration agreed; shadow mode for any step that takes an action.
• Secrets in a secret store; separate keys per environment.
• Model pinned by exact version id; prompt version logged per call.
• Cost per outcome and p95 latency budgets set, with alerts.
• The five signals instrumented, each with an owner and a runbook; weekly review sample assigned to a named person.
• Per-run logs sufficient to replay any incident.
• A configuration flag and a tested fallback path to switch the feature off.

If you have a workflow in staging and no rollout plan yet, tell us what it does and who depends on it and we will send back the canary criteria and the metrics we would watch first.