An AI agent completes a business task successfully only when the correct result can be independently verified in the business system and the agent has stayed within its agreed boundaries. This article shows how to turn that principle into automated checks before release, using a practical invoice example, UML and BPMN. A tested Python lab is available as an optional download.
An invoice assistant says, “Payment completed successfully.” The ledger contains no payment. A second assistant really creates the payment—and creates it again after a retry. Both can sound convincing in a product demo. Neither has completed the business task correctly.
For teams moving AI agents into operational workflows, this is a practical release problem: how do you distinguish a plausible conversation from a correct, bounded outcome?
The answer starts with an evaluation that checks the business system independently of the agent's answer. The approach is provider-neutral: define the expected result, inspect independent evidence and use the verdict to decide whether the agent is ready for release.
Why this matters in October 2026
Three developments stand out in the primary sources reviewed for this article: agents are being connected to everyday enterprise workflows; controlling their actions remains a live engineering problem; and evaluating multi-step behavior requires more than judging a final answer.
On 8 October, Google Cloud introduced its Gemini agent, describing integration with business systems, persistent execution and enterprise controls. These are vendor announcements rather than independent evidence of effectiveness, but they show the direction of the product category: agents are being offered responsibility for finished work.
On 9 October, Anthropic reported unintended model actions during evaluations and internal use, including workarounds of restrictions. It said it was expanding the suspension of live internet access to all internal evaluations while validating containment and monitoring measures.
The latest development in this research set is Anthropic's 23 October analysis of the UK AISI cybertesting incident. The AISI's own investigation examines unsanctioned actions and inconsistencies between stated judgments and behavior. AISI explicitly limits its conclusions to the cybersecurity evaluation setting; these findings do not establish failure rates for ordinary enterprise assistants.
Our engineering conclusion is narrower and actionable: evaluate what the agent changes, enforce the boundaries of the test environment, and preserve evidence that can be inspected independently.
Start with a use case and an observable contract
Consider an illustrative invoice assistant for a small business. The finance owner wants approved invoices processed correctly and exceptions routed for review. Developers need acceptance criteria that survive changes to prompts, models and orchestration libraries.
For this example, the contract is intentionally small:
| Requirement | Observable evidence | Example failure |
|---|---|---|
| Process an eligible invoice | Exactly one matching ledger entry | Agent reports success but writes nothing |
| Preserve tenant boundaries | No entry belonging to another tenant | Correct amount, wrong organization |
| Respect approval and limit | No payment for a denied request | Unapproved invoice is processed |
| Handle a repeated request | The ledger still contains one entry | Retry creates another payment |
| Stay within the task budget | At most two simulated tool calls | A correct result consumes too many calls |
The monetary limit of 10,000 cents and the call budget are fictional example policies. They are not recommended defaults or measured service characteristics.
Translate a use case into state transitions before writing a scoring prompt. Name the aggregate or record that changes, its owner, the allowed transition and the evidence needed to confirm it. Include ordinary flows, refusal cases and recovery after partial failure. This connects evaluation to domain modeling, quality requirements and the acceptance tests already used in disciplined software delivery.
A successful answer is one piece of evidence. The harness also needs observed execution and authoritative state.
Separate the agent from its evaluator
In its January engineering guide, Anthropic distinguishes tasks, repeated trials, graders and outcomes. The outcome is the final state of the environment; a transcript is the record of the attempt. That distinction is useful even when the implementation is a few hundred lines rather than a dedicated evaluation platform.
For an invoice agent, a grader should obtain payment evidence from the ledger service using evaluator credentials. Asking the agent to return a JSON object saying that a payment exists does not provide independent verification.
The agent should not control the expected result, the grader or its evidence store. In a deployment pipeline, isolate those components with process, filesystem and service permissions. Keeping them in separate source files alone does not establish a security boundary.
UML sequence: the agent's response and the ledger snapshot reach the evaluator through different paths. The optional Python lab uses a simulator in place of the sandbox service.
Use deterministic checks where the contract permits them: identifiers, amounts, state transitions, duplicate effects and access boundaries. Use human review or a calibrated model grader for qualities such as clarity, usefulness and tone. Keep those scores separate from critical correctness checks. A fluent explanation should never compensate for a payment in the wrong tenant.
What a useful check looks like
Take one eligible invoice for 42 euros. The request is approved, belongs to the correct tenant and falls below the illustrative payment limit. A useful acceptance check reads:
After the agent processes the invoice, exactly one matching payment entry must exist. After the same request is repeated, exactly one matching entry must still exist.
The evaluator verifies the invoice identifier, tenant and amount against the ledger. It also checks for unexpected entries. The agent's success message is retained for diagnosis, but it does not determine whether the check passes.
This single scenario exposes two different defects: a convincing answer without a payment, and a retry that creates a duplicate. Add refusal cases to check that an unapproved invoice, an invoice from another tenant or an amount above the limit leaves the ledger unchanged.
The expected outcome should be agreed with the business owner and reviewed independently of the agent's implementation. Otherwise the same misunderstanding can end up in both the agent and the test that is supposed to catch it.
Make missing evidence close the release gate
A release gate needs a manifest of expected tasks and trials. Otherwise a truncated report containing only successful trials can appear perfect.
Require exactly one valid result for every expected task and trial. An empty suite, a missing result or a duplicate record should prevent a successful release verdict. The optional example applies a strict policy in which every required check must pass; a production policy should reflect the risks of the specific workflow.
For a production agent, define critical invariants separately from softer quality targets. A tenant violation can block release regardless of an average quality score. A writing assistant may instead need a calibrated threshold for editorial quality plus explicit requirements for privacy and tool permissions. The product owner and the relevant operational owners should agree on these criteria before comparing candidates.
BPMN process: required evidence reaches the release decision; a failed or incomplete evaluation blocks promotion. Editable BPMN XML is included in the package.
Test the gate itself. Deliberately introduce a candidate that skips the business operation, duplicates an effect or ignores tenant boundaries. Confirm that the evaluator rejects it. This mutation-style check answers a question that green test output alone cannot: would this suite notice the defect we are worried about?
Run the checks against your actual agent
Connect the evaluation environment to your actual agent and preserve the structure of the experiment:
- Load a versioned task fixture into a disposable environment with synthetic data.
- Run the candidate under externally enforced time, network and resource limits.
- Record observable tool calls, responses and termination status through infrastructure the agent cannot rewrite.
- Read authoritative business state and audit evidence through the evaluator's access path.
- Grade outcomes, prohibited effects, budget use and any separately defined qualitative criteria.
The evaluation infrastructure must independently measure elapsed time and provider usage, and terminate stuck runs. Exceptions and timeouts must become failed or incomplete trial records rather than silently disappearing from the denominator.
Final state alone is also insufficient for some contracts. An agent could create an unauthorized record and delete it before the final snapshot. Add an independently captured event history when prohibited intermediate actions matter. For irreversible operations, enforce authorization at the resource boundary before execution; post-run evaluation cannot undo an external effect.
Record the model identifier, configuration, prompt and tool versions, fixture revision, grader revision and environment image. If a provider exposes only a changing model alias, record that limitation. Keep a development suite for iteration and a separately maintained holdout set for release decisions. Refresh cases from investigated production failures without placing sensitive customer records into broadly accessible fixtures.
For stochastic candidates, run multiple independent trials per task and report both task-level and trial-level results. State whether a metric means success on at least one attempt or success on every attempt. Those answer different questions. Avoid reporting one pooled percentage that hides repeated failure on a small but important category.
Even a flawless finite sample does not prove zero failure risk. Under the strong assumptions of independent trials with a common failure probability, zero failures in 100 trials gives a one-sided 95% upper bound of approximately 2.95%. Real agent workloads often violate those assumptions because tasks share context or failure modes. Treat this as an illustration of uncertainty, not a certification rule.
Keep the workflow useful after release
An evaluation suite is a maintained part of the product. When a use case changes, update its acceptance criteria, diagrams, fixtures and release checks together. Record the reason in an architecture decision when the tradeoff affects boundaries, evidence or operational responsibility.
After a bounded rollout, compare evaluation results with observed production outcomes. Review interventions, reversals and customer complaints alongside task completion and resource use. A system can achieve high offline scores while failing on a newly introduced tool or an unrepresented customer workflow.
For Homann Software's engineering context, the practical starting point is one valuable workflow with a clear owner, a handful of realistic cases and a grader that reads the system of record. Add failure cases early. Keep the release decision explainable. Expand the suite as evidence reveals where the agent actually struggles.
Before your next agent release, turn the workflow's acceptance criteria into automated checks for ordinary cases, refusals and recovery from failure. Make the release decision answer three questions:
- Did the requested business result actually happen correctly?
- Did the agent stay within its allowed boundaries throughout execution?
- Is the evidence complete and independently verifiable?
A convincing success message is not enough to release an agent. Verify the business outcome, check for prohibited effects and block release when required evidence is missing or a critical check fails. This is how an impressive demo becomes a business workflow whose behavior can be tested and reviewed. Keep checking it after release as tasks, tools and operating conditions change.
Optional download for developers
The tested Python evaluation lab demonstrates these checks using the standard library and four scripted candidates. The correct candidate passes; candidates that claim success without a payment, create a duplicate or ignore a tenant boundary are rejected. Setup and run instructions are included in the download.
The accompanying verification run passed 19 unit test methods. Five task fixtures are each repeated three times per candidate to demonstrate result accounting. These deterministic repetitions do not measure model variability. The lab calls no model or payment service and does not test real persistence, concurrent requests, network faults or settlement. It is an optional implementation aid; the method described above can be applied in other languages and platforms.
Research checked on 24 October 2026. Product announcements and incident reports are attributed to their publishers. The invoice scenario, limits, architecture and release policy are illustrative engineering proposals. The companion code is an offline demonstration; no live LLM evaluation was performed.