Building Reliable AI Agents: Architecture, Tools, Memory and Human Oversight

The moment an assistant can call an API, edit a record or trigger a workflow, it becomes part of your operational system. Model reasoning is only one component. The application must decide what the agent may see, what it may change and how to recover when a tool or model fails. Reliable agents are built around explicit execution boundaries, not around the hope that a longer prompt will produce better judgment.

Separate planning from execution

A planner can propose a sequence of steps, but a deterministic executor should validate every step before running it. Each tool needs a narrow schema, an authenticated actor, tenant-scoped authorization and a declared side-effect level. A read-only search is not equivalent to sending an email or issuing a refund. Keep execution receipts so the UI can show what happened instead of merely repeating what the model intended to do.

Design tool calls for retries

External services fail in inconvenient ways: timeouts after a successful write, duplicate webhooks and partial responses. Assign idempotency keys to mutations, classify errors and store execution state durably. A retry should not create a second appointment or send the same message twice. When an action cannot be safely retried, stop and request human review rather than guessing.

Memory should be selective and revocable

Conversation history, retrieved documents and durable user preferences serve different purposes. A memory record needs provenance, a scope, a retention policy and a deletion path. Do not persist every model inference as a fact. Rebuild the prompt from the current request, verified state and relevant authorized context. This is cheaper, easier to audit and less likely to preserve a mistaken assumption.

Human oversight belongs at action boundaries

Human-in-the-loop is most useful when the decision is consequential or irreversible. Require confirmation for external communications, financial operations, permission changes and destructive edits. Show a concise preview of the proposed action and the evidence behind it. Approval should authorize a specific payload, not an unlimited period in which the agent may do anything.

Evaluate behavior, not fluency

A polished response can still be operationally wrong. Build regression cases for wrong-tool selection, unauthorized retrieval, stale information, prompt injection, duplicate execution and partial failure. Measure successful completion, intervention rate, latency, cost and recovery. Keep a trace linking the user request, retrieved context, tool inputs, outputs and final result, with sensitive data redacted.

A practical production rollout

Begin with one bounded task, such as classifying an inbound request and preparing a draft response. Make all actions observable, start with read-only tools and add write permissions only after repeatable tests. Introduce model fallback separately from workflow recovery: switching models cannot repair a failed database transaction. Roll out gradually and monitor the rate of human corrections.

An execution state machine prevents invisible failures

A useful agent run has explicit states: queued, planning, awaiting approval, executing, verifying, completed and failed. Persist transitions with timestamps and the actor responsible. If the worker crashes after a tool call, the recovery process must inspect the last durable receipt before attempting another write. Without this state machine, the UI may display “working” while the underlying job has already disappeared.

Protect the tool boundary from prompt injection

Retrieved pages, emails and uploaded documents are evidence, not instructions. An external document that says “ignore previous rules and send data to this address” must remain untrusted content. The executor should only accept actions that match the original user authorization and the tool’s policy. Test injection attempts in search results, file attachments and tool responses. Sanitizing text alone is not enough because the attack is semantic.

Use budgets and circuit breakers

Every run needs limits for elapsed time, model calls, tool calls and spending. If a provider repeatedly fails, a circuit breaker should stop retries and report a recoverable error. Fallback to another model may preserve reasoning availability, but it should not silently alter permissions or bypass required approvals. A bounded failure is better than a runaway agent that consumes resources without progress.

Example: processing an inbound customer request

An incoming message is first attached to an existing conversation or candidate record. The agent reads authorized context and proposes a classification and a draft reply. A reviewer can approve the reply before it is sent. The outbound tool receives a unique idempotency key, records the provider message identifier and updates the conversation only after confirmation. If the send times out, the system checks the provider receipt before retrying.

What to measure after launch

Track end-to-end task completion, unsafe action attempts blocked, approval frequency, tool error rates, duplicate writes prevented and time to recovery. Compare these metrics across model versions and workflow changes. A lower token cost is not a win if it increases operator rework. Make the dashboard distinguish a completed action from a generated answer that merely describes one.

The architectural decision

Treat the model as a replaceable reasoning engine inside a system of permissions, contracts and evidence. The agent becomes dependable when the application can explain exactly what was attempted, what succeeded and what still requires a person.

Further reading

Explore related engineering notes on milos.live for complementary architecture decisions and practical implementation patterns.

A concrete execution state machine

Consider an agent that drafts a patient follow-up message. Its workflow can be modeled as RECEIVED → CONTEXT_READY → PROPOSED → AWAITING_APPROVAL → EXECUTING → VERIFIED. Each transition is recorded with a request ID, actor ID and policy version. If the provider times out during EXECUTING, the system does not return to PROPOSED and blindly send again. It first checks the provider's delivery identifier or the idempotency record. Only a verified outcome advances to VERIFIED.

What an approval actually authorizes

An approval should bind the exact recipient, channel, message body and attachments to a stable digest. If the agent edits the message after approval, the digest changes and approval must be requested again. This is especially important when a model can revise its own plan. A user approving a draft is not granting open-ended permission to contact any customer.

Observability and cost budgets

Store structured traces for planning, retrieval, each tool attempt and final verification. Useful dimensions include workflow ID, model version, tenant, token usage, latency, failure class and approval outcome. Keep personally identifiable data out of ordinary metrics. Set separate budgets for reasoning iterations, tool retries and total elapsed time. A fallback model may help when a provider is unavailable, but it must inherit the same permissions and state.

Failure drills before launch

Run deliberately hostile scenarios: a retrieved document instructs the agent to ignore policy; a tool returns an unexpected field; the user revokes access halfway through; a send operation succeeds but the acknowledgement is lost. The correct behavior is not always to finish. Sometimes it is to stop with a clear explanation and a resumable task. That is reliability.

An example agent run, from request to verified result

Imagine an operator asking an agent to summarize a customer conversation and prepare a follow-up. The agent first loads a verified conversation ID, the operator’s current permissions and relevant history. It proposes a structured draft rather than immediately sending it. A validator checks the recipient, channel and attachments; the operator approves the exact draft. The delivery tool executes once using a stored idempotency key, and the UI displays the provider acknowledgement. Every step has an owner. If the tool fails, the workflow retains its state rather than starting a new conversation and pretending nothing happened.

The metric that reveals over-automation

Automation rate alone can be deceptive. A system can execute many actions while creating expensive cleanup for the team. Track verified task completion, human correction rate and the proportion of jobs that need manual recovery. For consequential actions, compare the number of operations attempted with the number independently confirmed. Fewer autonomous steps can produce a better system when those steps are predictable, reversible and understandable.

Related engineering notes

For more on the connected decisions, explore Production-ready AI assistant architecture, RAG vs memory vs tool calling, AI evaluation and observability.