The boring parts of an AI agent that actually matter
Tool-calling and prompts are the fun part. Evals, observability, and rate-limiting are the parts that decide whether the agent makes it past month two.
The demo lies to you
An agent demo is a single happy-path run: input goes in, a clean chain of tool calls happens, the right answer comes out, everyone claps. Production is a thousand runs a day, a third of which hit a rate limit, a flaky API, a malformed tool response, or an input nobody anticipated. The gap between those two worlds is almost entirely boring engineering, and it’s the part that decides whether the agent survives its first real week.
Retries, done specifically
“Just retry on failure” is not a retry policy, it’s a way to double-charge a customer or send a duplicate email. Every retry needs to answer three questions: what counts as retryable (a timeout, yes, a successful call with a bad answer, usually no), how many attempts before giving up, and crucially, is the underlying action safe to repeat. A model call that only reads data is trivially safe to retry. A tool call that sends a message or charges a card needs an idempotency key so a retry is provably a no-op if the first attempt actually succeeded.
Idempotency is the quiet foundation
Every agent step that has a side effect gets a deterministic key, usually a hash of the run ID and the step name, stored before the side effect happens. If the worker crashes mid-step and retries from the top, it checks that key first. Already done, skip. Not done, proceed. This one pattern prevents the entire category of “the agent emailed the customer twice” bugs, and it has to be designed in from the first step, not patched on after the first incident.
Logging that answers “what did it actually do”
When something goes wrong, the question is never “did the agent work.” It’s “what exact sequence of tool calls, with what exact inputs, led to this exact output, at this exact time.” That needs structured, queryable logs of every step: which tool, what arguments, what it returned, how long it took, what it cost. Not a text blob in a log file nobody greps. A row in a table someone can filter by run ID, five minutes after a customer complains.
Checkpointing and resumability
Long-running agent workflows (multi-step research, a multi-stage document pipeline) need to persist state after every step, not just at the end. If step 7 of 12 fails, a well-built agent resumes from step 7, not step 1. This sounds obvious until you’re the team that burned three hours of model spend rerunning a pipeline from scratch because step 11 of 12 timed out and there was nowhere to resume from.
Cost tracking, per run, not per month
A surprise bill at the end of the month tells you nothing actionable. Per-run cost tracking, logged alongside the run itself, tells you immediately that one specific workflow is burning ten times the tokens it should, often because of an unbounded retry loop or a prompt that’s silently grown too long. We log cost as a first-class field on every agent run, right next to status and duration, because it’s a reliability signal as much as a budget one.
The human handoff, designed, not improvised
Every agent needs a clean exit to a human for the cases it shouldn’t handle alone: low confidence, a refusal case, something genuinely novel. That handoff needs to carry full context (what the agent already tried, why it stopped) so the human isn’t starting from zero. The handoff is a feature, not an admission of failure, and skipping it is the single most common reason a genuinely capable agent gets a reputation for being untrustworthy.
Why this is the actual work
None of this shows up in a demo. All of it shows up in the first month of real traffic. Teams that skip it ship something that looks finished and spends its first production month failing in slow motion: duplicate side effects, unexplainable outputs, surprise costs, and no clean way for a human to step in. The model call is maybe 10% of what makes an agent trustworthy. The other 90% is this list.