How to evaluate an AI agent development company

Why this is harder to evaluate than normal software work

Most software vendors can show you a portfolio and a past client will vouch for them, and that’s most of the diligence done. AI agent work is different, because a demo and a production system can look identical and behave completely differently the moment real, messy, adversarial input hits them. A good agency can make a broken approach look impressive for fifteen minutes. The questions below are designed to get past the demo.

Ask to see their eval process, not their demo

Any team that builds agents seriously will have a concrete answer to “how do you know when a prompt change made things better or worse.” If the answer is “we test it manually” or “we just try it a few times,” that’s a team that will ship you something that degrades silently over time. Look for a specific answer: a golden set of test cases, a pass-rate number, a CI step that runs before deploy. (See our own take on the minimum eval bar we hold ourselves to.)

Ask what happens when a step fails halfway through

This is the single best question to separate a team that has shipped real agents from a team that has only built demos. A production agent workflow fails constantly: a timeout, a rate limit, a malformed response. Ask specifically: does a retry risk duplicating a side effect (sending an email twice, charging a card twice)? Is there an idempotency key? Can a long-running workflow resume from where it failed, or does it start over? A team that hasn’t thought about this hasn’t shipped anything that survived contact with real traffic.

Ask how they track cost, specifically

“We’ll keep an eye on your OpenAI bill” is not a cost strategy. Ask whether cost is tracked per run, visible per workflow, and alertable when it spikes. Teams that only check cost at the end of the month find out about a runaway retry loop or an unbounded prompt thirty days too late, after the invoice.

Ask who owns the code and the model layer

You should own your repository outright, with documentation and a runbook, not a wrapper around a vendor’s proprietary platform you can’t leave. Ask directly: if we ended this engagement tomorrow, could another team pick up the codebase without needing you? If the honest answer is no, that’s vendor lock-in with extra steps.

Ask what they won’t build as an agent

This is a quieter signal but a reliable one. A team that pitches “we’ll build you an AI agent for everything” is optimizing for the sale, not for what’s actually right for your product. A team worth hiring will tell you, unprompted, when a single button or a plain workflow beats an agent for a given use case (we wrote more on this specifically in why most “AI features” should be a button). The willingness to talk you out of the more impressive-sounding option is one of the more reliable signals you’ll get in a sales conversation.

Ask for a specific, bounded first engagement

Teams confident in their own process are comfortable scoping a short first engagement, a few weeks, with a concrete deliverable, rather than asking for a multi-month commitment up front. We default to six-week engagements for exactly this reason; see why six-week projects beat six-month ones. If every proposal you’re getting is a six-month retainer with no clear first milestone, that’s worth questioning.

What good looks like, concretely

A team worth hiring will talk specifically about evals, retries, idempotency, and cost before you ask. They’ll push back on scope that doesn’t need an agent. They’ll hand you a codebase you can walk away with. And they’ll be comfortable proving it in a few weeks rather than asking you to commit to a few months on faith.

If you’re evaluating us against that list, the contact form is the fastest way to get specific answers for your project, not a sales pitch.