Gavel Playbook · Strategy

How good does an AI feature have to be before launch? Nine operators on where the bar actually sits.

Every guide on this question was written by somebody selling an evaluation tool. These nine calls come from teams shipping AI into production, with the timestamps so you can check them.

Tactics
9 cited plays
Sources
5 channels
Read time
8 minutes
Updated
August 2026

Do it in this order

How good does an AI feature need to be before launch?

There is no universal accuracy number. The bar is set by what breaks when the feature is wrong. A reversible surface, a draft a human sends or an internal tool, ships early and hardens in public. Anything touching money, customer data or a promise you cannot retract needs an eval suite gating the deploy first. Kavak runs 96 percent of customer interactions through agents and credits evals for the speed, not for the caution.

The launch-readiness sequence

  1. 1 Name what breaks. Write the single worst outcome if the feature is wrong. That sentence, not a benchmark score, sets your bar.
  2. 2 Write ten evals from real outcomes. Score every prompt and model change against them. Without evals you can ship a change but you cannot tell whether you made it worse.
  3. 3 Freeze your ten weirdest inputs. Real traffic rarely produces the failures that hurt. Turn the rare cases into a fixture you run every time.
  4. 4 Cut the surface area. Every setting you expose is a supported path and an attack surface. Delete the ones almost nobody touches before strangers arrive.
  5. 5 Price the tokens, name the owner. Put inference cost in the unit economics, and decide who is accountable when the agent is wrong, before a customer has to ask.

You got the thing working on Friday and it felt like the hard part was over. Search for what to do next and you get accuracy thresholds, a metrics table and a free trial of an evaluation platform. All technically useful, none of it written by somebody who has shipped an agent to real customers and watched what actually broke.

This playbook is the opposite input. Nine calls, each from an operator shipping AI into production, each deep-linked to the moment they say it. What they agree on is that the number you are looking for does not exist. The bar is set by consequence, not by a benchmark, and the teams moving fastest built the machinery to detect failure early instead of waiting to stop having any. This is not the engineering last-20 percent, which the vibe-coded to production playbook already covers. This is the go or no-go call.

"Evals serve as the critical brakes that allow rapid iteration."

Kavak, on running 96 percent of customer interactions through agents

The Plays

Nine calls, from teams already shipping it

01

Dmitri Dolgov, Waymo · Y Combinator

The demo is 1% of the work

Dmitri Dolgov opened his YC talk by separating two things founders routinely merge: an early working prototype, and a production-grade service. The gap between them is not a percentage of remaining effort. Dolgov describes reliability as living on an exponential ladder of nines, where climbing from 90 percent to 99 is expensive and climbing from 99 to 99.9 costs more again. Each nine is a different engineering problem, not more of the last one.

Waymo now drives over four million fully autonomous miles every week, and that number is what the ladder buys after years of paying for it. The founder version is smaller and identical in shape. Your demo cleared the first rung. Price the rest of the climb before you promise anyone a date.

Steal it

Before you commit to a launch date, write down the failure rate you ship at today and the one you need. If you cannot name both numbers, you are estimating the demo.

02

Ali Massa, Kavak · a16z

Evals are the brakes, not the paperwork

Kavak sells used cars, finances them and moves them, and now runs 96 percent of customer interactions and 95 percent of transactions through agents. That is not a pilot. The thing their team credits for the speed is evals, and the word they use is brakes. A car with no brakes does not go faster.

It goes slower, because the driver never commits to a corner. Evals do the same job for a shipping team. They let you change a prompt, swap a model or widen a workflow on a Tuesday and know by Tuesday afternoon whether you broke a customer path. Without them every change is a guess, so every change gets deferred, and the team that skipped the eval suite to move faster ends up moving slowest.

Steal it

Write ten evals from real customer outcomes before your next model change, and make the pass bar the thing that gates the deploy.

03

Peter Steinberger, OpenClaw · Y Combinator

Review for risk, not for lines

Peter Steinberger maintains an open-source agent with thousands of contributors, so his code review problem is the extreme version of yours. His answer is to stop treating review as reading. He treats it as risk management: the question is not whether every line is correct, it is what breaks if this is wrong and who it hurts. That reframe scales down to a solo founder shipping AI-written code, where reading every line is both impossible and beside the point.

The model wrote it faster than you can audit it. What you can audit is blast radius. His companion rule for early product points the same instinct at scope: build what annoys you, for your first users, and let the irritation choose the work.

Steal it

On your next AI-written change, skip the line-by-line read. Write one sentence naming what breaks if it is wrong, then review only the paths in that sentence.

04

Dmitri Dolgov, Waymo · Y Combinator

Simulate the case your traffic never produces

The failures that hurt are the ones your real traffic almost never produces. Waymo answers with large-scale generative world models and closed-loop simulation, built to train and evaluate against rare scenarios a fleet would take years to encounter naturally. Closed-loop is the load-bearing word: the simulated world reacts to what the agent does, so you are testing a whole chain of decisions instead of one frozen input. Founders read this as a self-driving budget problem and skip it.

The cheap version is not a budget problem. Take the ten weirdest inputs your product has ever seen, the malformed paste, the hostile prompt, the empty state, the customer who typed in a second language, and make them a fixture. You do not need a world model to stop being surprised by the same ten things.

Steal it

Collect the ten strangest real inputs your product has ever received, freeze them as a fixture, and run them on every change from now on.

05

Peter Steinberger, OpenClaw · Y Combinator

Configuration options are the surface area that bites

OpenClaw went vertical on Discord over New Year's Eve, and the reward was not a moat. It was security pressure, false media claims about malicious skills, and a codebase carrying thousands of configuration options plus dependency churn nobody had signed up to manage. Every option is a supported path. Every supported path has to keep working, get documented and get patched.

Flexibility feels free while you are building and turns expensive the moment strangers arrive, which for an AI product is also the moment the option space becomes an attack surface. The launch-readiness read is blunt: an unshipped setting is cheaper to delete than to defend, and the number of switches you expose is a reliability decision wearing a product costume.

Steal it

Count the settings you expose. Delete every one that fewer than a tenth of your users will touch, and do it before launch while deleting is still cheap.

06

Brett · The Brett Way

Find the step where people quit before you add a feature

Brett gave AI seven days to get real users for an app and spent two of them building. The first paying customer arrived on day four, immediately after he fixed a geo-targeting setting that had been quietly buying junk traffic and repaired the mobile onboarding bugs killing completion. Neither fix was a feature. Both were worth more than one.

The pattern generalizes past paid ads: an AI product that answers well and loses people at the third screen has a plumbing problem, not a model problem, and model problems are the ones founders enjoy solving. He finished at a 15 dollar acquisition cost with seven trials on day six, a validation signal produced almost entirely by debugging the path he had already built.

Steal it

Open your funnel, find the single step with the largest drop, and fix that one before you touch the model or ship anything new.

07

Joe Schmidt and Andy Warfield · a16z

Put an end date on the pilot

Enterprise AI pilots have a way of becoming permanent residents. The a16z sales conversation names the fix without ceremony: set a hard end date and written success criteria before the proof of concept starts, or you are running a science project on your own payroll. For an AI feature this matters more than it does for ordinary software, because a pilot with no bar can always be extended on the promise that the next model release will close the gap. It usually will not close the gap you care about, and while you wait the pilot eats the engineering attention that would have won a second customer.

The end date is what converts a pilot into either revenue or a decision.

Steal it

Agree two things in writing before the next pilot starts: the date it ends, and the number that counts as success.

08

Matt Swulinski, Wispr Flow · 20VC

Put tokens in the unit economics before you scale

Matt Swulinski, who ran growth at Superhuman and Wispr Flow, treats a three-to-one ratio of lifetime value to acquisition cost as the target and flags where AI products quietly miss it. Inference is a variable cost that moves with usage, so the customer who loves your product most is also the one eroding your margin fastest. Traditional software has the opposite shape, which means the instinct founders bring to this is wrong by default. He is equally direct that early burn is acceptable while you are still learning.

What is not acceptable is a model of the business with tokens left out of it, because then you cannot tell growth from a leak. Launch readiness includes your cost per heavy user, not just your cost per signup.

Steal it

Work out what your heaviest ten percent of users cost you in inference this month, and put that number in the unit economics before you scale spend.

09

Greg Isenberg · Greg Isenberg

Decide who is accountable when it is wrong

Greg Isenberg's answer to why the SaaS apocalypse keeps not arriving is unglamorous and correct. Enterprises stay on established tools because software maintenance is a real pain they are paying somebody else to own, and because they need a clear accountability structure when something goes wrong. AI-only vendors cannot sign that line yet. For a founder selling into any company with a procurement process, this belongs on the launch checklist.

Somebody has to be answerable when the agent gets a number wrong, and if your answer is that the model did it, the product is not finished. The bar was never perfect output. It is a named owner, a way to detect the error, and a path to make the customer whole.

Steal it

Name the person who answers when the agent is wrong, the alert that tells them, and the remedy the customer gets. Write all three down before launch.

Where the operators disagree

Ship it raw, or is the minimum more rigor than you think?

Andrej Karpathy says ship it. Vibe coding is a legitimate and now dominant way to build, with a quarter of YC founders reporting codebases over 95 percent AI-generated. His position is that embracing it is the necessary move: put the thing in front of real use and let reality tell you which parts need hardening.

Eric Ries says minimum is not the same as rough. An MVP is a minimum viable experiment, not a low-quality product. What counts as minimum is set by the hypothesis you are testing, and in his framing that usually demands more rigor than founders assume, not less.

The call: the split turns on which surface you are shipping. Reversible surfaces go Karpathy. An internal tool, a landing page, an agent that drafts while a human sends. Break it, fix it, nobody is harmed. Anything touching money, customer data or a promise you cannot retract goes Ries, and evals are how you buy the speed back there, which is exactly the trade Kavak made. A generic chatbot flattens this into move fast but be careful and hands you nothing you can act on.

Deep dive: Duke on setting kill criteria before you start

Read it for your situation

How to use this playbook

Solo founder with a working demo and no users
Start at 06, not 01. Your reliability bar is low because your surface is reversible and your blast radius is one person. The bar that actually binds you is whether anyone finishes onboarding.
Shipping an AI feature into an existing product
02 and 04 together, in that order. Evals give you the gate, the rare-case fixture gives you the inputs to run through it, and the pair is what lets you change a model without holding your breath.
Selling into a company with procurement
09 and 07. Accountability is a sales objection before it is an architecture choice, and a pilot without an end date is how a promising deal turns into unpaid engineering.
Usage is growing and the inference bill is growing faster
08 first, then 05. Work out the cost of your heaviest users before you scale spend, then cut the option surface that is quietly multiplying the paths those tokens run through.

Nine plays is a menu, not a sequence. The point of Gavel is that you can bring your actual situation, the surface you are shipping and what it touches, and get the two or three that apply to you, with the operators who disagree about your specific case named instead of averaged away.

Common questions

What founders ask about the bar

What accuracy does an AI feature need before launch?
There is no fixed number. The bar comes from the cost of being wrong. A summary a human reviews tolerates errors that a payment or a medical claim does not. Kavak runs 96 percent of customer interactions through agents because evals catch the failures, not because the model is perfect.
What are AI evals and do I need them before launch?
Evals are scored tests you build from real customer outcomes. Public benchmarks cannot tell you whether your own product broke. Kavak's team calls them the brakes that let you go fast. You need them before any change you cannot manually verify, which for an AI feature is every change.
Should I ship an AI feature that is only 80 percent reliable?
Yes, if the surface is reversible and a human stays in the loop. Andrej Karpathy's position is to ship and harden what breaks. Eric Ries counters that minimum viable means minimum viable experiment, and the bar is set by the hypothesis you are testing.
How do I test rare failure cases before launch?
Waymo trains and evaluates against closed-loop simulation because rare scenarios almost never appear in real driving miles. The founder version costs nothing: collect the ten strangest real inputs your product has received and run them as a fixture on every change.
Who is responsible when an AI feature gets something wrong?
You are, and enterprise buyers will ask. Greg Isenberg's point is that companies stay on established software partly for maintenance and a clear accountability structure, which AI-only vendors cannot yet provide. Launch readiness includes a named owner, error detection and a way to make the customer whole.

What is your bar, for your surface?

Bring what you are shipping and what it touches. Get the two plays that fit, the operators who disagree about your case, and the citations to check them.

Free with Google. 20 credits/month forever. Pick a plan in 30 seconds after signin.

The Gavel Playbook Newsletter

One new playbook
every Monday morning.

Cited frameworks from operators who've shipped, in your inbox before your week starts. No spam, no upsells, no recycled LinkedIn takes.

One email a week. Unsubscribe with one click. We never sell or share email.