Your Next Move #04 — How to Run an AI Pilot That Survives Contact With Your Org
Design your AI pilot around the one protective metric that determines success before kickoff — so the org cannot redefine success after the fact, and you control the read-out.
Most AI pilots fail in the design phase — before anyone touches the tool.
Not because the technology doesn't work. Because the organization never agreed on what "working" means. Then the pilot runs, produces ambiguous results, and the people who wanted to greenlight the investment call it a success while the people who wanted to kill it call it a failure. The manager who ran it gets caught in the middle, with no clean read-out and no clear next step.
This is not a technology problem. It is a measurement problem that has a measurement solution.
The move is to set the protective metric before kickoff — the one number that determines, in advance, whether the pilot worked. Then run the pilot against that number, not against the room.
Why Most Pilots Don't Survive Contact With the Org
Here is the failure pattern. It runs the same way almost every time.
The pilot launches without a defined success metric. Leadership says "let's try it and see." The team runs the pilot enthusiastically. Three weeks in, someone from finance asks how it's going. The manager says "it's going well, the team is adapting." Finance hears: outputs not yet measurable.
Goalposts move mid-pilot. At the four-week check-in, the original sponsor has added new questions: can it handle [new use case]? What does the vendor think about [adjacent capability]? The pilot is now carrying three different mandates and none of them has a clean metric.
The read-out is contested. The manager presents results. The productivity improvement is real but modest. The person who was skeptical says it's not enough to justify the cost. The person who championed it says the team is still in the learning curve. There is no number that resolves the argument — because no one agreed on what the number should be before the pilot started.
The manager walks away from that meeting without a decision. The pilot drifts. The tool gets used informally or abandoned quietly. Nobody calls it a failure because nobody agreed on what failure looked like.
The Pilot-Design Checklist
Complete this before the pilot starts — ideally in a document you share with your manager and the pilot sponsor. The act of sharing it forces the agreements that prevent the failure pattern above.
Part A — Before Kickoff (Days –7 to 0)
Scope
[ ] One workflow named: [what specific process is this pilot testing?]
[ ] One team scoped: [which team, how many people, which roles?]
[ ] Duration fixed: [start date → end date, maximum 30 days for a first pilot]
[ ] Tool identified: [vendor / product name, version, licensing status confirmed]
Success metric — the most important field on this list
[ ] One primary metric named and written down: [time saved per task / error rate / output volume / cost per output — pick one]
[ ] Baseline established: [what is the current performance on this metric before the AI tool is introduced?]
[ ] Target set: [what number constitutes success? Not "improvement" — a specific number or threshold.]
[ ] Agreed in writing: [who has signed off on this metric? Name them.]
Kill criterion
[ ] Named explicitly: [what result or condition means we stop the pilot?]
[ ] Timeline for kill decision: [if we haven't seen [X] by [date], we stop. Written.]
Stakeholders
[ ] Read-out date is in calendars: [date, who attends, format — 30-minute meeting or written summary?]
[ ] The one stakeholder who can block the result has been briefed before the pilot starts
[ ] Data collection responsibility assigned: [who is tracking the metric? How? How often?]
Part B — During the Pilot (Weekly, 10 Minutes)
Each week, write three sentences:
