Every learning investment eventually faces the same question: Did it actually change anything in the work? Too often, learning answers that question too late and with the wrong evidence, such as completion rates and satisfaction scores.
Artificial intelligence did not create the impact problem, but it has certainly exposed it anew.
The evidence that AI makes individuals faster is solid: In a study of 5,179 customer support agents, researchers found that access to a generative AI assistant raised issues resolved per hour by 14 percent on average, and by 34 percent for novice and less-skilled agents. Sponsors now expect value just as fast as the tools deliver it.
But faster people don’t automatically make for a better organization. That also depends on who reviews the output, where the saved time goes and whether quality holds. When it doesn’t, a draft can be created in minutes, but the review that follows takes longer than the draft ever did. The effort hasn’t disappeared; it’s moved to someone else’s desk.
The fix isn’t a longer evaluation; it’s three questions asked before launch: What should people be doing differently? What’s actually stopping them? And what result would make us stop? They apply to any learning investment, from an AI rollout to mandatory data-protection training. I learned that by failing them myself.
A plan that looked finished
Earlier this year, I wrote an AI enablement plan for client-facing consultants. I was happy with it. It identified three specific use cases per role: proposal drafts, research synthesis and meeting write-ups. The same pattern applies to any team whose work involves writing things like sales proposals, policy drafts or client reports. The plan avoided demos in favor of 90-minute practice labs on live client work. Each practice got a champion with protected time, and client-approved AI sat inside the role pathway instead of in a separate module.
The goal was to convert saved time from faster proposal creation into more capacity for billable work. We would measure that through usage, confidence and time saved on drafts. The plan named a specific barrier (people not knowing what good looks like) rather than a blanket skills gap. It also set a week-four pulse check, a 60-day impact report and a recommendation for phase two.
Next, I pressure tested the plan against those three questions, and it failed all three. It measured usage, confidence and draft speed, not whether proposals got better. The plan assumed that time saved on drafts would become billable capacity, and it assumed the barrier. Its review points were dates on a calendar, not decisions anyone had agreed to make.
This is the gap learning has faced for decades: We measure the learning instead of the work, and we have no concrete way to decide whether to keep going.
3 questions that give speed a direction
Learning has too often chosen perfection over speed. But the answer isn’t speed for its own sake. It’s to agree on the evidence you need before you build. Making the traditional build a little faster with AI still leaves the question of value until the end, because no one decides in advance what would make them stop. Moving fast toward a measured outcome is the only speed that compounds, and these three questions supply that direction.
What should people be doing differently? Activity is not improvement. A respected learning leader at a media company recently shared that their organization’s mandatory AI course had reached 95 percent completion. Nearly everyone finished because they were required to, so that number says nothing about what people now do differently or what new value they create.
Completion is worth tracking, but it is not worth celebrating as an outcome. In my plan, “confident using AI” and “time saved on drafts” became this: consultants draft proposals with AI, and the drafts reach review needing less rework. Total effort per proposal falls, and quality holds.
Behavior drives work output, and work output drives business results, so the discipline is simple: pair every speed or volume measure with a quality condition. Identify these targets by defining what good looks like. You can do this by asking who already does this well and what they produce that others don’t.
Then track:
- Use: Did people use the tool or the intervention at all?
- Total effort: How many review cycles did a proposal take, not just how fast the first draft appeared?
- Quality: Did the standard hold, according to the people who review and buy the work?
- Time saved: Where did it go? Manager reports, self-reports and timesheets each give a partial answer.
Be sure to avoid vague indicators like states of mind (“understands,” “confident,” “embracing”), usage reported as the outcome and counts with no quality condition attached. A useful test: How would an outsider spot the change on a normal Tuesday?
What’s actually stopping them? Most learning requests arrive with the solution (not the desired outcome) already decided. My favorite example is the sales program. Sales are down, so the business demands a sales skills program. But if you ask the sales team why deals aren’t closing, they may tell you they have no leads. Teaching people to have better conversations helps only if there are conversations to be had.
Before assuming a skills gap, check the work before the worker, in this order:
- Clarity: Do people know what good looks like?
- Ability to do the work: Do they have the tools, access, time, process and handoffs?
- Consequences: What happens if they do it? What happens if they don’t?
- Skills: Only then look at skills, capacity (what comes off the day?) and motivation.
For each, separate what you know from what you are assuming. Then ask one more question: What will people do to make this look successful, even if it isn’t? That question matters because measures shape behavior. If the organization celebrates usage, people will find ways to use the tool whether or not it helps the work, and some may go around review steps to look productive.
Each workaround can register as a win on a dashboard while creating risk the dashboard never sees: client material in the wrong tool, or unchecked output reaching a customer.
My plan assumed the barrier was an unclear standard. That was plausible. So was a tool policy that kept client material out of the AI tool, a poor fit with how proposals get written, or no spare time at bid deadlines.
Any of those would have sunk the plan, and traini
