Prototype ≠ Product. Sometimes It Isn't Supposed to Be.
What I learned building a chatbot I never planned to keep, and why that's exactly how innovation should work.
Good morning, all, and I hope you all were able to enjoy a rewarding Memorial Day Weekend with your families and loved ones. My family and I shared a few moments talking about folks we served with and remember, but mostly, it was to ask what they would do with this day—or how we remember those we’ve lost.
I picked that up from my mom. Her younger brother died of cancer, and she often thinks of him and asks, “What would Leroy do with this day?” And I can’t think of a more fitting way to remember someone than to celebrate them by living.
Now that we’re back at it, let’s talk innovation, experimentation, prototyping, and what we actually can and should do with the little things we cobble together.
I love to build things. With the proliferation of AI tools out there now, I feel like I can be Tony Stark, hanging out and tinkering and creating amazingly things. And a couple months ago, I built a particular tiny agent.
It’s not the most complicated one I’ve built, not by far. And it has a singular purpose. I called it AskDCP and it answers questions about the Army’s Direct Commissioning Program—how to apply, what Detachment 201 is, where your packet goes in the process, what documentation you need.
I loaded in about 50 FAQs, scraped the relevant regulations and HRC pages, wrote a set of instructions, and stood it up on Chatbase, because that was a simple user-friendly forward-facing platform that could handle the volume if it ticked up and that I didn’t have to do a whole heck of a lot to monitor.
It got about 1,100 chats in its first week, got shared around, and one of my friends who had gone through the direct commission program remarked that he wished he’d had a tool like this available when he was in the program.
People asked how I was going to scale it and I think I shocked a couple people by saying…I wasn’t, at least not that particular tool. It was a prototype, built to learn from.
Why would you build something that useful and not deploy it at scale?
Because it’s not made for deployment. And at the time, we didn’t know enough about what it needed to do, what systems and data it needed access to, what autonomous actions it needed to perform, to figure out where it needed to be built. We were just using it to learn, grow, and figure out what we need.
And confusing what something is with what it could become is one of the most reliable ways to kill both the thing and the vision.
Let’s decode it. 🚀
Innovation Operationalized
What actually gets ideas from “interesting” to “people can use this at scale”
In 2022, when I was at Army Human Resources Command, I built a framework I called Innovation Operationalized (it’s also included and explained in detail in my book). It came out of a specific frustration: the fact that I was running an innovation cell and no one could tell me specifically what innovation was supposed to be or do, so it devolved to innovation theater.
You’ve seen innovation theater. It’s the hackathon that produces a beautiful slide deck and nothing else. The pilot that gets funded, gets applauded in a brief, and then quietly disappears because nobody asked the hard question of how it gets from here to a thing 1.3 million soldiers can actually use. The AI tool that gets deployed to a team of twelve before anyone figured out what it was supposed to do for them.
Organizations are extraordinarily good at generating ideas. They are much less good at the deliberate, disciplined work of taking an idea through the full arc: testing it, scaling it, integrating it, and assessing whether it’s actually working.
My framework has six phases: Ideate. Design. Prototype. Scale. Integrate. Assess.
They sound obvious. The reason I had to draw the flowchart is that most organizations skip straight from Ideate to “let’s deploy this everywhere” — bypassing the middle four entirely. And the middle four are where almost all the real work lives.
AskDCP is sitting squarely in Prototype and that’s all it’s meant to do.
So…what exactly do you mean by “prototype?”
A prototype is not a bad version of the final product. It’s a specific kind of tool, built for a specific purpose: to answer questions you cannot answer any other way.
When I built AskDCP, the questions I needed answered were:
What are people actually asking? You can theorize all day about what a direct commission applicant wants to know. You can write FAQs based on the questions that come to your inbox. But until you put something in front of real users and let them ask in their own words, you’re guessing.
After one week and 1,100 chats, I’m not guessing anymore. I know what confuses people. I know where the process is opaque. I know which questions come up over and over that the official documentation doesn’t answer clearly.
What does a good answer actually look like? This sounds obvious until you try to build it. Does the answer need to include a next step? (Yes, always.) Does it need to avoid acronyms? (Largely.) How long is too long before people stop reading? (Shorter than I thought.)
You don’t figure this out with surveys and emails. You figure it out by watching real interactions and iterating.
What are the failure modes? I had colleagues try to break it. Army-Navy rivalry questions. Attempts to get it to do math problems. Edge cases the instructions didn’t anticipate. Every failure mode I found in prototype is a failure mode I can engineer out (or someone, unlike me, who’s actually good at software development can engineer out) before this runs on an official platform that Soldiers actually depend on.
Is this idea worth building for real? This is the hardest question, and it’s the one most organizations never actually ask because they’ve already committed resources and nobody wants to be the person who says “actually, this shouldn’t scale.”
A prototype gives you the data to answer it honestly and the license to pivot or stop without having burned millions on a massive contract.
AskDCP answered all four of those questions. The answer to the last one was yes. So now the conversation shifts.
From prototyping to “please build this so that a million active users won’t break it”
I have a working prototype. I have 1,100 conversations worth of data on what people need. I have a process flow diagram. I have a contact matrix showing who owns which part of the direct commissioning pipeline. I have a tested set of guardrails: what it should and shouldn’t answer, how it should route requests, what tone works for this audience.
What I don’t have is a software engineering team, an Army-approved hosting environment, IL-5 certification, or the infrastructure to make this work at the scale of the entire Army’s direct commissioning program.
So I’m not trying to build that part. That’s not my job and it shouldn’t be.
My job was to figure out what we needed and to prove it was worth investing in and building. That’s the prototype phase. The next phase, Scale, is where you hand the blueprint to people who have the infrastructure, the contracting vehicles, and the engineers to build something that 1.3 million people can actually use without breaking it.
I had that conversation with one of our vendor teams, who looked at AskDCP and immediately recognized it as, in their words, their bread and butter, the kind of grounded knowledge-based agent their platform is designed to run at institutional scale. They can take the documents I trained it on, my protocols, my findings on what worked and what didn’t, and then rebuild it in a production environment, host it on an official platform, and give the Direct Commissioning Program a tool that the team can actually use and sustain.
The prototype made that conversation possible. It gave me something concrete to show. It gave the vendor team enough to understand the use case immediately. It gave leadership enough to say yes to a pilot without having to trust a slide deck.
That’s what prototypes are for.
Innovation operationalization explained.
Most organizations don’t fail at innovation because they lack ideas. They fail because they conflate different phases of the innovation process and apply the wrong standards to each one.
A prototype evaluated like a production system will always fail the evaluation. It’s supposed to be rough, iterative, and provisional. It’s not meant to go from zero to sixty, it’s meant to learn how the solution impacts the system.
A production system built like a prototype will collapse under real load. Prototypes are quick and dirty. They’re not designed for scale, security, or sustainability.
And an idea evaluated before it’s been prototyped will often get the “What’s the ROI?” question very, very, very wrong.
Innovation theater happens when organizations skip phases. They jump from a great idea to a full deployment because the idea got someone excited at a senior level, or because there was budget to spend, or because the optics of “launching an AI initiative” felt more valuable than the patient, unglamorous work of actually figuring out whether the thing does what it’s supposed to, or at least something useful.
Because the most important phase of the operation isn’t Scale or Ideate or even Prototype. It’s Integrate. Because if the solution doesn’t integrate into your systems and workflows, it’s the tree that fell in the woods. Often a very expensive tree.
The flowchart I drew in 2022 was an attempt to give people permission to do the right thing at each phase and to resist the pressure to do the wrong thing at the wrong time.
Ideate without worrying about scale.
Design without worrying about budget.
Prototype without worrying about perfection.
Scale without trying to do it yourself if you’re not the right person to scale it.
Integrate by asking hard questions about whether the environment is actually ready to absorb the thing you built.
Assess by measuring whether anything actually changed — not whether the initiative launched.
What this looks like in YOUR organization.
You don’t have to be building chatbots and agents and platforms (oh my) for this to apply to your organization.
If you’re a manager who had a great idea for automating a workflow and you’re frustrated that IT won’t buy it immediately, build a prototype. Show it works. Then make the ask with evidence.
If you’re an executive who keeps approving initiatives that never seem to deliver, ask whether your organization has permission to prototype without being pressured to scale before it’s ready. Chances are that good ideas are dying at in their infancy because they “don’t work” according to Scale phase standards.
If you’re a technologist or an AI enthusiast building tools for your organization, figure out which phase you’re actually in, and resist the temptation to treat a prototype like a product. The discipline of knowing what you’re trying to learn and being willing to stop or pivot based on what you find is what separates useful innovation from theater.
And if someone asks you why you built something you’re not planning to scale or sell, it’s because it’s not meant for that, and not time.
AskDCP is doing its job. The data is good. The blueprint is clear. The handoff is in progress. That’s how this is supposed to work. And that’s all it’s supposed to do now.
