THE DESIGN QUESTION
What must a pilot demonstrate under real conditions before a larger investment is justified?
A demonstration can show that a system produces an answer, completes a task, or connects several technologies. A pilot should help an organization make a more consequential decision: is this worth taking further?
That decision requires evidence about usefulness, reliability, human involvement, and the cost of operating the system. It also requires clarity about what remains untested.
A well-designed pilot gives a promising idea enough contact with reality to expose the assumptions that matter.
Begin with the decision the pilot must inform
Before defining features, identify the decision that will follow the pilot.
Is the organization deciding whether to introduce a new service, expand an internal capability, change a customer experience, or invest in further development?
Then ask what evidence would change that decision.
Consider an organization testing a knowledge-based assistant for customer onboarding. Showing that the assistant can answer common questions may be useful, but insufficient. The investment decision may depend on whether customers can complete the right next step, whether staff can handle exceptions, and whether maintaining the service is commercially worthwhile.
These questions should shape the pilot from the beginning.
Keep the scope narrow enough to learn
A limited scope makes it easier to observe what works, understand failures, and revise the design.
The onboarding pilot might cover one service, one customer group, and a defined set of questions. It could use approved knowledge sources and clear rules for referring uncertain cases to staff.
The scope should be manageable, but still contain the relationships being tested. If human review is essential to the proposed service, the pilot needs to include that review. If the value depends on customers taking action, testing answers alone will leave the central assumption unresolved.
Define both what the pilot includes and what its results cannot establish.
Test the work around the technology
Real systems operate through more than their most visible interface.
Knowledge needs updating. Access needs managing. Exceptions need attention. Someone must recognize errors, decide what to do, and carry out corrections.
A pilot should make this work visible.
In the onboarding example, who checks that guidance remains current? What happens when a customer provides incomplete information? Can staff understand why a case was referred to them? Can the service recover when a dependency becomes unavailable?
The answers reveal whether the operating model can support the proposed value—and which parts need further preparation.
Define evidence before interpreting results
Agree in advance on what to observe and what would count as a promising, unacceptable, or inconclusive result.
A practical set of measures might include:
- Value: whether users complete the intended task or receive a meaningfully better service.
- Reliability: whether the system behaves consistently across the situations included in the pilot.
- Human workload: the time required for review, exceptions, support, and knowledge maintenance.
- Safeguards: whether access restrictions, escalation rules, and correction processes work as intended.
- Economics: the delivery and operating costs relative to the value the organization expects to create.
Use an existing process as a comparison where feasible. Record relevant differences in users, cases, and operating conditions so that apparent improvements can be interpreted fairly.
Feedback matters, but enthusiasm alone should not determine the investment decision.
Make the limits of the evidence visible
A successful pilot supports a decision within the conditions it has tested.
A small group of engaged users may behave differently from a wider customer base. A carefully prepared knowledge set may not represent future information quality. Low usage may leave workload peaks or unusual failures unexplored.
Separate what was observed from what is still assumed.
The evidence may justify expanding one part of the system while keeping another under review. It may also show that a further test is needed before making a larger commitment.
This makes the next investment more specific: what should be expanded, under which conditions, and with what additional checks?
End with a decision, not just a presentation
A pilot should conclude with a clear recommendation supported by the evidence.
The next step may be to expand within defined limits, prepare missing knowledge or operating capabilities, revise the design and test again, or stop because the case for further investment is too weak.
Stopping can be a useful outcome when it prevents a larger commitment to an unsuitable approach.
At CultureTech, we help clients connect pilot design, system architecture, and evaluation to these decisions. The aim is to establish what works, what it takes to sustain it, and whether the evidence supports the next step.
