THE DESIGN QUESTION
What may AI recommend or execute, and which decisions require an authorized person?
“Keep a human in the loop” sounds reassuring. But it leaves several practical questions unanswered.
Which person? At what point? With what information and authority? What happens if they disagree with the system—or are unavailable?
When AI participates in a service or business process, these details shape who controls the outcome. They need to be designed into the system.
Consider a service provider handling a customer’s request for an exception to a cancellation policy. AI could summarize the request, retrieve relevant terms, identify missing information, and suggest a response. Whether it should approve the exception is a separate decision.
Separate recommendation from authority
A system’s ability to produce a plausible recommendation does not establish its authority to act.
For the cancellation request, several permissions can be distinguished:
- Read the information needed to assess the case.
- Prepare a summary and draft response.
- Recommend an outcome with supporting reasons.
- Execute a decision within explicitly authorized limits.
- Approve an exception outside those limits.
These permissions should be assigned deliberately. The person authorized to send a routine response may not be authorized to waive a substantial fee or change a contractual commitment.
The same distinction applies to AI. Access to information should not automatically confer permission to change a record, make a commitment, or approve an exception.
Allocate decisions according to their consequences
Not every action requires the same level of control.
Correcting a draft before it is sent differs from reversing a decision that has already affected a customer. A routine, reversible administrative action differs from a decision with significant financial or personal consequences.
Useful questions include:
- How serious would an incorrect outcome be?
- How easily could the action be reversed?
- Does the case fit an established rule, or require an exception?
- Is the supporting information sufficient and consistent?
- Does the decision require a specifically authorized person?
A routine request might be handled automatically within approved rules. An unusual request involving conflicting terms, missing facts, or a substantial concession may require human review.
Where policy or applicable requirements reserve a decision for a person, that boundary should be explicit. Model confidence alone is an insufficient basis for assigning authority.
Make human review a workable task
A review step is useful only when the reviewer can meaningfully assess the case.
Presenting an “Approve” button beside an AI-generated conclusion may encourage acceptance without examination. A reviewer needs access to the original request, relevant policy provisions, missing or conflicting information, and the proposed action.
They also need practical ways to disagree: edit the response, request more information, choose a different outcome, or refer the case to someone with greater authority.
In the cancellation example, the system should distinguish the customer’s stated circumstances from its own interpretation. It should make the relevant policy version available rather than requiring the reviewer to trust a summary.
The operating plan must allow enough time and appropriate expertise for this work. Adding human approval to a process without providing capacity can create a bottleneck or a rubber stamp.
Define what happens when the case cannot proceed
A system needs a clear path for situations it cannot resolve within its authority.
The customer may dispute the facts. Two policies may appear to conflict. The authorized reviewer may be unavailable. A required source may be inaccessible.
Decide how each situation should be handled.
The system might pause the action, ask for missing information, route the case to a designated role, or provide an approved interim message. It should avoid treating silence or a timeout as approval unless that behavior has been explicitly authorized for the particular action.
Any handoff should preserve the case history and make ownership visible. A customer should not have to restart the process because responsibility moved from the system to a person.
Enforce boundaries in the system
Written instructions are one part of control. Technical permissions and workflow rules need to support them.
For example, an AI assistant may prepare a fee-waiver recommendation while a separate service checks whether the amount falls within an approved limit. An action outside that limit can require an authorized approval before execution.
Access controls, validation checks, and approval requirements should prevent the assistant from bypassing the intended decision path.
Keep a proportionate record of the relevant inputs, policy version, recommendation, approval, and action taken. This helps the organization investigate disagreements and correct errors, while avoiding unnecessary retention of sensitive information.
Someone should also own changes to these boundaries. A policy update, expanded service, or new system integration can change what an existing permission allows in practice.
Make the decision boundary explicit
A practical starting point is a short decision-authority table.
For each consequential action, record:
- What AI may prepare or recommend.
- What it may execute automatically, and under which conditions.
- Which role may approve exceptions.
- What triggers a pause or escalation.
- Who owns correction when something goes wrong.
This turns a broad commitment to human oversight into an operating arrangement that people can understand and technology can enforce.
At CultureTech, we help clients define these boundaries across business processes, AI capabilities, and software architecture. The objective is useful assistance with clear authority, workable review, and accountable decisions.
