The AI has explained its recommendation in six immaculate paragraphs. The manager has one remaining problem: she still cannot explain why she should follow it.
The answer contains a rationale, a source list and a reassuring conclusion. What it does not make clear is which assumption matters most, what evidence would change the recommendation or whether the proposed action fits the situation in front of her.
This is a useful distinction for organisations introducing AI. Producing an explanation and creating understanding are different achievements. The first belongs to the system. The second has to happen in a person.
An effective explanation should help that person assess a recommendation, recognise its limits and choose what to do next. Its success cannot be inferred from its length, fluency or ability to make the answer sound reasonable.
Explanation has a purpose
A September review by Georgia Tech researchers, covering 33 publications on trust and interaction design, argues for evaluating whether explanations support appropriate reliance. Its findings are mixed across contexts, and much of the underlying work uses simulations or self-reported trust. It does not establish a universal interface recipe.
It does sharpen the question. A person feeling more confident after an explanation is not enough. We should also care whether they can tell when the recommendation deserves that confidence.
For a business, this distinction matters at the point of use. A project director may have to justify a change to a customer. An operations manager may have to decide which exception requires attention. A team leader may have to explain why a familiar process is being altered.
In each case, a plausible paragraph is useful only if it helps the person do the work.
Give the reader something they can inspect
Consider an illustrative proposal to combine two customer onboarding steps. An AI system recommends the change because the steps collect similar information and appear to delay account activation.
The recommendation may be sensible. It may also overlook the fact that one step establishes a commercial requirement while the other confirms operational readiness. Similar forms can perform different jobs.
An explanation that says the proposal will reduce duplication does little to settle that question. The useful material is more specific: which information is duplicated, what each step establishes, which customers require both checks and what remains uncertain.
The person should be able to follow the relationship between the evidence and the proposal. They should also be able to identify the point at which a judgement entered the process.
This does not require exposing a model's private internal reasoning. It requires an inspectable basis for the recommendation: relevant source material, stated assumptions, the method where it matters and the conditions under which the proposal holds.
Another generated explanation is not automatically independent verification of the first answer.
Understanding is specific to a task
The same explanation can be adequate for one decision and insufficient for another.
A colleague deciding whether to investigate the onboarding proposal needs a useful starting point. Someone authorising a change for every customer needs evidence about exceptions, responsibilities and what happens if the new process fails.
A preprint submitted on 29 September followed 21 developers over four working days at two SAP sites. It found associations between perceived cognitive load, AI use and task context. The study is small and observational. It supports examining the actual work situation, rather than assuming that using AI generally makes thinking easier.
The practical implication is to evaluate explanations where people use them. Can the user locate the relevant fact during a busy review? Can they identify an unresolved condition without reading a page of general caveats? Can they distinguish a suggestion from a commitment?
These questions make cognitive ergonomics concrete. The discipline concerns how a system fits human attention, understanding and judgement. A beautifully written answer can still place the difficult work in the wrong place.
Put the important uncertainty near the decision
There are at least two unhelpful ways to communicate uncertainty. One is to conceal it behind a confident conclusion. The other is to attach so many qualifications that the reader cannot tell which one matters.
For the onboarding proposal, a useful qualification might be: “The analysis does not establish whether the second check is required for these customer categories.” That tells the reviewer where to look and why the proposal remains conditional.
“AI may make mistakes” offers no comparable direction. It leaves the user responsible for discovering the important gap unaided.
The aim is to make uncertainty actionable. Name the missing fact, connect it to the affected part of the recommendation and identify the next useful check. Where the evidence is sufficient, let the person see that too.
This can make progress easier. A team may be able to proceed with a limited change while investigating a separate exception, instead of treating one unresolved point as a reason to distrust the entire system.
Ask the person to explain the decision
One practical evaluation is to ask a participant to work through a realistic recommendation and then explain their choice in their own words. Keep the relevant evidence available. This is a test of usable understanding, not memory.
Ask them to identify the fact that mattered, the assumption that remained and the consequence they were accepting. Then change one material condition and see whether they can recognise which part of the recommendation needs reconsideration.
A confident summary that repeats the AI's wording is less informative than a short explanation grounded in the evidence. Equally, a careful decision to defer can be a successful outcome if the missing information is genuinely necessary.
This is a proposed evaluation exercise, not a validated measurement scale. It should sit alongside checks of factual accuracy, task quality, review effort and the consequences of error. A system should not win by making users articulate about a wrong answer.
The useful comparison is between the work before and after the tool was introduced. Did people find relevant evidence more easily? Did they notice consequential gaps? Could another colleague understand the decision without starting the investigation again?
Build capability that belongs to the person
At Attimo, our belief is that intelligence should increase human capability, not reduce human authorship. That gives an AI explanation a demanding job. It should leave the person better able to understand the situation and take responsibility for a choice.
Our Attention Cycle, Notice, Clarity, Action and Memory, provides a practical way to examine that experience. The user needs to notice what matters, understand enough to choose, act within their role and retain something useful for the next decision.
The opportunity is considerable. Systems can assemble evidence that would otherwise remain scattered and make complex alternatives easier to examine. Human expertise can then be spent on the parts that need judgement.
The explanation is finished when it becomes useful to that judgement.
Questions readers may be asking
What is the difference between explainable AI and human understanding?
Explainability concerns making a system's output or basis interpretable. Human understanding concerns what the person can actually assess and do with that information. An explanation can be available without being sufficient for the decision.
How can an organisation evaluate an AI explanation?
Check factual support, source relevance and whether users can identify important assumptions and consequences in a realistic task. Measure decision quality and review effort as well as perceived confidence.
Does a longer explanation create more trust?
There is no universal rule. The relevant question is whether the explanation helps the intended user make an appropriate judgement in that context. Length alone does not demonstrate that.