Healthcare AI is not software you simply buy

When a healthcare organisation says “we want to introduce artificial intelligence”, its first question should not be which platform to buy. It should ask which decision it intends to improve, for whom, with which data and under whose responsibility. This is not administrative caution. It is the difference between innovation and technological furniture.

The problem precedes the product

A serious use case describes a concrete process: the user, the available information, the output, the required timing and the action that may follow. If these elements cannot be explained without using the word algorithm, there is no project yet—only a desire. Desires become expensive very quickly when they enter procurement documents.

Quality is more than accuracy

An impressive percentage in a demonstration does not explain how a system will behave in a specific organisation. Leaders must ask which populations and data were used, how missing cases are handled, how context changes affect performance and what an error would cause. A technically accurate system can still be organisationally dangerous when placed in the wrong workflow or granted authority it does not possess.

Oversight means being able to disagree

Human oversight is not a signature added at the end. The person reviewing the output needs competence, information, time and authority to challenge it. If accepting the suggestion is easier than examining it, the human exists only in the organisational chart. A genuine human-in-the-loop process is designed, trained and measured.

Governing the full life cycle

Selection, local testing, deployment, monitoring, updating and retirement belong to the same cycle. Data changes, behaviour changes, vendors update models and users invent shortcuts. Organisations therefore need an AI inventory, identifiable ownership, quality indicators, incident recording and a stopping procedure. Governance does not slow innovation; it prevents innovation from moving blindfolded.

Competence has become an obligation

AI literacy is not one introductory course for everyone. Boards, buyers, technical teams, users and reviewers face different risks. Training must be role-based and verified through real behaviour. European enforcement and transparency provisions make waiting for the regulator to knock on the door a remarkably poor strategy.

The useful question

Before purchasing a system, I would ask the board to complete one sentence: “If this project worked, which decision would become concretely better, and how would we know?” If the answer is not measurable, understandable and attributable, the AI is not ready. The organisation probably is not either.

Questions a board should demand

Before approval, written answers are required: which population is represented in the data, which error is most costly, who watches the system after launch, how an incident is reported and who has authority to suspend it. The supplier relationship must remain intelligible too: versions, updates, subcontractors, data location and the ability to export evidence. A serious project leaves verifiable traces before, during and after adoption.

Measure what genuinely changes

Time saved and number of users are convenient indicators, but they are insufficient. Evaluation should include decision quality, equity between groups, cognitive load, corrections required, new errors introduced and user experience. Some benefits appear quickly; others take months. Monitoring therefore needs quantitative evidence, observation of real work and conversations with the people who live with the system’s consequences.

Technology does not replace responsibility. It makes it harder to hide.