How do we build AI products that can operate in regulated domains?

HealthWords wanted to build a healthcare product where someone could describe their symptoms in ordinary language and leave with the right next step.

That next step might be an evidence-backed article, an appropriate self-care route or an over-the-counter product. It might be an appointment with a doctor. When a conversation revealed something more serious, the system needed to recognise that automation should stop and qualified care should take over.

A chatbot only needs to produce a plausible response. A regulated healthcare product has to decide whether it has enough information to respond at all. It has to know which question to ask next, which evidence it may use, which action it may recommend and which signal should immediately change the route.

We rebuilt the complete system

Gumtech did not wrap a general model in a medical prompt. We rebuilt HealthWords from the ground up: the clinical conversation, follow-up question flows, patient product, professional tools, content and knowledge systems, ordering and booking journeys, specialist data, model behaviour, evaluations, backend services and event infrastructure.

Doctors and pharmacists helped turn clinical judgement into structured examples the product could use. The patient journey could move from a symptom to an approved article, an allowed self-care route, an over-the-counter product, an appointment or qualified care. Red flags could change the state of the product rather than merely add a warning underneath an answer.

The result is not simply an AI that can talk about health. HealthWords became a regulated, UKCA-marked Class I medical device. That status is backed by clinical governance, quality management and documented controls across the product, not a badge added to a chatbot.

The normal way to begin an AI product is to compare models, run prompts and pick the one that gives the most impressive answer. That order works for a demo. It is backwards for a regulated product.

The first question is not, 'What can this model do?' It is, 'What is this product allowed to do?' A model can generate thousands of plausible responses to a symptom. HealthWords has a bounded set of routes, and every route needs a reason for being selected.

1. 01

   Define the required evidence

   Write down what the system must know before a route becomes available. If the evidence is incomplete, another question or a safer fallback should remain possible.

2. 02

   Define the permitted action

   Be precise about what the product may inform, recommend, book, order or hand over. A broad goal such as 'help the patient' is not an operational boundary.

3. 03

   Define the prohibited action

   State what the route must never infer, claim or complete, even when a model can produce a convincing response.

4. 04

   Define the fallback

   Decide where the journey goes when evidence is missing, contradictory or high risk. Refusal and escalation are valid product outcomes.

5. 05

   Define the record

   Retain enough context, evidence, versions and route decisions for an authorised person to reconstruct what happened.

Use models where judgement is useful, and rules where freedom is dangerous

Bounded routes do not mean the product must become a rigid decision tree. Models are useful for understanding ordinary language, choosing a useful follow-up question, retrieving relevant evidence and turning a long exchange into context a clinician can use.

But flexibility should exist inside a controlled route. It should not give the model permission to invent a medical workflow, call any tool it wants or continue after a clinical rule has found a reason to stop. The model helps the product move through defined routes. It does not own the product's authority.

Most product teams design the journey they want users to complete, then treat error states as exceptions. In a regulated product, the uncommon route may be the one that matters most.

If a conversation contains signs of a serious condition, the product needs to stop behaving like a helpful chatbot and start behaving like a controlled medical system. A generic disclaimer beneath a detailed answer does not achieve that. The model may already have given unsafe advice, and warnings shown everywhere quickly become invisible.

Build a red-flag register, not a longer prompt

Escalation is a successful outcome. If the system recognises that it does not have enough evidence, identifies a red flag and moves the user towards qualified care, it has done its job. Forcing a model to complete every conversation is not a higher automation rate. It is a refusal to admit where automation should end.

A general model knows a lot about medicine. That is not the same as knowing how HealthWords should handle a specific patient, in a specific product, under a specific set of clinical and regulatory constraints.

Doctors and pharmacists do not only know facts. They know which detail changes the route, which question should come next, when the evidence is insufficient and when an ordinary-looking symptom becomes concerning. You cannot recover all of that judgement by making the prompt longer. You have to turn it into data.

This is what human-in-the-loop looks like before a model reaches a patient. A clinician could define the next question, connect each answer to another question or assessment and make the route explicit. The product captured clinical reasoning as structured, versionable data instead of burying it in documents, prompts or one-off feedback.

Capture the decision, not only the answer

1. 01

   Situation

   Record what the patient said and the relevant context already known at that point in the conversation.

2. 02

   Missing information

   Capture what a specialist still needs before selecting a route and whether it is safe to ask for it.

3. 03

   Decision and reason

   Specify whether the system should ask, inform, recommend, stop or escalate, and which signal changed that route.

4. 04

   Unsafe alternatives

   Include plausible responses that would be wrong. Negative examples teach the system boundary more clearly than ideal answers alone.

5. 05

   Grading criteria

   Define what another model, prompt or workflow must do to pass the same example.

Clinicians created and reviewed triage data inside the tools we built for HealthWords. Their judgement defined question paths, assessments, expected outcomes and red-flag routes, then supported model development and evaluation. Gumtech also built a clinical-triage benchmark from this specialist data so system changes could be tested against the behaviours that matter in the product.

Build the benchmark before you fine-tune

If every good example is used for training, the team has no honest way to know whether the model improved. Reserve a representative evaluation set first. Include common journeys, difficult edge cases, red flags, ambiguous language and examples where the correct behaviour is to ask for more information or refuse to continue.

Some criteria can be checked automatically: route selection, approved-source use and prohibited actions. Others still need specialist review: whether the next question was clinically useful, whether escalation was proportionate and whether uncertainty was communicated without false reassurance.

A system prompt is useful. It is not a control architecture. Any instruction that matters only while a model chooses to follow it will eventually fail in a different wording, a longer conversation, a tool call or a later release.

For HealthWords, the safe behaviour comes from several layers working together. The model can interpret language and help select the next useful action, but product rules, approved knowledge, tool permissions, specialist review and escalation remain outside the model's discretion.

The handover is part of the product

A human-in-the-loop button is not enough. By the time a clinician takes over, the system may already have collected symptoms, asked follow-up questions, retrieved information and considered several routes. Losing that context forces the patient to start again and makes the professional repeat work the system has already done.

A useful handover packages the conversation, relevant answers, evidence, route state and reason for escalation into a form the receiving professional can inspect quickly. Access remains role-aware because the same operational record should not expose the same information to every user in the organisation.

Agents need narrower jobs, not broader autonomy

Agentic behaviour is valuable when a task requires several steps: gather the missing context, retrieve an approved source, prepare an appointment, create an order or assemble a handover. The mistake is to give one agent a vague goal and every available tool.

We prefer small, inspectable workflows with explicit inputs, allowed tools, completion criteria, timeouts and fallbacks. The system should know what the agent is trying to complete, what it may change and which action still requires a person or deterministic check.

The final gap between a safe prototype and a dependable product is operational. A model may pass a notebook evaluation and still fail when conversations become longer, sources change, tools time out, services are redeployed or several workflows update independently.

HealthWords is not one model call. It is a patient application, professional tooling, reviewed content, model and retrieval services, appointment and ordering workflows, access controls and records that need to move together. Gumtech built the platform as services connected through an event-driven architecture, including Kafka for service-to-service communication where durable events and decoupling were useful.

Evaluate the system, not only the answer

This is why evaluations belong in the release process. A new model, prompt or retrieval strategy may improve average answer quality and still break red-flag recall, source use or a downstream tool. The relevant suite needs to run whenever any part of the decision system changes.

A practical blueprint for regulated AI

1. 01

   Choose one consequential journey

   Do not begin with the entire product. Pick one decision where a wrong answer changes what the user does next.

2. 02

   Map every permitted route

   Define the required evidence, prohibited action, fallback and retained record for each outcome.

3. 03

   Design hazards and escalation first

   Create the red-flag register and prove the dangerous routes before optimising the happy path.

4. 04

   Bring specialists into the build

   Capture their decisions, reasons, unsafe alternatives and disagreements as versioned data rather than late-stage feedback.

5. 05

   Reserve the benchmark

   Separate evaluation examples before model adaptation and grade route behaviour as well as response quality.

6. 06

   Layer the controls

   Combine product rules, approved knowledge, constrained tools, runtime checks, human oversight and release gates.

7. 07

   Instrument the complete journey

   Trace the evidence, decisions, versions, tool calls, service events and handovers required to reconstruct an outcome.

8. 08

   Turn production failures into new tests

   Review uncertain and failed journeys, add approved examples to the evaluation set and improve the product without losing previous behaviour.