AI governance and security: the checklist every serious buyer wants to see

1. Who is accountable for this system? A named human with budget authority, not a team inbox. If you cannot name the person, you fail this question.

2. What data does the system see, and where does that data go? A data flow diagram, updated with every material change, that shows every model provider, every log destination, and every retention window.

3. What can the system do on its own? A list of every action the system takes without human review, with the rationale for each. If the list is longer than one page, expect follow-up questions.

4. What happens when something goes wrong? An incident response plan that names the on-call, the escalation path, the rollback procedure, and the customer communication template. Untested plans do not count.

Access controls. Least-privilege service accounts, scoped API keys, and a documented rotation cadence. Maps directly to SOC 2 CC6 and ISO 27001 A.9.

Logging and audit. Every prompt, every retrieved chunk, every tool call, every model response, retained for the period your legal team requires. Non-negotiable for any workflow that touches customer data.

Model provenance. Which model version answered which request. When a provider deprecates a model, you need to know what you were relying on and what changed.

Evaluation and monitoring. A held-out eval set that runs on every deploy, plus production monitoring for regression. The eval set is a governance artefact, not just an engineering artefact.

Third-party risk. A DPA with every model provider, plus a documented review of their security posture. If your model provider does not publish a security page, escalate.

If your system falls into the Act's high-risk categories (recruitment, credit scoring, critical infrastructure, education, law enforcement, among others), you have obligations around risk management, data governance, human oversight, transparency, and post-market monitoring. The obligations are non-trivial and the fines are real, up to 7% of global turnover for the most serious violations. If you are not sure whether you are in a high-risk category, assume you might be and get a written opinion from counsel. The cost of the opinion is a rounding error compared to the cost of being wrong.

Prompt injection. Untrusted content in the retrieval pipeline can override your instructions. Mitigate with input sanitisation, output validation, and never letting user-controlled text set the model's tools.

Data exfiltration through tool calls. A compromised prompt can ask the model to send data to an attacker-controlled endpoint. Mitigate with an allowlist of destinations and per-tool rate limits.

Model output as attack vector. Model output that is executed as code, SQL, or shell commands is a direct path to compromise. Mitigate by treating every model output as untrusted user input and gating it through the same validation as any other input.

Supply chain. Open-source models and libraries can be tampered with. Pin versions, verify hashes, and scan for known CVEs on every build.

Ship the first workflow with the minimum viable governance: access controls, logging, an eval set, and a named owner. That is enough to be defensible while you build the rest.

Between workflow one and workflow two, add the model provenance registry and the data flow diagram. This is the moment to codify what you learned.

By workflow three, publish an internal AI standard. One page. What we build, what we log, who signs off, what we will not do. This is the artefact that turns governance from a bottleneck into a shortcut.

For the strategy layer around this, see the AI strategy playbook for 2027. For where governance fits inside a single control plane, see one connected AI system beats twenty disconnected tools.