An AI Audit Isn't a Technical Review. Here's What It Actually Is.
The term 'AI audit' means different things to different people. Most definitions are too narrow. Here is how we think about it, and why the scope matters more than the methodology.
When most people hear “AI audit”, they picture an engineer checking a model's accuracy. That is one useful activity, and it is nowhere near an audit. A model can score beautifully on every technical metric while the organisation around it has no idea who owns it, no record of why it was approved, no controls on who can change it, and no answer for a regulator who asks how it treats different groups of customers.
The technical review tells you whether the model works. The audit tells you whether the organisation is in control of it. Those are different questions, and the second one is the one that ends up in front of boards, insurers and regulators.
Why the narrow definition fails
Consider the AI incidents that actually make the news and the case law. Very few are stories about a model with poor accuracy. They are stories about systems nobody was monitoring, deployed for purposes nobody had signed off, using data nobody had checked the rights to, producing outcomes nobody could explain afterwards. Every one of those failures is a governance failure, and a technical review would have caught none of them.
A model can pass every technical test while the organisation around it fails every question that matters: who owns this, who approved it, who is watching it, and can anyone explain what it did?
The five domains a real audit covers
Our audit practice, aligned with ISACA's frameworks, works across five domains. The first is risk: identifying and quantifying model bias, accuracy degradation and adversarial exposure. The second is governance: whether accountability, ownership and oversight structures exist and function. The third is control: access management, versioning, change management and monitoring in production. The fourth is regulatory alignment: UK GDPR, the EU AI Act, FCA rules, NHS requirements or whatever applies to your sector. The fifth is the output that makes the other four useful: a plain-language report for leadership and a prioritised remediation plan for engineers.
The order matters less than the coverage. An audit that skips governance because the auditors were all engineers, or skips the technical layer because they were all compliance people, produces a document that reassures everyone and protects no one.
“But we only use third-party AI”
This is the most common reason organisations assume audits don't apply to them, and it is wrong in an expensive way. If your team makes decisions with an AI-powered tool, your organisation owns the outcomes, regardless of who built the model. Regulators do not accept “the vendor handles that” as an accountability structure, and neither do courts.
Auditing bought-in AI looks different from auditing your own build. The emphasis shifts to vendor due diligence, contractual controls, data flows and the human oversight wrapped around the tool. But the core question is identical: can you demonstrate that you are in control of the AI making decisions in your name?
What you should have at the end
A good audit ends with two documents. Leadership gets findings in plain language, rated by severity, with a clear statement of what each one means for the business. Engineering gets a technical annex with the evidence and a remediation roadmap ordered by priority, so the most consequential gaps close first. What nobody gets is a hundred pages of methodology that took longer to format than the fieldwork took to do.
The best time to do this is before someone else makes you. An audit commissioned by your own board is a management tool. The same exercise commissioned by a regulator after an incident is evidence. The work is identical. The circumstances are not.
