AI Governance in Practice: What Real AI Failures Teach PMO Leaders
⏱ Reading time: about 15-20 minutes | For PMO managers, portfolio leads, programme directors and transformation leaders
In November 2022, a Canadian traveller whose grandmother had just died asked Air Canada’s website chatbot about bereavement (grief) fares. The chatbot told him he could book a full-price ticket and claim the discount afterwards. That was wrong; the airline’s actual policy did not allow it. When he asked for the refund, Air Canada refused, and in the tribunal hearing it argued that the chatbot was responsible for its own statements.
In February 2024, the British Columbia Civil Resolution Tribunal rejected that argument outright. The company owned its website, so it owned everything on it, including what the chatbot said. Air Canada was ordered to pay the difference, roughly CA$800.
The money was small. The lesson was not. The chatbot itself was never the real problem. The problem was that nobody had clearly owned what the AI was allowed to say, how it was tested, and who would answer for it. That gap has a name: AI governance.
How this guide works. Instead of starting with frameworks and definitions, we follow an AI initiative through the stages a PMO already knows: the idea, the rules, the data, the build, the launch and the run. At each stage you will find a real case that went wrong, the governance lesson behind it, and the PMO move that would have helped. A running illustrative scenario, a fictional insurer building an AI claims assistant, shows how the same ideas look inside an ordinary project.
In this guide
The basics: AI governance, Cloud AI and AI operations
Stage 1, the idea: should we do this at all?
Stage 2, the rules: update policies before people improvise
Stage 3, the data: your AI inherits your history
Stage 4, the build: design for the user who tries to break it
Stage 5, the launch: compliance is not a future problem
Stage 6, the run: AI risk keeps moving after go-live
The toolkit: components of AI governance, and the failure each one prevents
The PMO’s role, and a 90-day starter plan
Further reading
The Basics: AI Governance, Cloud AI and AI Operations
What is AI governance?
AI governance is how an organisation decides which AI it will use, sets the rules for building and running it, and makes sure someone is accountable when it goes wrong. In practice, it comes down to four questions you should be able to answer for every AI system in the organisation:
- Should we do this? Is it appropriate, legal and worth the risk?
- Who owns it? A named person, not “the vendor” or “IT”.
- Is it still working? Accurate, fair and secure, this month and not just at go-live.
- Can we prove it? Approvals, test results and logs that would satisfy an auditor, a regulator or a journalist.
Air Canada could not answer the second question cleanly, and the tribunal answered it for them.
What is Cloud AI?
Most organisations do not build AI from scratch. They rent it. Cloud AI is AI delivered through cloud providers, and it reaches your organisation through four doors:
- Rented compute for teams that train or fine-tune their own models.
- AI platforms and model APIs, where developers call a large language model from their own application.
- Ready-made AI services such as document extraction, translation or speech-to-text.
- AI switched on inside software you already pay for: the copilot in your office suite, the “AI summary” button in your CRM, the smart triage in your service desk.
The fourth door is the one PMOs miss. A typical example: an administrator enables an AI assistant across the email and document suite on a Friday because it came with the licence. On Monday, it can summarise any file that each user has access to, including the HR folder that was accidentally shared with “everyone” three years ago. No project was raised, no risk was logged, and nothing was technically broken. The AI simply made an old permissions mistake easy to find.
What is Cloud AI operations?
Cloud AI operations (often called MLOps, or LLMOps for language models) is the day-to-day work of keeping AI reliable in production. On an ordinary week it looks like this: a team notices the chatbot’s answers about a new product are outdated and updates its reference documents; the finance lead asks why the AI bill tripled after a new feature launched; the vendor announces that the model version you tested will be retired in 90 days; a prompt change that improved tone quietly broke the summary format another system depends on. None of this is exotic. It is release, change, cost and capacity management, applied to something that behaves probabilistically.
What is Cloud AI governance?
Cloud AI governance is about the split of responsibility between you and the provider. The provider secures the platform and trains the base model. You remain responsible for what data you send, how outputs are used, who has access, and whether the use case is lawful and fair.
Real case: a vendor model nobody had validated locally
Epic’s sepsis prediction model was built into a health-records platform used by hundreds of US hospitals. In 2021, University of Michigan researchers tested it independently on their own patients and published the results in JAMA Internal Medicine. The model missed about two-thirds of sepsis cases while generating alerts on 18% of all hospitalised patients, a heavy source of alert fatigue. Its measured performance was well below what had been reported to customers.
Governance lesson: buying AI does not transfer accountability. A vendor’s accuracy figures describe the vendor’s data, not yours. Validate on your own population before you rely on it.
Stage 1, the Idea: Should We Do This at All?
Every AI failure in this guide started as a reasonable-sounding idea. Screen CVs faster. Catch fraud earlier. Answer customers at night. The foundation of AI governance is asking, before money is spent, what this system will decide, about whom, and what happens to those people when it is wrong.
Real case: the Dutch childcare benefits scandal
For years, the Dutch tax authority used risk profiling, including a self-learning risk model, to select childcare benefit claims for fraud checks. Holding a second nationality was used as a risk indicator. Tens of thousands of families were wrongly treated as fraudsters, and many were ordered to repay large sums, pushing some into severe financial hardship. The algorithm was one part of a wider institutional failure, but it amplified it. The Dutch government resigned over the scandal in January 2021, and the Dutch data protection authority later fined the tax administration for its discriminatory processing of nationality data.
Governance lesson: the question “what if the model is wrong about someone?” must be answered at the idea stage, together with how a person can challenge the outcome. Efficiency targets alone are not a sufficient reason to automate decisions about people.
Sort ideas by who gets hurt, not by how clever the technology is
The same model can be harmless in one use and dangerous in another. A practical triage looks like this:
| Use case | Who is affected if it is wrong | Indicative tier | What governance looks like |
|---|---|---|---|
| Summarising steering committee minutes | Internal readers | Low | Approved tool, human check before circulation |
| Internal policy assistant | Employees acting on wrong answers | Low to medium | Answers cite sources, access rights respected |
| Customer-facing chatbot | Customers, and your legal position | Medium | Scoped answers, human handover, testing against misuse |
| AI agent that sends emails or updates records | Customers, data integrity | Medium to high | Limited permissions, approval before irreversible actions |
| CV screening, credit, benefits or claims decisions | Individuals’ jobs, money and rights | High | Impact assessment, bias testing, appeal route, legal review |
Name the owner before you name the vendor
Every AI system needs a business owner who can explain in one sentence what it does and who would stand in front of a regulator if needed. If the only candidate is “the project”, the idea is not ready. Projects close; AI systems keep running.
PMO move: add three questions to your intake form. Does this involve AI (including AI features in purchased software)? Will it influence decisions about people? Who will own it after the project closes? These three questions route most requests correctly on their own.
Illustrative scenario: the insurer, week 1
A mid-sized insurer (fictional) wants an AI assistant that reads incoming motor claims and recommends “fast-track” or “investigate”. At intake, the PMO flags that “investigate” delays payouts to real customers, so this is a high-tier use case. The idea is approved on one condition: the AI recommends, a claims handler decides, and every customer flagged for investigation can ask why. The Head of Claims, not IT, is named as system owner.
Stage 2, the Rules: Update Policies Before People Improvise
Your employees are already using AI. The only question is whether they are following your rules or inventing their own. The two cases below are not about bad people. They are about capable professionals using a new tool with no clear guidance.
Real case: source code pasted into a public chatbot
In spring 2023, shortly after Samsung’s semiconductor division allowed engineers to use ChatGPT, media reported several incidents in which staff entered confidential material, including source code and internal meeting content, to get help with their work. Samsung responded by restricting the use of generative AI tools on company devices while it developed its own safeguards.
Governance lesson: “be careful” is not a policy. People need to know which tools are approved, and which kinds of data may never go into them.
Real case: the court filing with invented judgements
In 2023, in Mata v. Avianca in New York, lawyers submitted a brief citing court decisions that did not exist. They had been generated by ChatGPT, and nobody verified them before filing. When the court asked for copies, the problem unravelled. The judge sanctioned the lawyers and their firm with a US$5,000 penalty, and the story made headlines around the world.
Governance lesson: the policy has to cover outputs as well as inputs. Anything that leaves the organisation must be checked by a person who is accountable for it.
Which policies actually need updating
You rarely need a stack of new documents. You need one short AI policy, a plain-language acceptable-use guide, and targeted edits to policies you already have:
| Existing policy | The AI update, in one line | The mistake it prevents |
|---|---|---|
| Acceptable use / IT | List approved AI tools and data that must never be entered | Confidential code in a public chatbot |
| Quality and review | AI-generated content that leaves the building needs a named reviewer | Invented citations in a client or court document |
| Procurement | Ask vendors how AI features use your data, and who is notified when models change | AI switched on silently in purchased software |
| Information security | Add AI threats: manipulated prompts, deepfakes, data leakage through answers | Chatbots talked into bad promises; impersonation fraud |
| HR | Extra checks before AI is used in hiring, performance or monitoring | Discriminatory screening |
| Change management | Model, prompt and vendor-version changes are controlled changes | An “update” that changes behaviour in production |
A useful test for any AI policy: can a new joiner read it in five minutes and know what they can do on Monday morning? If it takes a lawyer to interpret, people will ignore it and improvise.
Illustrative scenario: the insurer, week 3
During discovery, the business analyst admits she has been pasting real claim descriptions into a free chatbot to draft requirements. Nobody told her not to. Rather than a disciplinary issue, the PMO treats it as a finding: the insurer publishes a one-page acceptable-use guide, provides an approved enterprise AI tool, and adds a “no customer data in unapproved tools” line to the project’s onboarding checklist.
Stage 3, the Data: Your AI Inherits Your History
An AI system learns from data about the past. If the past was unfair, incomplete or messy, the AI will faithfully reproduce it, at speed and at scale.
Real case: the recruiting model that learned to prefer men
As reported by Reuters in 2018, Amazon had built an experimental tool to rate job applicants, trained on about ten years of CVs submitted to the company. Because most of those CVs came from men, the model learned to penalise CVs that included the word “women’s” (as in “women’s chess club captain”) and downgraded graduates of two all-women’s colleges. Engineers tried to correct it but could not be confident it would not find other ways to discriminate, and the project was abandoned. Amazon said recruiters never used the tool alone to evaluate candidates.
Governance lesson: nobody programmed the bias. It came from historical data. Data review is not a technical formality; it is where fairness is won or lost.
What data safety means on a real project
- Ask where the training data came from and who is missing from it. A claims model trained only on urban customers will behave oddly with rural ones.
- Check you are allowed to use it. Data collected to process a claim was not necessarily collected to train a model. Data protection laws, including the EU GDPR, the UAE and Saudi personal data protection laws, care about purpose.
- Use less. If the model does not need names, ID numbers or dates of birth, remove them before they enter the pipeline.
- Fix permissions before connecting AI to documents. An assistant that searches company files will surface whatever each user can technically access. Old over-sharing becomes instantly visible.
- Decide what happens to prompts and outputs. Where are they stored, for how long, and who can read them? Chat logs can contain more sensitive detail than the source systems.
- Remember that data can be an attack. Documents, emails or web pages the AI reads can contain hidden instructions designed to manipulate it. This is called prompt injection, and it is a data problem as much as a security one.
Quick test before any AI assistant goes live: log in as a junior employee and ask it for senior salaries, a colleague’s performance review and the board pack. If you get answers, you have a permissions problem, not an AI problem, but the AI is what will expose it.
Illustrative scenario: the insurer, week 6
The data team finds that historic “investigate” decisions were far more common in two postcodes that happen to have large migrant communities. Nobody intended this, but a model trained on it would learn it. The PMO logs it as a high-rated risk, the steering committee approves an extra four weeks for data rebalancing and fairness testing, and postcode is removed as a model input.
Stage 4, the Build: Design for the User Who Tries to Break It
Traditional testing asks “does it do what we specified?” AI testing must also ask “what will it do when someone tries to make it misbehave?” and “what will it do after the next update?”
Real case: the $1 SUV
In December 2023, a car dealership in California added a chatbot powered by a general-purpose language model to its website. Users quickly discovered they could instruct it to agree with anything. One persuaded it to “agree” to sell a new Chevrolet Tahoe for one dollar and to describe that as a legally binding offer. Screenshots spread across social media. The dealership did not honour the “deal” and took the chatbot down.
Governance lesson: a general-purpose model placed in a narrow business role needs limits on what it can discuss and commit to, and it must be tested by people deliberately trying to misuse it.
Real case: the chatbot that turned on its own company
In January 2024, a customer of the UK parcel delivery firm DPD, frustrated at not finding his parcel, got the company’s chatbot to swear and to write a poem criticising DPD. DPD said an error had occurred after a system update, and it disabled the AI element while it was fixed.
Governance lesson: a model or configuration update is a release. It needs regression testing, including behaviour testing, before it reaches customers.
Design decisions that should pass through a gate
- Is AI the right tool? Sometimes a rules engine, a better form or a search box does the job with less risk.
- Scope: what the AI may and may not talk about, decide or commit to.
- Human in the loop: who reviews outputs, and do they have the time and authority to disagree?
- Fallback: how does the process continue if the AI is switched off tomorrow?
- Testing: accuracy on your own data, fairness across groups, and adversarial testing (often called red-teaming). This extends a normal test strategy rather than replacing it; 25 Critical Components of a Testing Strategy is a good base to build on.
- Release control: model versions, prompts and reference documents are configuration items with owners and change records. See The Complete Release Management Guide for the controls you can reuse.
Illustrative scenario: the insurer, week 10
Before user acceptance testing, the PMO invites three claims handlers to spend a day trying to break the assistant. They find it can be nudged into recommending fast-track for a claim with obvious fraud markers if the description is written emotionally enough. The fix goes in, and “adversarial test pass” becomes a formal go-live criterion alongside functional UAT.
Stage 5, the Launch: Compliance Is Not a Future Problem
Many teams assume AI regulation is something coming later. In reality, regulators and courts are already applying existing law to AI, and AI-specific laws are now arriving on top.
Existing law already applies
- Consumer law: Air Canada was held to what its chatbot promised, under ordinary principles of misrepresentation.
- Employment law: in 2023, iTutorGroup settled a US Equal Employment Opportunity Commission case for US$365,000 after its hiring software automatically rejected older applicants (women aged 55 and over, men aged 60 and over).
- Data protection law: in March 2023, Italy’s data protection authority temporarily restricted ChatGPT in Italy over privacy concerns. Access returned about a month later after OpenAI made changes.
- Consumer protection regulators: in December 2023, the US Federal Trade Commission banned the pharmacy chain Rite Aid from using facial recognition for surveillance for five years, after its system falsely flagged customers as suspected shoplifters.
AI-specific law: the EU AI Act in practical terms
The EU AI Act sorts AI by what it is used for. A few examples make it concrete:
- Banned: for example, social scoring of people, and certain manipulative or exploitative uses.
- High-risk: AI used to screen job applicants, assess creditworthiness, grade students or manage critical infrastructure. These need risk management, data quality controls, documentation, logging, human oversight and accuracy testing.
- Transparency duties: a customer chatbot must make clear it is an AI, and certain AI-generated content must be labelled.
- Everything else: a meeting summariser or a spam filter carries no specific obligations, beyond general law and the requirement that staff using AI have adequate AI literacy.
It also reaches beyond Europe. A Gulf-based company that uses AI to screen candidates for roles in Germany, or sells an AI-enabled product into France, can fall within scope.
Timeline update (2026): in June 2026, the EU approved the “Digital Omnibus on AI”, which moved high-risk obligations for stand-alone systems (such as hiring and credit scoring) from 2 August 2026 to 2 December 2027, and for AI built into regulated products to 2 August 2028. The obligations were postponed, not removed. If your portfolio includes an AI hiring or lending project, it will likely go live under these rules, so design for them now.
Illustrative scenario: the insurer, week 14
Legal review confirms the claims assistant must tell customers when AI has been involved in routing their claim, and that any customer flagged for investigation can ask for a human review. The PMO adds both to the release scope and makes the customer notice wording a go-live dependency owned by the compliance lead.
Stage 6, the Run: AI Risk Keeps Moving After Go-Live
With traditional software, the risk profile at go-live is roughly the risk profile a year later. With AI, it is not. The world changes, the data changes, the vendor changes the model, and attackers find new ways in. AI risk management follows the process your PMO already runs, identify, assess, treat, monitor and report, but monitoring carries far more weight. If you already use a structured approach like the one in Risk Management Process for a Transformation Program, extend it rather than building a separate AI process.
Real case: when the forecast met a moving market
Zillow Offers used algorithmic home-price forecasts to buy houses, renovate them and resell them. In late 2021, as the housing market swung sharply, Zillow found it had been buying homes at prices above what it could sell them for. According to its 2021 annual report, the company recorded an inventory write-down of about US$408 million and decided to wind down the business, with roughly a quarter of its workforce affected.
Governance lesson: a model that works in a stable environment can fail quickly when conditions change. Set limits on how much the business will stake on model outputs, and define in advance the signals that trigger a pause.
Real case: AI used against you
In early 2024, an employee in the Hong Kong office of the engineering firm Arup joined a video call with what appeared to be the company’s CFO and several colleagues. All of them were AI-generated deepfakes. Following their instructions, the employee made transfers totalling around HK$200 million (about US$25 million). Arup later confirmed the incident.
Governance lesson: AI risk is not only about the AI you deploy. Payment approvals, identity checks and executive instructions all need controls that assume voices and faces can be faked.
Add these fields to your risk register for every AI system
- Risk tier and the people affected if it is wrong.
- Performance thresholds that trigger review (for example, accuracy below 90% or complaint rate above a set level).
- Human oversight check: what percentage of AI recommendations do reviewers override? Near-zero overrides can mean reviewers are rubber-stamping.
- Vendor dependency: model version, retirement date, and your exit option.
- Kill switch: who can turn it off, how quickly, and what the manual fallback is.
- Next review date, because AI risks need periodic reassessment, not a one-off check at initiation.
Illustrative scenario: the insurer, month 5
Three months after go-live, the monthly dashboard shows claims handlers accept 99% of the AI’s recommendations, in an average of eight seconds each. On paper, there is human oversight; in practice, there is none. The PMO raises it at the steering committee, and the insurer introduces random sampling of accepted recommendations and reviews handler workload. Two months later, a regional storm shifts the claims mix and the model’s fast-track accuracy drops. The threshold alert fires, and the model is paused for that claim type within a day.
The Toolkit: Components of AI Governance, and the Failure Each One Prevents
Pulling the stages together, here are the building blocks of an AI governance programme. Rather than listing them in the abstract, each one is paired with the kind of failure it exists to prevent.
| Component | What it is in practice | The failure it prevents |
|---|---|---|
| Named ownership | A business owner for every AI system, written into a RACI | “The chatbot is responsible for itself” (Air Canada) |
| AI inventory | One register of every AI system, built, bought or switched on | AI features enabled inside SaaS with no oversight |
| Intake and impact assessment | A single front door with risk-based triage | Automated decisions about people with no route to challenge (Dutch benefits) |
| Acceptable-use policy | Approved tools, forbidden data, mandatory review of outputs | Leaked source code (Samsung), invented citations (Mata v. Avianca) |
| Data governance | Provenance, permitted use, minimisation and fairness review | Models that learn historic bias (Amazon recruiting) |
| Adversarial testing | Deliberate attempts to misuse the system before launch | Chatbots talked into absurd commitments ($1 SUV) |
| Change and release control | Model, prompt and vendor updates treated as releases | Behaviour changing after an update (DPD) |
| Vendor oversight | Local validation, contract terms, notice of model changes | Trusting vendor performance claims (Epic sepsis model) |
| Compliance mapping | Each system checked against the laws that apply to it | Discriminatory hiring software (iTutorGroup) |
| Monitoring and kill switch | Thresholds, alerts, and authority to pause | Scaling a model as conditions shift (Zillow Offers) |
| AI-aware security | Controls that assume voices, faces and documents can be faked | Deepfake-enabled fraud (Arup) |
| AI literacy | Role-based training for leaders, builders, reviewers and users | Nearly every case above |
The PMO’s Role, and a 90-Day Starter Plan
Look back at the cases. Very few failed because of a missing algorithm. They failed because of a missing owner, a missing gate, a missing test, or a missing check after go-live. Those are PMO disciplines.
- Own the front door. AI screening in demand intake is the single most effective control a PMO can add. See Demand Management Guide: 9 Governance Pillars.
- Make AI visible in the portfolio. Tag AI initiatives and report their risk tier and monitoring status next to cost and schedule. See A Formal Guide to Portfolio Management.
- Put evidence into stage gates. Impact assessment, adversarial test results and a monitoring plan become gate criteria, not optional attachments.
- Keep AI tied to the transformation roadmap, so it does not become a scatter of pilots. See A Beginner’s Guide to Digital Transformation.
- Build your own AI delivery skills. Governing AI projects is easier when you understand how they are delivered. Upgrade Yourself to an AI Project Manager covers the skills shift in more detail.
A 90-day starter plan
Days 1 to 30, find out what you already have. Survey teams on which AI tools they use, including free ones. Ask IT which AI features are enabled in existing software. Build a first AI inventory, even if it is a simple spreadsheet. Publish a one-page interim acceptable-use guide.
Days 31 to 60, put the gates in place. Add the three AI questions to intake. Agree risk tiers using the table in Stage 1. Name owners for every system in the inventory. Set up a small governance group with legal, security, data, business and PMO representation.
Days 61 to 90, test the riskiest thing you run. Pick the highest-tier AI system and run it through this guide: validate it on your own data, try to break it, check its permissions, and define its thresholds and kill switch. Report what you found to leadership. That first report usually secures the sponsorship for everything else.
Conclusion
An airline bound by its chatbot. Engineers leaking code while trying to work faster. A recruiting model that learned bias from history. A forecasting model that failed when the market moved. A finance employee fooled by a fake CFO. None of these organisations lacked technical talent. What they lacked, at the critical moment, was a clear owner, a gate, a test or a trigger.
That is good news for PMO professionals. AI governance does not require you to become a data scientist. It requires you to apply the disciplines you already have, intake, ownership, gates, risk, release and benefits tracking, to a technology that keeps changing after go-live. Start with the inventory, fix the front door, and test the riskiest system you run. The rest follows.
Further reading
- EU AI Act explorer: searchable text and timelines of the EU’s AI law.
- NIST AI Risk Management Framework: a free, widely used voluntary framework for AI risk.
- ISO/IEC 42001: the certifiable management-system standard for AI.
- OECD AI Principles: the values many national AI policies are built on.
- IAPP AIGP certification: a professional credential for those who want to specialise in AI governance.
This article is for general educational purposes and does not constitute legal advice. Real cases are summarised from public reporting, court and regulatory decisions, and company filings; figures are as reported. The insurer scenario is fictional and included for illustration only. Regulatory dates reflect information available as of September 2026 and may change.