Lecture 14: When AI Systems Fail: Pipelines, Monitoring, and Documentation
Department of Data and Decision Sciences
Emory University
Source: Astera Software
When you ask an LLM a question, here’s what happens:
Each step can fail independently!
The whole process is a pipeline! 🤓
Even “What’s the weather?” touches load balancers, GPUs, and safety filters before you see an answer
Not all failures are the same:
| Failure type | What it looks like | Root cause |
|---|---|---|
| Hallucination | Confident but false content | Model limitations, missing grounding |
| Policy failure | Unsafe or inappropriate output | Weak guardrails or bad updates |
| Data drift | Outdated or weird answers | World changes faster than training |
| Pipeline failure | Slow, down, or inconsistent responses | Infrastructure, scaling, broken steps |
Spot which type of failure happened, then choose the right response. Pipeline failures are physical: overloaded GPUs, network timeouts, rate limits
| Incident | What Happened | Impact |
|---|---|---|
| Air Canada chatbot (2024) | Invented a refund policy | Company lost lawsuit |
| DPD chatbot (2024) | Swore at customers, criticised its own company | Emergency shutdown |
| Google Gemini images (2024) | Historically inaccurate diverse images | Feature paused, CEO apologised |
| Chevy chatbot (2023) | Agreed to sell car for $1 | Prompt injection attack |
| Snapchat My AI (2023) | Privacy concerns, couldn’t be removed | User backlash |
Common causes:
Source: https://x.com/ChrisJBakke/status/1736533308849443121. The whole thread is hilarious 😂
What happened (January 2024):
DPD’s response:
Blamed a system update and disabled the AI element.
What went wrong:
A system update broke the guardrails, and customers noticed first, on social media
Source: The Guardian
Pipeline failure point:
No monitoring after deployment: no automated check caught it
What happened (February 2024):
Why did this happen? Can you guess?
Source: BBC News
Pipeline failure point:
Testing didn’t catch edge cases
Data drift: real-world data stops looking like the training data. A sentiment model trained in 2020:
Three types:
| Type | What changes | Example |
|---|---|---|
| Data drift | The inputs, \(P(X)\) | New users, new devices |
| Label drift | How often each answer occurs, \(P(Y)\) | Fraud rises from 1% to 3% at holidays |
| Concept drift | What inputs mean, \(P(Y \mid X)\) | “Sick” now means “great” |
Source: Spot Intelligence
Models assume the future looks like the training data. But the world changes, users differ, outputs feed back into inputs, and shocks like COVID hit
Performance often decays, even with little drift:
Signs your AI tool is degrading:
The boiling frog problem: the change is so gradual you don’t notice
Source: James Howard
Nerf Bench: tracks whether AI models get worse after launch
Essential metrics for AI systems:
| Category | Metric | Why It Matters |
|---|---|---|
| Performance | Latency | How fast are responses? |
| Throughput | How many requests per second? | |
| Error rate | What % fail completely? | |
| Quality | Accuracy/relevance | Are answers correct? |
| Hallucination rate | How often does it make things up? | |
| User satisfaction | Thumbs up/down? | |
| Resources | Token usage | How much does each request cost? |
| GPU memory | Are we close to capacity? | |
| Business | User engagement | Are users coming back? |
As a user: you experience these as speed, accuracy, and availability!
Source: Oracle
Teams watch dashboards like this to catch problems before users notice
| Test | What it checks | Example |
|---|---|---|
| Safety | No harmful content | Refuses illegal requests? |
| Format | Output structure | Is the JSON valid? |
| Consistency | Stable core facts | Is Paris still in France? |
| Boundary | Edge cases | A 10,000-word prompt? |
| Bias | Fairness | Treats groups equally? |
| Regression | Old bugs stay fixed | Does the DPD fix still work? |
Source: Medium
Check inputs BEFORE they reach the model:
Example:
Garbage in → garbage out still applies!
Source: ApX Machine Learning
Check outputs BEFORE sending to users:
Tools:
What DPD should have done:
Source: ApX Machine Learning
ImageNet: the most influential computer vision dataset, behind 10+ years of research
Documentation lets us check consent, trace bias, match models to uses, and assign responsibility
Monitoring catches problems after deployment. Documentation prevents some before
Source: Excavating AI (great resource, check it out!)
Datasheets describe data, model cards describe models, and system cards describe whole products.
Gebru et al. (2018) borrowed the idea from electronics spec sheets:
“…every dataset be accompanied with a datasheet that documents its motivation, composition, collection process, recommended uses, and so on.”
What a datasheet answers:
Margaret Mitchell et al. (2019):
“Model cards are short documents accompanying trained ML models that provide benchmarked evaluation in a variety of conditions.”
Core sections:
System cards cover a whole product: its models, safeguards and red-team tests (example)
Source: Mitchell et al. (2019)