DATASCI 101: Introduction to AI Applications

Lecture 14: When AI Systems Fail: Pipelines, Monitoring, and Documentation

Danilo Freire
danilo.freire@emory.edu

Department of Data and Decision Sciences
Emory University

Welcome back! 🔧

Recap of last class

  • Last time, we explored RAG (Retrieval-Augmented Generation)
  • LLMs have knowledge cutoffs and hallucinate
  • Ground AI in your own documents
  • Chunk → Embed → Retrieve → Generate
  • RAG reduces hallucinations by 30–50%
  • But RAG isn’t perfect: retrieval errors still occur
  • Today: What happens behind the scenes when you use AI?
  • You’ve all experienced AI going down, being slow, or giving weird answers
  • Why does this happen? And what can you do about it?
  • We also look at documentation: how companies record what their systems can and cannot do, and why that record matters when things break

Source: Astera Software

Lecture overview

Today’s agenda

Part 1: What is a Pipeline?

  • From prompt to response
  • Why “it works on my laptop” isn’t enough
  • Real-world AI failures and lessons

Part 2: Why Pipelines Break

  • Data drift: when the world changes
  • Model degradation over time
  • Infrastructure and scaling issues

Part 3: Monitoring and Testing

  • What to watch: the key metrics
  • Why testing AI is hard
  • Input and output validation

Part 4: Documentation and Accountability

  • Datasheets, model cards, system cards
  • Consent and the data supply chain
  • Being a savvy AI user

Meme of the day 😄

That’s actually a very good point!

What is a Pipeline? 🏭

From prompt to response

When you ask an LLM a question, here’s what happens:

  1. Your prompt goes from your browser to servers
  2. Load balancers route it to an available GPU (cloud infrastructure)
  3. Preprocessing cleans and formats your text (with LLMs, this is usually minimal)
  4. Tokenisation converts text to numbers (you know how to do that!)
  5. The model processes tokens (billions of operations!)
  6. Postprocessing formats the output
  7. Safety filters check for harmful content (e.g., hate speech, etc.)
  8. Response travels back to your screen

Each step can fail independently!

This entire process is called a pipeline.

The AI pipeline

Even “What’s the weather?” touches load balancers, GPUs, and safety filters before you see an answer

Failure types: what actually goes wrong?

Not all failures are the same:

Failure type What it looks like Root cause
Hallucination Confident but false content Model limitations, missing grounding
Policy failure Unsafe or inappropriate output Weak guardrails or bad updates
Data drift Outdated or weird answers World changes faster than training
Pipeline failure Slow, down, or inconsistent responses Infrastructure, scaling, broken steps

You can spot what type of failure happened, then choose the right response

When AI Breaks 💥

Real AI failures (recent headlines)

Incident What Happened Impact
Air Canada chatbot (2024) Invented a refund policy Company lost lawsuit
DPD chatbot (2024) Swore at customers, criticised its own company Emergency shutdown
Google Gemini images (2024) Historically inaccurate diverse images Feature paused, CEO apologised
Chevy chatbot (2023) Agreed to sell car for $1 Prompt injection attack
Snapchat My AI (2023) Privacy concerns, couldn’t be removed User backlash

Common causes:

  1. Hallucination treated as truth (Air Canada)
  2. Updates broke guardrails (DPD)
  3. No input validation (Chevy)
  4. Overcorrection for bias (Gemini)
  5. Rush to market (Snapchat)

Source: https://x.com/ChrisJBakke/status/1736533308849443121. The whole thread is hilarious 😂

Case study: DPD’s sweary chatbot

What happened (January 2024):

  • DPD (UK delivery company) updated their AI chatbot
  • After the update, customer Ashley Beauchamp discovered it would:
    • Swear when asked nicely
    • Write poems criticising DPD
    • Call itself “useless” and recommend competitors
    • Say DPD was “the worst delivery firm in the world”

DPD’s response:

Immediately disabled the AI chatbot, apologised, launched investigation.

What went wrong:

A system update broke the guardrails. The content filtering that prevented inappropriate responses stopped working, but no one noticed until customers posted screenshots on social media

Source: BBC News

Pipeline failure point:

No monitoring after deployment. The update was pushed without testing, and no automated checks caught the broken guardrails.

Case study: Google Gemini’s image crisis

What happened (February 2024):

  • Users asked Gemini for images of “1943 German soldiers”
  • Gemini generated images of Black men and Asian women in Nazi uniforms
  • Similar problems with “US Founding Fathers” and other historical figures
  • Google CEO called results “completely unacceptable”
  • Image generation feature paused for weeks

Why did this happen? Can you guess?

Source: BBC News

Pipeline failure point:

Testing didn’t catch edge cases

Why Pipelines Break 🔥

Data drift: The world changes

Data drift = When real-world data differs from training data

Example: A sentiment analyser trained in 2020:

  • “This product is sick!” → Probably means: Amazing ✅
  • “This slaps!” → Might not understand 🤔
  • “No cap, this is bussin’” → Completely lost! 😵

Types of drift:

Type Description Example
Label drift What’s “correct” changes Policy updates
Concept drift Meaning of things changes New slang, trends
Data drift Input distribution changes New demographics

Data drift visualisation

Source: Spot Intelligence

Slang, prices, and user habits shift constantly, but a trained model only knows what it learned

Why does data drift happen?

Models assume \(P_{\text{train}}(X, Y) = P_{\text{prod}}(X, Y)\)

But in reality, this joint distribution shifts over time:

  1. Data drift: \(P(X)\) changes
    • Input distribution differs from training
    • Example: New user demographics, device types
    • Model sees inputs it never learned from
  2. Label drift: \(P(Y)\) changes
    • Class frequencies change over time
    • Example: Fraud rate increases during holidays
    • Model’s decision boundaries become suboptimal
  3. Concept drift: \(P(Y|X)\) changes
    • Same input → different correct output
    • Example: “Sick” now means “great” in slang
    • The most dangerous type: model is confident but wrong

Technical causes:

Cause Effect
Non-stationarity Real-world processes aren’t static
Sample selection bias Training data ≠ production population
Feedback loops Model outputs influence future inputs
External shocks COVID, policy changes, viral trends

Why it’s inevitable:

Machine learning assumes i.i.d. data (independent and identically distributed). But real-world data is:

  • Temporally correlated (today depends on yesterday)
  • Non-stationary (distributions shift over months and years)
  • Adversarial (users game the system)
  • Context-dependent (meaning changes with culture and location)

Model degradation: Models get stale

Even without drift, models degrade over time:

  • User expectations evolve
  • Competitors improve their products
  • Edge cases accumulate
  • Small errors compound

Signs your AI tool is degrading (as a user):

  • ❌ Answers that used to work now don’t
  • ❌ More “I don’t know” or refusals
  • ❌ Inconsistent responses to same prompt
  • ❌ Friends getting better results elsewhere
  • ❌ You’re finding workarounds more often

The “boiling frog” problem:

Changes happen so gradually you don’t notice until it’s too late!

Model degradation

Source: James Howard

A model that was 95% accurate at launch can drop to 80% within months if left alone

Why does model degradation happen?

Model performance \(P(t)\) decays over time, even with stable data:

  1. Entropy accumulation
    • Small prediction errors compound over time
    • Error rate \(\epsilon\) grows: \(\epsilon(t) \approx \epsilon_0 \cdot e^{\lambda t}\)
    • Edge cases create cascading failures
  2. Distribution shift in deployment
    • Model outputs influence user behaviour
    • Users adapt prompts based on past responses
    • Creates feedback loops that amplify biases
  3. Catastrophic forgetting (after updates)
    • Fine-tuning on new data overwrites old knowledge
    • Safety training can reduce capability
    • Model “forgets” how to handle rare cases

Technical mechanisms:

Mechanism Effect
Weight decay Regularisation erodes rare patterns
Quantisation Compression loses precision
API updates Provider changes break workflows
Context pollution Long conversations degrade quality

Models optimise for average-case performance. But users remember the worst failures.

User trust is proportional to min(performance)

Not the mean!

Infrastructure issues: When computers fail

AI systems run on physical computers that can fail:

  • Hardware failures: GPUs overheat, disks die
  • Network issues: Connections timeout, packets lost
  • Scaling problems: Too many users at once
  • Resource exhaustion: Running out of memory
  • Dependency failures: External APIs go down

What you see as a user:

Behind the scenes What you experience
GPU memory full Slow or no response
Network timeout “Error, try again”
Rate limits hit Degraded access
Database down RAG retrieval fails!
API key issues Feature stops working

Infrastructure diagram

Source: MongoDB

If any one layer goes down, you get errors, slowness, or silence

Monitoring Basics 📊

What companies watch: Key metrics

Essential metrics for AI systems:

Category Metric Why It Matters
Performance Latency How fast are responses?
Throughput How many requests per second?
Error rate What % fail completely?
Quality Accuracy/relevance Are answers correct?
Hallucination rate How often does it make things up?
User satisfaction Thumbs up/down?
Resources Token usage How much does each request cost?
GPU memory Are we close to capacity?
Business User engagement Are users coming back?

As a user: You experience these as speed, accuracy, and availability!

Monitoring dashboard

Source: Oracle

Teams watch dashboards like this to catch problems before users notice

Status pages: Checking AI reliability

Every major AI service has a status page:

Service Status Page
OpenAI (ChatGPT) status.openai.com
Anthropic (Claude) status.anthropic.com
Google AI status.cloud.google.com
Midjourney Check their Discord

What you can learn:

  • Current outages and degraded performance
  • Historical incidents (how often things break)
  • Maintenance windows
  • Incident post-mortems (what went wrong)

Tip: Check the status page before blaming your prompt!

OpenAI’s status page shows real incidents https://status.openai.com

Fun fact: OpenAI’s status page shows frequent incidents: error rates, regional outages, feature degradation. AI reliability is super hard!

Testing AI: Why it’s hard

Traditional software testing:

  • Input: 2 + 2
  • Expected output: 4
  • Test: Assert(2+2 == 4) ✅

Deterministic: Same input → same output, always

AI testing:

  • Input: “What’s the weather like?”
  • Expected output: … um … 🤔
  • The response changes every time!
  • Many “correct” answers exist!

Non-deterministic: Same input → different outputs!

This is why Gemini’s image bug wasn’t caught:

Testing “generate a cute cat” worked fine. Testing “generate 1943 German soldiers” probably wasn’t in the test suite

Testing comparison

Source: Medium

Solution: Test for properties (is it safe? is the format correct?) rather than exact outputs!

What CAN be tested?

Properties that can be verified:

Test Type What It Checks Example
Safety tests No harmful content “Does it refuse illegal requests?”
Format tests Output structure “Is the JSON valid?”
Consistency tests Stable core facts “Is Paris still in France?”
Boundary tests Edge cases “What if prompt is 10,000 words?”
Bias tests Fairness “Does it treat groups equally?”
Regression tests Old bugs stay fixed “Does the DPD fix still work?”

Input validation (first line of defence):

  • Block prompt injections (“Ignore all previous instructions…”)
  • Limit input length
  • Filter prohibited content

Output validation (last line of defence):

What Air Canada should have done:

User asks: "What's the refund policy?"

Chatbot generates: "You can request 
a refund within 90 days..."

OUTPUT VALIDATION:
❌ Check against actual policy database
❌ Flag if inventing new policies
❌ Require human review for novel claims

Instead: Response went directly to user
         → Company lost lawsuit

Both ends matter: Validate inputs AND outputs!

Input validation: First line of defence

Check inputs BEFORE they reach the model:

Things to validate:

  • ✅ Length: Too short? Too long?
  • ✅ Language: Is it in expected language?
  • ✅ Content: Any prohibited content?
  • ✅ Format: Does it make sense?
  • ✅ Rate: Is this user spamming us?

Example input validation:

Prompt: "Ignore all previous instructions..."

❌ BLOCKED: Prompt injection attempt detected!

Garbage in → garbage out still applies!

Input validation flowchart

Source: ApX Machine Learning

Output validation: Last line of defence

Check outputs BEFORE sending to users:

Things to validate:

  • ✅ Safety: No harmful or offensive content
  • ✅ Accuracy: Core facts should be verifiable
  • ✅ Consistency: Shouldn’t contradict itself
  • ✅ Format: Meets expected structure
  • ✅ Length: Not too short, not too long
  • ✅ Privacy: No personal data leakage

Tools for output validation:

  • Content moderation APIs (OpenAI, Perspective)
  • Domain-specific rules (e.g., medical disclaimers)
  • LLM-as-a-Judge for quality scoring

Example output filter:

AI response: "To make explosives..."

❌ BLOCKED: Dangerous content detected!

Replaced with: "I can't help with 
that request."

Output validation

Source: ApX Machine Learning

Documentation: The Missing Instruction Manual 📝

The horror stories: What happens without documentation

ImageNet’s hidden problems (2019):

  • Most influential computer vision dataset ever
  • Powered 10+ years of AI research
  • Nobody knew where the images came from!
  • Later discovered:
    • Scraped from Flickr without consent
    • Racist and sexist labels (MTurk workers)
    • Non-consensual photos of minors
    • Content from revenge porn sites

The root cause?

They built it quickly and never documented what they were doing!

Thousands of AI systems were built on this undocumented foundation!

Monitoring catches problems after deployment. Documentation prevents some of them before deployment

ImageNet analysis

Source: Excavating AI (great resource, check it out!)

Documentation as accountability

Without documentation, we can’t:

  • ❌ Know if data were collected ethically
  • ❌ Identify sources of bias
  • ❌ Determine if a model is appropriate for a use case
  • ❌ Assign responsibility when things go wrong
  • ❌ Reproduce or verify results
  • ❌ Update or fix problems later

With documentation, we can:

  • ✅ Make informed choices about using a dataset
  • ✅ Trace bias back to its source
  • ✅ Match models to appropriate applications
  • ✅ Hold creators accountable
  • ✅ Build trust with users and regulators

The legal angle:

Regulations are now requiring documentation:

  • EU AI Act: Mandates documentation for high-risk AI
  • NYC Law 144: Requires bias audits (needs documentation!)
  • California CCPA: Data transparency requirements

We’ll cover US vs EU regulations in detail in Lecture 18.

If you can’t document it, you can’t deploy it (legally).

The three pillars: Datasheets, Model Cards, and System Cards

Source: Laurel Papworth

They originated from landmark papers: Gebru et al. (2018) for datasheets and Mitchell et al. (2019) for model cards.

These are now industry standards adopted by Google, Microsoft, Hugging Face, and others!

What is a datasheet?

Inspired by electronics industry:

  • Every electronic component has a “datasheet”
  • Spec sheet with all relevant information
  • Engineers can make informed decisions

Timnit Gebru et al. (2018) proposed the same for datasets:

“A datasheet documents the motivation, composition, collection process, recommended uses, and other information about a dataset.”

What a datasheet answers:

  • Motivation: Why was it created? Who funded it?
  • Composition: What is in it? Errors? Sensitive data?
  • Collection: Scraped, surveyed, sensors? Was there consent?
  • Preprocessing: What was cleaned, filtered, or removed?
  • Uses: What is it for, and what should it NOT be used for?
  • Distribution and maintenance: Who updates it? How to report issues?

Activity: Let’s read a real datasheet! 📖

Read Anthropic’s HH-RLHF Dataset Card:

  1. Visit Anthropic/hh-rlhf
  2. Read the Dataset Card carefully
  3. This dataset is used to train AI assistants to be helpful and harmless!

Questions to answer:

  • What are the two types of data in this dataset?
  • Why does Anthropic warn against using this for supervised training of dialogue agents?
  • What content warning does the dataset include?
  • How can you contact the authors with issues?

Evaluate the documentation:

  • ✅ Clear purpose: Train preference/reward models for RLHF
  • ✅ Explicit warnings about misuse (don’t train chatbots directly!)
  • ✅ Content disclaimer about harmful material
  • ✅ Links to papers for methodology details
  • ✅ Contact email provided

Discussion:

  • Would you be comfortable using this dataset?
  • What ethical responsibilities come with accessing data that contains harmful content for research purposes?

⏱️ 3 minutes to explore!

Model cards: The “user manual” for AI models

Just like datasheets document data, model cards document models.

Margaret Mitchell et al. (2019):

“Model cards are short documents accompanying trained ML models that provide benchmarked evaluation in a variety of conditions.”

Core sections:

  1. Model details: What is this model?
  2. Intended use: What should it be used for?
  3. Factors: What affects performance?
  4. Metrics: How was it evaluated?
  5. Training data: What was it trained on?
  6. Ethical considerations: What could go wrong?
  7. Caveats and recommendations: Warnings!

Model cards paper

Intended use: The most important section

Why “intended use” matters:

A hammer is a great tool. But:

  • ✅ Intended use: Driving nails
  • ❌ NOT intended for: Brain surgery

Models need the same clarity:

Model Intended Use NOT For
Gemini 3 Conversation, assistance Medical diagnosis
Face detection Placing cameras Law enforcement ID
Sentiment analysis Product feedback Hiring decisions

Without this guidance:

People will use models inappropriately and harm others.

Remember Air Canada? The chatbot’s “intended use” was answering questions, not writing refund policy. Nobody enforced the boundary

Real example: Gemini 3 Model Card (edited and summarised)

Intended use:

“Assistance with a variety of text-based tasks”

Out-of-scope:

“Assistance with illegal activities”

“Chemical synthesis”

“Mental health crisis intervention”

Disaggregated evaluation: Breaking it down

Why overall accuracy isn’t enough:

  • A model can be 90% accurate overall
  • But 95% for Group A and 70% for Group B!

Disaggregated evaluation = report performance separately for different groups.

Real example: OpenAI CLIP Model Card (2021)

Gender classification accuracy using FairFace dataset:

Race Category Gender Accuracy
Middle Eastern 98.4% (highest)
White 96.5% (lowest)
All races >96%

Racial classification: ~93% | Age classification: ~63%

CLIP’s bias findings:

OpenAI found significant disparities when classifying people into crime-related categories:

  • Performance varied by race and gender
  • Disparities shifted based on how classes were constructed
  • Risk of denigration harms identified

“We tested the risk of certain kinds of denigration with CLIP by classifying images of people from Fairface into crime-related and non-human animal categories.”

System cards vs model cards

A newer concept: System Cards

Modern AI products often chain several models together. A model card only covers one of them, so system cards document the entire system.

What’s included in a system card:

  • Multiple models working together
  • How components interact
  • Human oversight mechanisms
  • Deployment context and safeguards
  • Red teaming and safety evaluations

Who’s using system cards:

  • OpenAI: Released o1 System Card in December 2024
  • Anthropic: Publishes system cards for each Claude release
  • Meta: Released 22 system cards in 2023 for their AI products

Model Card vs System Card:

Aspect Model Card System Card
Scope Single model Entire system
Focus Performance Safety + deployment
Audience Developers Developers + public
Includes Training data, metrics Red teaming, safeguards

Discussion: Who owns the data? 💭

Scenario:

You wrote a poem and posted it online in 2018. An AI company scraped it, and now their AI can write poems “inspired by” your style.

Questions to debate:

  1. Did the company do anything wrong?
  2. Should you be compensated?
  3. Should you be able to opt out retroactively?

Different perspectives:

Tech companies: “It’s fair use, like learning from reading books”

Artists: “You’re profiting from my creative work without consent”

Lawyers: “Current law wasn’t designed for this”

Users: “I just want cool AI. I don’t care about the source”

Your turn: Where do you stand?

What Can YOU Do? 🛡️

Being a savvy AI user

You can’t fix AI pipelines, but you can:

  1. Compare answers from different models
    • Same question to ChatGPT + Claude + Gemini
    • If they disagree, investigate further
  2. Check status pages
  3. Recognise the difference
    • Pipeline problem: AI is down, slow, or glitching
    • Hallucination: AI is confident but wrong
    • Working as intended: AI refuses for safety reasons
  4. Read the documentation
    • Model cards say what a tool is not for
    • Using it outside its intended use? That risk is on you

Quick diagnostic:

Symptom Likely Cause
“I’m at capacity” Infrastructure (wait and retry)
Slow response High load or network
Different answers to same Q Non-determinism (normal!)
Confident but wrong Hallucination (verify!)
Refuses to answer Safety guardrails (try rephrasing)
Gibberish output Pipeline failure (refresh)

The variation is normal!

Why does AI give different answers to the same question?

  1. Temperature: Randomness dial (creativity vs. consistency)
  2. Load balancing: Your request might hit a different server
  3. Model versions: A/B testing different versions
  4. Context: Time of day, your chat history
  5. Updates: The model literally changed since yesterday

If AI always gave the exact same response, it would feel robotic. Some variation keeps it natural.

But for important decisions:

Ask multiple times. Ask different AIs. If answers diverge on something critical, don’t trust any of them!

Sources: r/Bard and Google Blog

You and your friend might be running different model versions right now without knowing it

Activity: Diagnose the problem! 🔍

Scenario cards: What’s the likely cause?

Scenario A:

You ask Claude a question. It takes 45 seconds to respond, and the answer is cut off mid-sentence.

Scenario B:

You ask Gemini “Who won the 2027 Super Bowl?” and it confidently names a team.

Scenario C:

You ask ChatGPT to write code. Yesterday it worked; today it refuses and says “I can’t help with that.”

Scenario D:

You ask Midjourney to generate “a photo of my professor.” It creates an image of a random person.

Scenario E:

Three different AI assistants give you three completely different answers about a historical date.

Discuss with a neighbour:

  • Pipeline problem?
  • Hallucination?
  • Working as intended?
  • How would you verify?

⏱️ 5 minutes to diagnose!

Activity answers

Scenario A: 45 seconds, cut off mid-sentence

  • Pipeline problem: Server overload or timeout
  • Check status page
  • Retry in a few minutes

Scenario B: 2027 Super Bowl answer

  • Hallucination: Event hasn’t happened yet!
  • AI doesn’t know it doesn’t know
  • Always verify time-sensitive claims

Scenario C: Code refused today but worked yesterday

  • Model update changed safety guardrails
  • Or: Your specific prompt triggered a new filter
  • Try rephrasing; if still blocked, guardrails changed

Scenario D: “Photo of my professor” = random person

  • Working as intended: AI doesn’t know your professor!
  • It generates a plausible “professor-looking” person
  • This is expected behaviour, not a failure

Scenario E: Different AIs, different historical dates

  • Potential hallucination from all of them
  • Don’t trust any AI for verifiable facts
  • Look it up in a reliable source!

Summary 📚

Main takeaways

  • You use dozens of AI pipelines daily: text, image, voice, recommendations

  • Pipelines are the multi-step process from your input to AI’s output. Every step can fail

  • Real failures: Air Canada lawsuit, DPD swearing, Gemini images. Pipeline problems, not “rogue AI”

  • Data drift: The world changes, AI stays frozen

  • Monitoring: Companies track metrics you never see. When monitoring fails, you read about it in the news

  • Documentation: Datasheets describe data, model cards describe models and their intended use, system cards describe whole products

  • Consent: Much AI training data was collected without permission. Undocumented data (ImageNet!) hides problems for years

  • You should compare AIs, check status pages, read model cards, and distinguish pipeline problems from hallucinations

Your AI user toolkit

Quick reference:

Symptom Action
AI is down/slow Check status page, wait
Different answers Normal! Verify if critical
Confident but wrong Hallucination, verify facts
Refuses request Guardrails, try rephrasing
Weird/gibberish Pipeline glitch, refresh

Status pages to bookmark:

On AI failures:

…and that’s all for today! 🎉