DATASCI 101: Introduction to AI Applications

Lecture 17: Types of Bias and How They Arise

Danilo Freire
danilo.freire@emory.edu

Department of Data and Decision Sciences
Emory University

Welcome back! ⚖️

Recap of last class

From setup to bias

  • Last time: the four layers of context: instructions, knowledge, memory, tools
  • A prompt is one request; a setup survives the tab closing
  • For agents the instructions become a file: CLAUDE.md, AGENTS.md, SKILL.md
  • Memory accumulates on its own, so read it, edit it, use incognito
  • MCP and connectors: least access, read-only by default, confirm every write
  • Today: types of bias, how they arise, and whose values AI encodes

Source: Anthropic

Lecture overview

Today’s agenda

Part 1: The big picture

  • What is bias? (It’s complicated!)
  • The mirror problem: AI reflects us
  • Real-world harms: who gets hurt?

Part 2: Types of bias

  • Historical bias: the past baked in
  • Representation bias: who’s missing?
  • Measurement bias: bad proxies
  • Aggregation, evaluation and deployment bias

Part 3: Applying the framework

  • Bias through the AI lifecycle
  • Feedback loops: bias that amplifies itself
  • Activity: spot the bias!

Part 4: The hard questions

  • The impossibility theorem: can we be fair?
  • Fairness tradeoffs: uncomfortable choices
  • Is AI bias fixable?

Neural network of the day!

https://gizmodo.com/a-dish-of-neurons-playing-doom-is-the-wildest-thing-ive-seen-in-ages-2000727674

Cortical Labs

Source: Cortical Labs

Well, let’s get back to our lecture… 😅

A story to start with

Robert Julian-Borchak Williams, Detroit, 2020

  • Arrested in front of his family, accused of stealing watches
  • Held for 30 hours in jail, then released: wrong person

What happened?

  • Facial recognition matched grainy shop footage to his licence photo, and was wrong
  • Robert is Black. Research shows facial recognition has higher error rates for darker-skinned faces

This was how the system was built

Robert Williams case

Source: ACLU

What is bias? 🤔

Defining bias: it’s complicated

“Bias” means different things:

Context Meaning Example
Statistical Systematic deviation Biased estimator under/overestimates
Cognitive Mental shortcuts Confirmation bias
Cultural Learned assumptions “Doctors are male”
Algorithmic Systematic unfairness Different error rates by group
Historical Past inequalities Redlining-era loan data

In AI, bias typically means:

A system that produces systematically unfair outcomes for certain groups of people.

But who defines “unfair”? That’s the hard part.

The mirror problem: AI reflects us

Is AI biased, or is it just showing us what we already are?

  • AI learns from human-generated data, which contains historical discrimination
  • If humans made biased decisions, AI learns those patterns
  • AI then amplifies those biases at scale

Example: Word embeddings

  • “Homemaker” and “nurse” sit closest to “she”; “captain” and “architect” lean “he”
  • The AI picked up our own sexism from millions of human texts

AI trained on human data inherits human biases

Source: UNESCO

If the bias comes from us, does that make AI less accountable, or more?

Discussion: pick your risk

You’re applying to university. Choose who evaluates your application:

Option A: A human admissions officer

  • Might favour applicants who remind them of themselves
  • Could be swayed by mood, time of day, or fatigue
  • You can appeal and read social cues
  • Bias is inconsistent and unpredictable

Option B: An AI admissions tool

  • Might penalise your ZIP code, name, or school
  • Applies the same criteria to everyone, every time
  • You cannot argue with it or explain context
  • Bias is systematic and invisible

Which do you choose, and why?

Think about:

  • Does it matter that you can’t argue with an algorithm?
  • Is consistent bias fairer than unpredictable bias?
  • Would your answer change if you knew which group the bias disadvantaged?
  • What would you want to know before choosing?

Neither option is safe. Which risk would you rather take, and what does that say about what we value?

⏱️ 2 minutes

Why this matters

AI systems are making decisions about:

Domain Decision Affected
Employment Who gets hired, promoted, fired Millions of applicants
Finance Who gets loans, credit, insurance Billions globally
Healthcare Who gets treatment, care priority Life and death
Criminal justice Bail, sentencing, parole Freedom and families
Education Admissions, resources, grading Future opportunities

The scale problem:

  • Algorithms decide faster than humans can review
  • One biased system can affect millions of people
  • Errors compound: a bad decision today feeds worse data into tomorrow’s model

Types of bias

A taxonomy of bias

Bias type When it occurs Example
Historical Past decisions encoded in data Loan data from discriminatory era
Representation Some groups underrepresented Few dark-skinned faces in training
Measurement Proxies used for unmeasurable concepts Using ZIP code for creditworthiness
Aggregation Treating diverse groups as one “One model fits all” fails
Evaluation Wrong benchmarks for testing Testing on unrepresentative data
Deployment Model used in wrong context US model applied globally

Bias can creep in at any stage:

Data collection → Data labelling → Model training → Model evaluation → Deployment → Use

Historical bias: the past repeats itself

Past discrimination baked into the training data, even when the data accurately reflects the world at the time

The Amazon hiring case (2018):

  • Amazon built an AI to screen CVs, trained on 10 years of CVs sent to the company
  • Most came from men, as in the tech industry as a whole
  • The AI penalised “women’s” (“women’s chess club captain”) and two all-women’s colleges
  • Amazon scrapped the tool

The data was accurate: it reflected a male-dominated industry. That industry was biased.

Amazon hiring AI

Source: Reuters

Accurate data ≠ fair data

Representation bias: who’s missing?

When a group is underrepresented in training data, the model performs poorly for them

Example: Voice assistants

  • Trained mostly on American and British accents, male voices and native speakers
  • Result: higher error rates for:
    • Black speakers: 35% of words wrong vs 19% for white speakers (Koenecke et al., 2020)
    • Non-native speakers; Scottish, Indian, Nigerian accents
    • Women (in some systems), children and older users

Why?

  • Whoever collects the data determines who’s in it, and developers sample themselves
  • “Edge cases” are actually most of the world

Voice recognition bias

Source: Axios

Measurement bias: bad proxies

A measurable proxy stands in for what you care about, but doesn’t work equally for everyone

Example: ZIP code as a credit proxy

  • Banks can’t use race in loan decisions, but ZIP codes can stand in for it
  • Housing discrimination makes ZIP codes correlate strongly with race
  • Result: a “race-neutral” variable that encodes race

Other problematic proxies:

What we want Proxy used Problem
Intelligence Standardised tests Reflects access to prep
Job quality Tenure Penalises caregivers
Creditworthiness Payment history Assumes equal opportunity
Health needs Past spending Reflects access barriers

Measurement bias

Source: Harvard Law Review

Remove race from the data and the model can still be racially biased: the proxies do the work instead.

Aggregation bias: one size doesn’t fit all

Treating diverse populations as if they’re all the same, when the underlying relationships differ across groups

Example: Diabetes prediction

  • One model is trained on all patients
  • Diabetes differs across ethnicities: genetic risk factors, dietary patterns, symptom presentations
  • The model works well on average but poorly for specific groups
  • Simpson’s paradox

The maths:

Model Overall Accuracy Group A Group B
Single model 85% 90% 75%
Group-specific 88% 89% 86%

Average performance can hide disparate impact

Source: Wikipedia

Question: Should we build separate models for different groups? What are the tradeoffs?

Evaluation bias: testing on the wrong people

The benchmark dataset used to test a model doesn’t represent the population it will be used on

The benchmark problem:

  • Standard benchmarks become industry standards, and everyone optimises for the same tests
  • If the test is biased, success on the test means nothing

Example: ImageNet

  • For years the gold standard in computer vision, but its images came mostly from the US and Europe
  • Models trained and tested on it worked great in the US
  • Deployed globally: failures on everyday objects from other cultures

Evaluation bias

Source: Wired

High scores on a biased benchmark just mean the model is good at being biased.

Deployment bias: wrong context

A model works fine where it was built but fails in a different context

Example: US model in India

  • Credit scoring model trained on US financial data, deployed in India for loan applications
  • Banking systems, income patterns and credit histories all differ
  • Result: inappropriate decisions for the new context

Example: COVID-19 triage

  • Models trained on UK hospital blood tests and vital signs, then used in Vietnam
  • Patients, recording practices and disease prevalence all differed
  • Accuracy fell to near coin-flip level

Deployment bias

Source: Yang et al (2024)

A model trained in one context can fail quietly when moved to another.

Bias through the AI lifecycle

Bias can appear at every stage

Feedback loops: bias that amplifies itself

A feedback loop: an algorithm’s predictions shape the data it is later trained on

Example: predictive policing

  1. Algorithm predicts hotspots from past arrest data
  2. Police patrol those areas more heavily
  3. More patrols → more arrests (whatever the real crime rate)
  4. New arrest data confirms the predictions
  5. The algorithm grows more confident; the cycle repeats

Why this is dangerous:

  • The algorithm creates evidence for its own predictions, so bias compounds over time
  • After a few cycles, nobody can tell what the real crime rate was

Feedback loop: hot-spots policing

Source: SpotCrime

Other feedback loops:

  • Loan denials → worse credit → more denials
  • CV filters → homogeneous workforce → more biased data

Activity: spot the bias!

Scenario: University admissions

An AI recommends admissions based on SAT scores, high school GPA, and extracurriculars

  • What types of bias might this contain?
  • Who might be disadvantaged?
  • What proxies are being used?

Discuss one scenario with a neighbour:

  1. Identify at least 2 types of bias
  2. Propose how you might mitigate them
  3. What tradeoffs would you face?

⏱️ 2 minutes

The hard questions

The impossibility theorem

You can’t have it all

Three fairness criteria (simplified):

  1. Calibration: among those given the same score, outcomes should be similar across groups

  2. Equal false positive rates: equal rates of being wrongly flagged across groups

  3. Equal false negative rates: equal rates of being wrongly missed across groups

The impossibility theorem:

If base rates differ between groups, you cannot satisfy all three simultaneously

If Group A reoffends at 40% and Group B at 20%, you must choose which type of error to equalise

There is no mathematically “fair” solution.

Impossibility theorem

No algorithm can tell you which errors matter more. That is a political and moral question.

Who defines “fair”?

What counts as fair depends on what you value:

Definition Prioritises Drawback
Equal treatment Consistency Ignores context
Equal outcomes Equity May require discrimination
Equal error rates Group parity May sacrifice accuracy
Calibration Individual accuracy Hides disparity
  • There is no neutral default
  • Whoever picks the fairness definition shapes who benefits
  • Saying “we just use the data” is itself a choice

Fairness tradeoffs

  1. Accuracy vs fairness
    • Equal accuracy across groups may cost overall accuracy. Who pays?
  2. Individual vs group fairness
    • Identical treatment can produce unequal group outcomes
    • Equal group treatment can disadvantage qualified individuals
  3. Transparency vs gaming
    • Reveal the algorithm and people game it; hide it and no one is accountable
  4. Short term vs long term
    • Current data repeats past inequality; correcting for it may cost accuracy today

Different stakeholders want different things:

  • Affected communities: equalise outcomes
  • Companies: maximise accuracy
  • Regulators: ensure process fairness
  • Courts: protect individual rights

Who gets to decide which tradeoff to make?

Perspectives: is AI bias fixable?

Where do you stand?

The optimists

  • AI bias is a technical problem: better data, algorithms, audits
  • AI might be less biased than humans
    • Humans are inconsistent; AI is at least consistent
    • AI decisions can be audited; gut feelings cannot
  • Datasets are improving, regulations are catching up

“A biased algorithm can be retrained. A biased hiring manager is harder to fix.”

The sceptics

  • AI bias reflects societal problems that can’t be coded away
  • “Fair AI” talk distracts from root causes
  • Who defines fairness? Usually those already in power
  • AI obscures human accountability
  • Some decisions shouldn’t be automated at all

“You can’t fix a discriminatory housing market by tweaking a credit-scoring model.”

Summary

Main takeaways

  • AI learns from human data, so it inherits human biases

  • Historical bias bakes past discrimination into models

  • If a group is missing from training data, the model fails for them

  • Neutral-looking proxies (ZIP code, test scores) can encode inequality

  • The impossibility theorem: you cannot satisfy all fairness criteria at once

  • Choosing a fairness definition is a values question, not a technical one

…and that’s all for today!