• We traced AI from Ancient Greece to modern transformers
• Symbolic AI (1950s-1980s): Hand-coded rules and logic
• AI Winters: When hype exceeded reality (twice!)
• Neural networks: From perceptrons to deep learning
• Transformers (2017): Attention is all you need!
• More data beat better algorithms… but where do the data come from? 🤔

• Today’s topic is the data AI systems learn from
• Why data quality matters
• Types of AI tasks
• Sourcing and collecting data
• Selection bias and its dangers
• Best practices for labelling data
• Working with multiple annotators
• Quality control in datasets

• AI models are only as good as the data they’re trained on
• A better algorithm cannot fix bad data
• Common data problems:
  • Missing important information (e.g., age, income, etc.)
  • Conflicting labels or formats (e.g., “yes” vs “1”)
  • Biases, not representative of reality (e.g., gender, race, etc.)
• Data-centric AI: Andrew Ng (2021) argues you should improve the data before you improve the model
  • Check: https://datacentricai.org/

Classification: Predicting categories

• Goal: Assign inputs to discrete categories
• Examples: Email → Spam/Not Spam; Image → Cat/Dog/Bird; Text → Positive/Negative/Neutral
• Binary classification: Two classes
• Multi-class: Many classes (one per input)
• Multi-label: Multiple labels per input
• How do you define the classes?

Regression: Predicting continuous values

• Goal: Predict a continuous numerical value
• Examples: Features → House price; Patient data → Hospital stay length; Weather data → Temperature
• The choice of target variable is a design decision
• Different targets can lead to very different models

• Classes should be:
  • Mutually exclusive: No overlap between categories
  • Collectively exhaustive: Cover all possible cases
  • Consistently definable: Clear boundaries
• Common problems:
  • Ambiguous categories: What counts as “spam”?
  • Subjective judgments: Is this review “positive”?
  • Cultural differences: What’s “offensive” varies
  • Evolving definitions: Categories change over time
• Solution: Clear annotation guidelines

• Often we can’t directly measure what we want to predict
• So we use proxy variables instead, and proxies can mislead
• Example 01: Predicting “health needs”
  • What we want: How sick is this patient?
  • Available data: Healthcare spending
  • Spending ≠ health needs: Black patients had less spent on them due to systemic barriers
  • Algorithm learned: “Black patients need less care” → Wrong!
• Example 02: Predicting expertise on social media
  • What we want: Who is an expert in a field?
  • Available data: Number of followers
  • Followers ≠ Expertise: Popularity can be gamed or unrelated to domain knowledge

• Always ask: Does this label actually measure what I care about?

• There’s no universal answer… it depends on:
  • Task complexity: harder tasks need more data
  • Number of classes/features: more categories = more examples needed
  • Model architecture: deep learning is data-hungry
  • Required accuracy: higher stakes = more data
• Rule of thumb: Start small, then scale up
• Scaling laws: Performance improves predictably with more data
  • You can estimate how much data you’ll need by training on subsets and extrapolating
• For most real-world problems, more high-quality data beats more sophisticated algorithms

• Selection bias: your training data differ systematically from the cases the model will meet
• Close relatives: sampling bias, distribution shift, survivorship bias
• The model learns patterns that won’t generalise
• Extremely hard to fix after the fact
• Prevention is better than cure!
• Always ask: “In what ways might my data not be representative?”

Source: Qwen via nanoGPT

Source: Gemini’s NanoBanana

Before collection:

• List the biases you expect before collecting data
• Design collection to cover important subgroups
• Over-sample rare but important cases

After collection:

• Use data augmentation to increase dataset size
  • Try it out at home! Use an image generator to create variations of the same object (different angles, lighting, etc)
• Use representative validation sets
• If deploying in new locations: validate on location-specific data
• Document known limitations of your dataset

• Most AI models require labelled data
• Labels come from humans → humans make mistakes
• Common issues:
  • Inconsistency: Different people label differently
  • Ambiguity: Some examples are genuinely unclear
  • Mislabelling: Human errors happen
  • Fatigue: Quality drops over time
  • Expertise: Some tasks need domain experts
• Labelling is expensive and time-consuming
  • See Alexandr Wang’s Scale AI
• Quality control is super important!

Why use multiple annotators:

• Individual annotators make mistakes
• Multiple annotators → more reliable labels
• Benefits: Catch errors, identify ambiguous cases, assess label quality, reduce biases
• Trade-off: More expensive

Aggregation strategies:

• Majority vote: Most common label wins (limitations: ties, equal weighting)
• Weighted voting: Trust reliable annotators more
• Adjudication: Expert makes final call
• Keep all labels: Train on disagreement (for uncertainty research)

Measuring agreement (IAA):

• Inter-annotator agreement: Do annotators agree?
• High agreement → Labels are reliable
• Low agreement → Problem with guidelines or task
• Common metric: Cohen’s Kappa (Fleiss’ Kappa for more than two annotators)

• Quality control (QC) examples: Examples where you already know the correct answer
• Slip QC examples into the labelling queue without telling annotators
• Use them to:
  • Monitor annotator accuracy over time
  • Identify annotators who need retraining
  • Catch low-quality work early
• Flag annotators who fail too many QC checks
• Combine with regular audits of random samples