• The pre-history of AI
• Birth of AI: Dartmouth 1956
• Symbolic AI and expert systems
• AI winters and why they happened
• The neural network comeback
• The data revolution
• Transformers explained simply
• From GPT to ChatGPT

• Humans have long dreamed of creating artificial life
• Ancient myths featured mechanical beings:
  • Talos (Greek): Bronze giant protecting Crete
  • Golem (Jewish folklore): Clay figure brought to life
  • Frankenstein (1818): Mary Shelley’s novel about creating life
• We’ve always been fascinated with replicating intelligence
• Can machines truly think, or only simulate thinking?

• Can intelligence be reduced to rules and symbols, or does it emerge from interconnected units?
• 1642: Blaise Pascal builds the Pascaline, a mechanical calculator for addition and subtraction
• 1673: Gottfried Leibniz creates a machine that can multiply and divide
• 1837: Charles Babbage designs the Analytical Engine, a programmable, general-purpose computer (never completed)
• Ada Lovelace writes what many consider the first computer program for Babbage’s machine
• Calculation could be separated from human minds

• For most of history, the word “computer” referred to a person, not a machine!
• Large teams of human computers performed complex calculations:
  • Astronomical tables
  • Ballistics trajectories
  • Census data
• Often women, who were paid less than male mathematicians
• This division of mental labour (Adam Smith!) showed that complex calculations could be broken into simple, mechanical steps
• Historian Lorraine Daston: Calculation shifted from “genius” to “merely mechanical”

• 1936: Alan Turing publishes “On Computable Numbers, with an Application to the Entscheidungsproblem” (decision problem)
• Introduces the concept of a universal machine that can compute anything computable
• 1950: Publishes “Computing Machinery and Intelligence” in Mind (philosophy journal)
  • Proposes the famous Turing Test: Can a machine fool a human into thinking it’s human?
  • Asks: “Can machines think?” and reframes it as a practical question (“Can machines do what we (as thinking entities) can do?”)
• Turing’s ideas laid the theoretical foundation for both computing and AI
• He suggested machines could learn, not just follow fixed rules

• Summer 1956: A workshop at Dartmouth College marks the birth of AI as a field
• Organised by John McCarthy, Marvin Minsky, Nathaniel Rochester, and Claude Shannon
• First use of the term “Artificial Intelligence”
• The founding conjecture:

“Every aspect of learning or any other feature of intelligence can in principle be so precisely described that a machine can be made to simulate it.”

• Attendees included Herbert Simon and Allen Newell
• Optimism was high: Many believed human-level AI was just decades away

Symbolic AI, 1956–1980s

• Intelligence = manipulating symbols according to rules
• Program the formal rules behind intelligent behaviour into machines
• Early successes: ELIZA (therapist, video here), SHRDLU (block world, video here)
  • Try ELIZA here: https://anthay.github.io/eliza.html (by the ELIZA Archaeology Project)
• Bold but overconfident predictions: Minsky and Simon expected human-level AI within a generation

Expert systems, 1970s–1980s

• Shift to narrow domains with expert knowledge

• Encode human expertise as if-then rules

• Examples: MYCIN (bacterial infections, video here), DENDRAL (chemical analysis, video here)

• Edward Feigenbaum: “The problem-solving power… is primarily a consequence of the specialist’s knowledge employed”

• Both approaches relied on hand-crafted knowledge, not learning from data

• Neither could learn or improve automatically

• 1943: McCulloch and Pitts propose artificial neurons
  • Binary units that mimic biological ones
• 1958: Frank Rosenblatt builds the Perceptron
  • The first network that could learn, adjusting its weights whenever it got something wrong
• The New York Times (1958) predicted it would “walk, talk, see, write, reproduce itself”
• Stop writing rules. Show the machine examples
• Single-layer perceptrons failed on simple problems, and interest waned
• Multi-layer networks could solve them, but nobody knew how to train them
• The comeback took nearly 20 years
• But how does an artificial neuron work?

• Each input (\(x_i\)) is a number, say a pixel’s brightness
• A weight (\(w_i\)) sets how much that input matters
• Add the weighted inputs, plus a bias (\(b\)) that decides when the neuron fires: \(z = \sum x_i \cdot w_i + b\)
• Pass \(z\) through an activation function, a non-linear bend: \(y = f(z)\)
• Without that bend, the whole network is just a linear model

• 1986: Rumelhart, Hinton, and Williams popularise backpropagation
• You can train multi-layer networks by propagating errors backwards through the layers
• It finds functions that are highly non-linear with very precise adjustments to many weights
• Finally, neural networks could learn complex patterns!
• Renewed interest in connectionism
• But challenges remained:
  • Limited computing power
  • Not enough training data
• Neural networks showed promise but couldn’t yet compete with other methods

• 2009: Google researchers (Halevy, Norvig, Pereira) publish a landmark paper called “The Unreasonable Effectiveness of Data”
• Simple models with lots of data beat complex models with less data
• Traditional AI: Design clever algorithms
• New AI: Feed massive amounts of data to simple learning algorithms

“Now go out and gather some data, and see what it can do.”

• Data, computing power, and better algorithms came together at the same time
• Any one alone wasn’t enough, but all three together changed the field

• ImageNet: A dataset of 14 million labelled images
• Annual competition: Classify images into 1,000 categories
• 2012: AlexNet (Krizhevsky, Sutskever, Hinton) wins by a huge margin
  • Error rate: 15.3% (next best: 26.2%)
  • Used deep neural networks
  • Trained on Graphics Processing Units (GPUs), which are better than Central Processing Units for parallel processing
• This was a great moment for deep learning!
• Within years, deep learning dominated computer vision
• Proved that neural networks + big data + GPUs = breakthrough

• 2017: Google researchers publish “Attention Is All You Need” (Vaswani et al.)
• Self-attention mechanism
• Instead of processing sequentially:
  • Each word can “look at” every other word directly
  • Computes relevance scores between all pairs
  • Parallel processing: Much faster to train
• This architecture powers GPT (Generative Pre-trained Transformer), BERT, and pretty much all modern AI
• Beyond text: Now used in images, audio, video, proteins, games…

• Text must be converted to numbers
• Tokenisation: Split text into tokens
  • Usually words, letters, or word pieces
  • “unhappiness” → “un”, “happiness”
  • GPT-2 has a vocabulary of 50,257 tokens
• Each token gets a unique ID number
• Think of it as creating a dictionary where each word/piece has a code

• Each token becomes a vector (a list of numbers)
• GPT-2: Each token → 768 numbers (nothing special about 768; models use different sizes)
• These vectors capture meaning:
  • Similar words are close together
  • “king” - “man” + “woman” ≈ “queen”
• Also add position information:
  • Word order matters in language!
  • “Dog bites man” ≠ “Man bites dog”
• Embeddings start as random numbers
  • Every bad guess nudges them, and words used alike drift together
• Nobody arranges them. Meaning is a side effect of predicting well

• The model is always guessing the next word:
  • “The animal didn’t cross the street because it was too ___”
• “tired” if “it” means the animal, “wide” if it means the street
• So the guess depends on what “it” refers to
• But after Step 2 each word sits alone, and “it” is the same 768 numbers in every sentence
• Attention lets “it” reach back to “animal” and “street”, weigh both, and “tired” wins
• Attention is the only step where words see each other

• “The cat sat on the ___”
• Attention brings “cat” and “sat” to the guessing position
• But nothing in the sentence says what a cat sits on, so it must come from memory
• That memory is the feed-forward network (MLP)
• Attention moves what is in the sentence. The network adds what is not
• The two make one block: 12 in GPT-2, 96 in GPT-3, far more since
• Each block adds to the word rather than replacing it, so nothing is lost on the way up
• Each round asks a better question than the last:
  • Early blocks: what kind of word is this?
  • Late blocks: what is this text about, and what comes next?

• After all the blocks, the model asks: “what comes next?”
• It gives every token in its vocabulary a score
• Most score roughly zero. A handful take almost everything
• Temperature controls how the winner is picked:
  • Low (0.2): takes the safe option
  • High (1.0+): takes more risks
• The chosen token joins the sentence, and the whole process repeats

• Modern AI isn’t just about text anymore
• Multimodal models can process:
  • Text ↔︎ Images (DALL-E, Midjourney)
  • Text ↔︎ Audio (Whisper, ElevenLabs)
  • Text ↔︎ Video (Sora, Runway)
  • Text ↔︎ Code (Codex, Copilot)
• Same transformer architecture, different inputs/outputs
• Vision Transformers (ViT): Treat images as sequences of patches
• The boundaries between modalities are blurring

• Georgia Tech’s Transformer Explainer
• Interactive visualisation of how transformers work
• See attention patterns in real time
• Experiment with temperature and sampling
• Runs GPT-2 directly in your browser!
• Great for building intuition

https://poloclub.github.io/transformer-explainer