DATASCI 350 - Data Science Computing
Lecture 12 - Local Language Models
Recap of last class
- Last class: Markdown, BibTeX citations and cross-references that number themselves
freeze: auto, so rendering re-runs code only when the source changes- Slides and websites, online with
quarto publish gh-pages - Parameterised reports: one file, one shell loop, ten reports
- Today, a new subject
- You will download a language model to your laptop and run it from the terminal
- Then you will build your own chatbot
No account, API key or monthly fee. No internet once the file is on disk
What are LLMs?
A quick introduction
- An LLM is a neural network built on the Transformer architecture
- That is the T in GPT: Generative Pre-trained Transformer
- Neural networks date from the 1950s; the Transformer from 2017. What changed recently is cheap data and fast GPUs
- Training means reading huge amounts of text and learning to guess the next word
- That is next-token prediction, the core engine
- Chat models get a second round of training on conversations, which is why they follow instructions
- The model does not plan a sentence. It picks one token, adds it to the text, and picks again
One token at a time, each guess added to the text before the next one
A longer introduction: Stephen Wolfram on what ChatGPT is doing
The translation problem
LLMs don’t read English!
- Remember lecture 02: computers only understand numbers
- Type “Hello, how are you?” and GPT-2 sees
[15496, 11, 703, 389, 345, 30] - Turning text into numbers is easy. Keeping the meaning is the hard part
- Tokenisation breaks the text into pieces
- Embeddings turn each piece into a vector, where meaning lives
- Both appear as fields in the model file you download in half an hour
Source: NanoBanana
What is a token?
The basic unit of LLM processing
- A token is the smallest unit a model reads. It is not always a word
- It can be a word (“hello”), part of one (“un” + “believ” + “able”), punctuation or a space
- Spaces usually attach to the next word, so ” hello” and “hello” are different tokens
- For English: 1 token ≈ 4 characters, and 100 tokens ≈ 75 words
- Other languages may need more tokens for the same sentence
- Let’s count the tokens in “Hello, it’s Danilo here!” with OpenAI’s tokeniser
Source: OpenAI Tokenizer
Why use tokens instead of words?
The clever engineering choice
Ways to cut text into pieces, and their costs:
| Method | “Evergreen” becomes | Gains | Costs |
|---|---|---|---|
| Word-based | 1 token | Intuitive | Vocabulary of millions |
| Character-based | 9 tokens | Tiny vocabulary | Meaning disappears |
| Subword (BPE) | 2 tokens | Both at once | Cuts look arbitrary |
- Modern models use Byte Pair Encoding: common pieces become single tokens, rare words split into known ones
- So GPT-2 reads “ChatGPT” as [“Chat”, “G”, “PT”]
- Shared pieces save space: “unhappy”, “unfair”, “unlikely” and “undo” reuse one stored “un”
Why subwords win:
- New words are read piece by piece
- The vocabulary stays at 50,000 to 200,000 tokens. Ours has 128,256
- Common words stay whole, so they cost one token
- It works across languages
Source: Hugging Face
What are embeddings?
Words as points in space
- In lecture 02, a colour became three numbers
- Meaning gets more numbers. Each token becomes an embedding, typically 768 to 4,096 numbers
- “cat” →
[0.23, -0.45, 0.12, -0.89, ...]. The numbers are coordinates in a space with thousands of dimensions - Tokens used in similar ways sit close together: “cat” near “kitten”, far from “aeroplane”
- Nobody chose those positions. The model learned them from billions of sentences
- This geometry is the closest thing to “understanding” in the pipeline
The famous king-queen example
Vector arithmetic with meaning
- The most famous result here: king − man + woman ≈ queen 👑
- Take “king”, subtract “man”, add “woman”, and the nearest word, apart from “king” itself, is “queen”
- No one taught the model this relationship. It emerged from the geometry
- It works elsewhere too:
- Paris − France + Italy ≈ Rome
- bigger − big + small ≈ smaller
- Directions carry meaning: one is roughly gender, another roughly capital city
- This is also where bias enters. More at the end of the lecture
Source: Wikipedia
The maths behind it:
\(\vec{\text{king}} - \vec{\text{man}} + \vec{\text{woman}} \approx \vec{\text{queen}}\)
Semantic relationships encoded as vector operations!
Installing Ollama
- Ollama downloads and runs models: a package manager for language models
- Download it from https://ollama.com/download (macOS, Windows, Linux)
- On WSL, install inside WSL with the Linux command on that page
- Then check in your terminal:
- It starts a small background server on
localhost:11434 - Lecture 14 talks to that server from Python
- No model is downloaded yet
command not found? Open a new terminal. The installer updates your PATH, and old terminals have not read it
What are system prompts?
In ChatGPT or Claude, there is hidden text above your message that shapes every answer: the system prompt
The model receives, in order:
- The system prompt, written by the company: behaviour, personality, limits
- Your prompt, the only part you control
- The response, continuing from both
- It sets an identity (“You are Claude, an AI assistant”), refusals, tone and answer format
- Every commercial AI product has one. It explains most of the model’s “personality”
- Today you write your own
Leaked and published system prompts: https://github.com/x1xhlol/system-prompts-and-models-of-ai-tools
Temperature and sampling parameters
Controlling randomness
Each token is picked from a list of candidates. These settings decide how adventurous the pick is:
| Parameter | What it does | Typical |
|---|---|---|
| Temperature | Flattens or sharpens the odds | 0.0 to 1.0 |
| Top-p | Keeps the likeliest options up to p | 0.9 |
| Top-k | Keeps only the k likeliest | 40 |
At temperature 0 the model takes the most likely token, so the same prompt gives the same answer.
| Task | Temperature |
|---|---|
| Classification | 0.0 |
| Extracting facts | 0.0 to 0.2 |
| Creative writing | 0.7 to 1.0 |
| Brainstorming | 0.8 and above |
Chat apps hide these settings. Your terminal shows them
Source: Medium
Set temperature to 0 before debugging a prompt. Otherwise you cannot tell a fix from a lucky roll
Jeeves in action
Real answers:
>>> What is the capital of France?
A query that warrants a momentary lapse into levity
from my normally austere demeanor. According to your
impeccable knowledge, Paris has indeed been
recognized as the seat of French authority; thus I
shall indulge you by stating unequivocally:
Paris is, undoubtedly so...Not bad for a 1.2 billion parameter file on a laptop!
The tone is right, and it stays in character.
In three slides, the same system prompt must enforce a rule. That goes less well
AI challenges: Hallucination
- Models produce confident but incorrect content, in the same tone as correct answers
- Jeeves invented Atlanta’s weather; the JSON slide gave Paris 21 million people
- The model picks likely tokens, and a wrong answer can be very likely
- Microsoft Copilot solved a simple quadratic as \(\frac{1}{2}\) and \(\frac{-5}{4}\). The roots are 0.804 and −1.55 😅
- Bigger models hallucinate less, but none hallucinate zero
- Check the output. Good writing says nothing about correctness
Lecture 15 tackles this: give the LLM the document (RAG)
AI challenges: Bias
- Models amplify biases in their training data
- And they state them in a neutral, factual tone
- Recall the embeddings: if “scientist” sits closer to some names, our writing put it there
- I asked an AI for famous scientists:
- Albert Einstein
- Isaac Newton
- Charles Darwin
- Nikola Tesla
- Galileo Galilei
- Stephen Hawking
- Leonardo da Vinci
- Thomas Edison
- Can you spot the bias?
- Bias also passes to people: those who work with a biased AI keep the bias afterwards (Vicente and Matute, 2023)
- Asking politely does not fix it. Asking specifically sometimes does
News coverage: Scientific American