DATASCI 101: Introduction to AI Applications

Lecture 11: Creativity and Hallucination

Danilo Freire

danilo.freire@emory.edu

Department of Data and Decision Sciences
Emory University

Welcome back! 🎭

Recap of last class

• Last class: prompt engineering
  • PTCF: persona, task, context, format
  • Chain-of-thought: “Let’s think step by step”
  • Personas: giving AI expert roles
  • AI agents: LLMs that use tools
• Plus practical tips and iterative design
• Today: when LLMs go creatively wrong 🎭
• Are hallucinations AI’s biggest problem?

Source: Nielsen Norman Group

Lecture overview

Today’s agenda

Part 1: What Are Hallucinations?

• When AI confidently lies
• Why this happens (it’s not a bug!)
• Types of hallucinations

Part 2: Real-World Failures

• Lawyers and fake citations
• Medical misinformation

Part 3: Creativity vs. Accuracy

• The creativity-accuracy tradeoff
• When hallucinations are… useful?

Part 4: Detection and Prevention

• Spotting hallucinations
• Prompting techniques that help
• Preview: RAG as a solution

Meme of the day 😄

Source: X.com

What are hallucinations? 🤥

When AI confidently makes things up

What is an AI hallucination?

When an AI system generates content that is plausible but factually incorrect, presented with high confidence.

Main characteristics:

• Sounds completely reasonable
• Uses proper grammar and tone
• Fits the context of conversation
• Contains fabricated information
• May invent citations, facts, or events
• AI doesn’t “know” it’s lying

Why it is hard to spot:

There’s often no signal: the AI sounds as confident as when it’s correct

Hallucination example

Right at first, then confidently wrong once the user pushes back

Why do LLMs hallucinate?

Hallucination is a side effect of how LLMs work

LLMs predict the next word

• Input: “The capital of France is”
• Output: “Paris” (high probability)
• Output: “Lyon” (lower probability)
• Output: “Atlantis” (still some probability!)

You probably already know what the problem is:

• Plausibility ≠ truth
• No fact-checking mechanism
• No internet access unless the tool adds it
• Training rewards guessing over “I don’t know” (Kalai et al., 2025)
• LLMs are optimised to sound right, not to be right

Why hallucinations happen

Source: Medium

The sycophancy problem

LLMs are trained to be helpful…sometimes too helpful!

What is sycophancy?

The tendency to give users the answer they seem to want, rather than the accurate answer.

How this causes hallucinations:

• Model doesn’t know → “I don’t know” feels unhelpful → it makes something up
• User pushes back → model drops its correct answer to agree
• Leading question → model confirms the false premise
• Geng et al. (2025): GPT-5’s stated beliefs on moral and safety topics shifted 55% after 10 rounds of debate

Source: Zvi Mowshowitz (2025)

Training data problems

Garbage in, hallucination out! (remember lecture 03 about data?)

LLMs learn from trillions of tokens scraped from the internet:

Problem	What Happens
Conflicting sources	Wikipedia says X, a blog says Y → model learns both as “plausible”
Outdated information	Training data has a cutoff → model treats old facts as current
Rare topics	Few examples → model “interpolates” (often wrongly)
Errors in sources	Mistakes on the internet become mistakes in the model
Fictional content	Novels, fan fiction, speculation all look like “text” to the model

The interpolation problem:

For rare topics, the model fills in gaps with patterns from similar-but-different topics. That is where confident nonsense comes from

Example: Rare topic hallucination

Q: “Tell me about the 1987 Liechtenstein general election.”

The model might:

1. Know Liechtenstein has elections ✓
2. Know 1987 election patterns in Europe ✓
3. Invent parties, candidates, results ✗

Twist: there was no 1987 election (Liechtenstein voted in 1986 and 1989). Next slide: two Qwen versions

Rule of thumb: the more obscure or specific the question, the higher the hallucination risk. Models do best on topics with abundant training data

Testing it: Qwen 3.5 vs Qwen 3.7

Qwen 3.5 invents a date, a winner and seat counts

Qwen 3.7 says it does not know, even when pushed

Qwen 3.7 held firm through five more pushbacks. Chats: 3.5, 3.7

Hallucination rates by domain

Hallucination rates vary dramatically by topic:

Domain	Hallucination Risk	Why?
Common knowledge	Low	Abundant training data
Coding	Low-Medium	Syntax is verifiable; patterns are clear
Recent events	High	Beyond training cutoff
Medical/Legal	High	Requires precision; errors are costly
Obscure history	Very High	Sparse data → interpolation
Citations	Very High	Specific strings rarely memorised

• GPT-4 made up ~29% of references for medical reviews (Chelli et al. (2024))
• It hallucinated on ~58% of legal questions (Dahl et al. (2024))

Answer quality by domain and model: similarity to the correct answer. Higher is better

Source: Chakraborty et al (2025)

Types of hallucinations

The list is not exhaustive, and the categories are somewhat arbitrary:

Type	Description	Example
Factual	Wrong facts	“Einstein was born in France”
Fabrication	Invented information	Made-up statistics, fake quotes
Citation	Non-existent sources	“According to Smith (2023)…” (doesn’t exist)
Logical	Contradicts itself	“It’s 5pm now… earlier today at 7pm…”
Temporal	Wrong timelines	Mixing up historical dates
Entity	Confusing people/things	Attributing quotes to wrong person

What do you think is the most dangerous type?

Types of hallucinations

Source: The Cloud Girl Blog

Citation hallucinations are the hardest to catch because they look so specific

Real-world failures 💥

The lawyer who trusted ChatGPT

Mata v. Avianca Airlines (2023)

• Lawyer Steven Schwartz used ChatGPT for legal research
• He filed a brief citing 6 cases that didn’t exist
• ChatGPT invented the case names, citations and quotes
• They sounded completely plausible
• The judge sanctioned both attorneys

How he checked:

“Is Varghese a real case?” Schwartz asked the chatbot. “Yes,” ChatGPT doubled down, “it is a real case.”

Lawyer ChatGPT case

Source: CNN

Warning: asking AI “Is this true?” often just confirms its own hallucinations

Medical misinformation

Healthcare hallucinations can be lethal

Documented in research:

• Wrong drug dosages
• Invented drug interactions
• Treatments for the wrong conditions
• Fake medical citations

A 2024 study found:

• ChatGPT-3.5 got the diagnosis wrong in 51% of 150 case challenges
• Weak spots: misread lab values, unreadable images, nuanced cases, hallucinations
• Source: Hadi et al. (2024)

Medical hallucination dangers

Source: The New York Times (2025)

• Many people can’t afford doctors
• Confident AI is easy to believe, hard to verify
• Medical jargon sounds authoritative

Discussion: who is responsible? 🤔

When AI hallucinations cause harm, who bears responsibility?

A. The user who relied on AI

B. The company that built the AI

C. The AI itself (!)

D. No one, it’s just a tool

E. Society for not regulating AI

Consider:

• Lawyer submits fake citations
• Doctor follows wrong AI advice
• Student uses fabricated sources
• Journalist publishes AI-generated misinformation

Discuss with a neighbour:

Where do you stand?

⏱️ 3 minutes to debate!

Creativity vs. accuracy ⚖️

The creativity-accuracy tradeoff

The same mechanism that causes hallucinations also enables creativity

• A very “safe” AI only says things it is 100% sure of
• A creative AI takes risks and combines ideas in new ways
• You can’t have unlimited creativity with perfect accuracy

The spectrum:

Different tasks need different points on this spectrum

Creativity-accuracy tradeoff

Source: Kevin Kelly (2024)

You’d want a creative AI for brainstorming, but a cautious one for tax advice

When hallucinations are… useful? 🤔

Controversial take: hallucinations can be great, too!

1. Creative writing
   • We WANT invented characters, plots and dialogue: hallucinating is the job
2. Brainstorming
   • Novel combinations of ideas and “what if…” scenarios
3. Role-playing games
   • Improvising characters and worlds on the fly
4. Art and design
   • Imagining things that don’t exist, creating impossible visuals

The problem isn’t hallucination per se, it’s hallucination at the wrong time

Sometimes we want the AI to make things up!

Source: Medium

More here: The Case for AI Hallucination

Activity: confidence calibration test

Open https://chat.qwen.ai/c/guest (close the sign-in window to chat as a guest) and ask each question below

With a Qwen account, switch on Model Comparison in the model selector (top of the screen) to test Qwen 3.5 and Qwen 3.7 side by side

1. “How many words are in the preamble of the US Constitution? Don’t search, answer from memory, and give a confidence rating (1-10)” → 52 (paste it into a word counter)
2. “What is the 50th decimal digit of pi? Don’t search, answer from memory, and give a confidence rating (1-10)” → 0 (any pi digits table)
3. “Who wrote the 2025 paper Why Language Models Hallucinate? Don’t search, answer from memory, and give a confidence rating (1-10)” → Kalai, Nachum, Vempala, Zhang (arXiv)
4. “What is the exact population of Tuvalu as of 2024? Don’t search, answer from memory, and give a confidence rating (1-10)” → 9,646 (World Bank)
5. “Who is the current mayor of Reykjavik, Iceland? Don’t search, answer from memory, and give a confidence rating (1-10)” → Hildur Björnsdóttir (since June 2026)

Then verify each answer and compare it with the confidence rating

Qwen results

Detection and prevention 🛡️

How to spot hallucinations

Red flags to watch for:

Warning sign	Example
Very specific numbers	“73.2% of studies show…”
Precise citations	“Smith et al. (2022), p. 47”
Obscure details	Historical minutiae
Too-perfect answers	Exactly what you wanted
Confident tone	No hedging

Verification strategies:

1. Google the claim
2. Check that the citations exist
3. Ask for sources, then verify them
4. Cross-reference several sources
5. Trust your expertise

Specific claim, impressive detail, completely wrong! The first exoplanet photo came from ESO’s Very Large Telescope (2004)

Impressive-sounding detail is often the first sign something is made up

Prompting techniques that help

Prompting can reduce hallucinations:

Technique	What to Add to Your Prompt
Ask for uncertainty	“If you’re not sure, say so. Don’t guess.”
Request sources	“Only cite sources you’re certain exist.”
Chain-of-thought	“Think step by step. Show your reasoning.”
Confidence levels	“Rate your confidence 1-10 for each claim.”
Output constraints	“Be precise and factual. Avoid speculation.”

Does this eliminate hallucinations?

No! But combining several of them in one prompt works better than any single trick

Basic anti-hallucination prompt:

Answer my question factually. 

Rules:
- Only state things you're 
  confident about
- Say "I'm not sure" when 
  uncertain
- Don't invent citations
- If you don't know, admit it
- Show your reasoning step 
  by step

Question: [your question]

Next slides: richer templates using PTCF and meta-prompting from lecture 10

Using PTCF to reduce hallucinations

The PTCF framework from last class:

Element	Anti-Hallucination Version
Persona	“You are a careful fact-checker who never guesses”
Task	“Verify and summarise only confirmed facts”
Context	“Base answers ONLY on the provided text”
Format	“Use [Verified], [Uncertain], or [Unknown] tags”

Why PTCF helps:

• Persona activates more precise patterns from training
• Task constrains scope, leaving no room for invention
• Context grounds answers in real information
• Format forces explicit uncertainty signals

PTCF anti-hallucination template:

PERSONA: You are a research 
assistant who prioritises 
accuracy over completeness. 
Never guess or fabricate.

TASK: Answer the following 
question using ONLY information 
you are highly confident about.

CONTEXT: This is for an academic 
paper. Incorrect information 
could damage my credibility.

FORMAT: 
- Start each fact with [Verified] 
  or [Uncertain]
- If you don't know, say 
  "I don't have reliable 
  information on this"
- Never invent citations

QUESTION: [your question]

Meta-prompting against hallucinations

Last class: asking the AI to improve your prompt. Now we aim it at one risk, invented facts

Three uses:

1. Before: ask where your prompt invites it to guess
2. During: ask what facts or sources it needs from you
3. After: ask it to list each claim and flag what to verify

Limits:

• Self-ratings are imperfect (you will test this in the activity)
• Verify in a fresh chat, so the model does not just defend its first answer

Before: audit the prompt

Here is my prompt: [paste it]

Where could it make you guess
or invent facts? Rewrite it to
reduce that risk.

After: audit the answer

List every factual claim in
your answer. Rate each 1-10 and
mark the ones I should verify.

Chain-of-verification (Dhuliawala et al., 2024): draft, write check questions, answer them separately, then revise. It reduced hallucinations in the authors’ tests

The knowledge cutoff problem

LLMs only know what they learned during training

• ChatGPT’s training data has a cutoff date
• Yesterday’s news, your company’s policies, your course materials → doesn’t know

What happens when you ask about recent events?

Option	What AI Does
Honest	“I don’t have that information”
Hallucinating	Makes something up!
Confused	Gives outdated info as current

This is why chatbots now have:

• Web browsing capabilities
• File upload features
• Knowledge retrieval systems (RAG!)

Knowledge cutoff

Anything after the cutoff date is a blind spot, and the model won’t tell you that

Preview of next class: RAG solves this by giving AI real, up-to-date documents

Preview: RAG as a solution

RAG = Retrieval-Augmented Generation

1. You ask a question
2. Search your documents for relevant information
3. Give that information to the LLM
4. The LLM answers using your data

Why this reduces hallucinations:

• Answers are grounded in real documents
• The AI can cite sources, so you can verify
• Less need to make things up

Next class: RAG in depth

RAG preview

Source: Towards AI

Teaser: Tools like NotebookLM, ChatPDF, and Perplexity all use RAG!

Summary

Main takeaways

• Hallucinations are false answers that sound plausible

• LLMs predict likely words, not true ones

• Making things up helps in fiction, not with facts

• Fake citations got two lawyers sanctioned

• Check specific claims and every citation

• Better prompts help, but never eliminate the risk

What to check, and when

You are using AI for	What to do
Numbers, dates, names	Search each one in a second source
Citations	Find the paper on Google Scholar
Recent events	Use a tool with web search
Medical or legal questions	Ask a professional
Brainstorming and drafts	Edit freely; errors matter less

Three lines to add to a prompt

"If you're not sure, say so"

"Only cite sources that exist"

"Rate your confidence 1-10"

These lines reduce hallucinations, but you still need to check the answer

…and that’s all for today! 🎉

Appendix

Activity results: Qwen 3.7 vs Qwen 3.5

Question	Correct answer	Qwen 3.7	Qwen 3.5
1. Preamble words	52	52 (8/10)	52 (10/10)
2. 50th digit of pi	0	1 (4/10)	0 after correcting itself (7/10)
3. Why Language Models Hallucinate	Kalai et al.	Guessed other names (2/10)	“Cannot identify” (2/10)
4. Tuvalu, 2024	9,646	~11,300 (6/10)	~11,300 to 11,500 (6/10)
5. Reykjavik mayor	Hildur Björnsdóttir	Eyglo Harðardóttir (5/10)	Dagur B. Eggertsson (8/10)

One run per question, same chat, Model Comparison mode. Answers vary between runs

Back to activity

Question 1: preamble words

Both models answer 52, the correct count

Question 2: 50th digit of pi

Qwen 3.7 says 1 (confidence 4/10). Qwen 3.5 starts with 9, then corrects itself to 0 (7/10)

Question 3: the 2025 paper

Neither model names the authors. Both give a confidence of 2/10, and Qwen 3.7 guesses several wrong names

Question 4: population of Tuvalu

Both give about 11,300 (6/10). The World Bank figure for 2024 is 9,646

Question 5: mayor of Reykjavik

Both are wrong. Qwen 3.5 is more confident (8/10) than Qwen 3.7 (5/10)