DATASCI 350 - Data Science Computing

Lecture 10 - Reproducible Research and Literate Programming

Danilo Freire

danilo.freire@emory.edu

Department of Data and Decision Sciences
Emory University

Nice to see you all again! 😊

Recap of our last lectures

Before the quiz, we covered

• The command line: navigation, file management, text tools, pipes, and scripts
• Git: repositories, staging, commits, branches, and merges
• GitHub: remotes, forks, pull requests, issues, and the gh CLI
• Last class you put all of it to work in quiz 01
• Congratulations, that was the hardest setup work of the course 🎉

I hope the image doesn’t bring back bad memories! :)

Today’s lecture

A new module starts today: can anyone check your work?

• The reproducibility crisis: why published research often fails when re-run
• Habits that prevent it: project structure, raw data, paths, seeds, environments
• Literate programming: text, code and results in one file
• Quarto: installing it and writing your first document
• Next class: PDFs, citations, websites and parameterised reports

https://quarto.org/

Keep your copy of the course up to date

• Did you fork the course repository? Your fork does not update by itself
• Please sync it often! 🤓
• On GitHub: open your fork, click Sync fork, then Update branch
• Then, in your terminal, inside your local copy: git pull
• Did you clone the repository directly? Then git pull is enough
• Keep your own work in a separate folder, so a sync never clashes with it

The same from the terminal, with the gh CLI:

# Update your fork on GitHub
gh repo sync your-username/datasci350

# Update the copy on your laptop
cd datasci350
git pull

Tweets of the day

https://x.com/v0xium/status/2101526107128529120

https://x.com/mitchellh/status/2100249348345057389

What mathematicians think about AI in academia

Recommendations from the AI and Mathematics PhD Summit (Harvard CMSA)

Why is reproducible research so important? 🤔

The reproducibility crisis

• Reproducibility: someone using your data and code gets your results
• A low bar, yet many published findings fail it
• The reproducibility crisis hit psychology, medicine and economics hardest. No empirical field escaped
• Causes: small samples, p-hacking, selective reporting, unshared code
• This course focuses on code and data. Most computational results fail for a mundane reason

Amy Cuddy’s “power posing”: 78 million TED views, but its hormone effects failed to replicate

How bad is it? Two famous papers

Medicine

• Ioannidis (2005), one of the most cited papers in medicine
• For most study designs, a positive finding is more likely false than true
• Causes: small samples, flexible analyses, financial interests, many teams chasing one question

Psychology

• Open Science Collaboration (2015): 270 researchers re-ran 100 psychology experiments
• Replications: 36% significant, 39% rated as replicated
• Journals responded with pre-registration, open data and registered reports

We all write code, so we are fine, right?

https://www.nature.com/articles/s41597-022-01143-6

• Trisovic et al. (2022) re-ran 9,078 R files from 2,109 replication packages on Harvard Dataverse
• 74% failed on the first run. 56% still failed after automated fixes
• Common causes: missing packages, hardcoded paths, changed functions, missing data
• Authors chose to deposit these files with their papers
• Imagine the code they did not publish 😅

Five selfish reasons to work reproducibly

Markowetz (2015) argues reproducibility helps you, too:

1. Avoid disaster: catch your errors before a reviewer, a retraction or a job interview does
2. Easier writing: when the data change, every number and figure updates
3. Reviewers see it your way: they run your analysis instead of guessing
4. Continuity: a new collaborator, or future-you, can pick it up tomorrow
5. Reputation: working, shared code shows you know what you are doing

Reason 2 will save you hours in this course. Reason 1 returns in a few slides, with two famous economists

What does it take to be reproducible?

The reproducibility ladder

• These words mean different things:

• Re-runnable: the code runs again on your machine, today
• Reproducible: same code and data give the same results, on any machine
• Reusable: others can apply your work to new questions
• Replicable: new, independent data give the same finding
• Each step is harder than the one below it
• Replicability needs data you may never get, so it is mostly outside your control

Reproducibility is fully in your control, so this course grades you on it

Easier

│

│

▼

Harder

Re-runnable

Reproducible

Reusable

Replicable

In one large study, most deposited R files did not clear the first step

What a reproducible project needs

A result is reproducible only if others can get all four of these:

1. Code: every step from raw data to final number, runnable
2. Data: the raw inputs, or a script that fetches them

3. Environment: the language and package versions
4. Documentation: what the project does and the run order

Leave one out and it fails:

• No data: readers cannot check your code
• No environment: it runs today and breaks next year
• No documentation: nobody knows which script runs first

Project structure

• One folder per project, with the same layout every time
• We use it for the rest of the course, and the final project requires it

my-project/
├── data/
│   ├── raw/          # never edited by hand
│   └── clean/        # built by scripts
├── scripts/          # the code, in run order
├── output/           # figures and tables
└── README.md         # what and how

• data/raw/ holds your inputs. Everything else can be rebuilt from it
• data/raw/ is read-only: download once, never edit
• Numbered scripts (01-clean.py, 02-analyse.py) show the run order
• data/clean/ and output/ are disposable: the scripts rebuild them
• Put them in .gitignore and commit only sources
• README.md explains the project to the next person, usually you in six months

Delete everything except data/raw/ and scripts/. Can you rebuild the project?

Every project needs a README

• GitHub shows a README.md on every repository page
• It answers what is this, how do I rebuild it and in what order
• List each data source and its download date, so stale data is easy to spot
• List the rebuild commands in run order, ready to paste
• Note the language and package versions (more in a minute)
• Ten lines are enough. Start early and keep it current

# Rainfall and crop yields in Brazil

Analysis of rainfall and maize yields, 2000-2024.

## Data
- data/raw/rainfall.csv: INMET, downloaded 12 Sep 2026
- data/raw/yields.csv: IBGE, downloaded 12 Sep 2026

## How to rebuild
Run the scripts in order:

    python scripts/01-clean.py
    python scripts/02-analyse.py
    quarto render report.qmd

## Environment
Python 3.13, pandas 3.0, matplotlib 3.10

The most famous spreadsheet error in economics

• Reinhart and Rogoff (2010), “Growth in a Time of Debt”: above 90% debt-to-GDP, growth turns negative
• After 2008, it became the standard citation for austerity
• In 2013, graduate student Thomas Herndon could not reproduce it and asked for the spreadsheet
• One formula averaged rows 30 to 44, not 30 to 49, dropping five countries
• With two other fixes (excluded data, odd weighting), growth above 90% was +2.2%, not -0.1%
• The analysis sat in a private spreadsheet, so three years of policy debate passed before anyone noticed
• A public script and dataset would have caught it in an afternoon

The absolute path problem

A top reason code fails on another machine:

analysis.py

import pandas as pd

df = pd.read_csv("/Users/bob/Desktop/project/data/raw/rainfall.csv")

Anyone else who runs it gets:

FileNotFoundError: [Errno 2] No such file
or directory:
'/Users/bob/Desktop/project/data/raw/rainfall.csv'

• An absolute path starts at / and names one spot on one machine
• Only Bob has /Users/bob, so the script fails everywhere else
• A relative path starts from the working directory, like pwd in the shell
• A program inherits it from where you launch it, so “it works for me” proves little
• Hardcoded paths were a top cause of failure in the Trisovic study

Relative paths fix it

Write paths relative to the project folder:

analysis.py

import pandas as pd

df = pd.read_csv("data/raw/rainfall.csv")

Or build them with pathlib:

analysis.py

from pathlib import Path

raw = Path("data") / "raw" / "rainfall.csv"
df = pd.read_csv(raw)

print(raw.exists())  # True
print(raw.parent)    # data/raw

• pathlib joins parts with / and uses the right separator on macOS, Linux and Windows
• A Path is more than text: it can answer questions about the file, like raw.exists() (is it there?) and raw.parent (which folder is it in?)
• Read more in the pathlib documentation
• Quarto runs each document from the folder holding the .qmd file
• os.chdir() moves the working directory for the whole session, so every later relative path silently points elsewhere
• Run os.chdir("data") twice in a notebook and the second run fails
• Prefer paths built from the project root; never move around

Randomness needs a seed

Without a seed, every run gives new numbers:

import numpy as np

print(np.random.normal(size=3).round(3))
print(np.random.normal(size=3).round(3))

[ 1.221 -0.07   1.372]
[-1.314 -1.824 -2.203]

With a seed, everyone gets the same sequence:

np.random.seed(350)
print(np.random.normal(size=3).round(3))

np.random.seed(350)
print(np.random.normal(size=3).round(3))

[ 1.697 -0.533 -0.311]
[ 1.697 -0.533 -0.311]

• np.random is not truly random. It is a pseudorandom generator: a formula that walks a fixed sequence
• The seed sets the starting point, which fixes the whole sequence
• Newer code uses a generator object:

rng = np.random.default_rng(350)
rng.normal(size=3)

• Python’s random has its own generator. scikit-learn needs random_state=
• Randomness hides in sampling, train-test splits and model starting weights
• Seed every random draw

Record your environment

• Packages change: new arguments, flipped defaults, removed methods
• Code from 2024 may give different numbers in 2026, or not run. This is version drift
• Record your versions so others can match them

At minimum, note them in your README:

import sys, numpy, pandas

print("python:", sys.version.split()[0])
print("numpy :", numpy.__version__)
print("pandas:", pandas.__version__)

python: 3.13.13
numpy : 2.4.3
pandas: 3.0.2

pip lists every package. python -m uses the Python you actually run:

Terminal

python -m pip list

Package         Version
--------------- -------
jupyter_client  8.6.3
matplotlib      3.10.8
numpy           2.4.3
pandas          3.0.2

A version list helps diagnose failures. Module 08 adds tools that prevent them: virtual environments, uv and containers

Literate programming

Literate programming

• Now a tool for the biggest habit: text, code and results in one file
• Donald Knuth coined literate programming in 1984: write for humans first, machines second
• Two operations split the file:
  • Weaving: the readable document (HTML, PDF)
  • Tangling: the executable code
• Explaining your steps catches bad logic early
• Numbers come from the code, so they cannot go stale

From LaTeX to Quarto

• LaTeX came first: beautiful output, verbose syntax, steep learning curve (I’ve been there 😅)
• Markdown kept the ideas with a simpler syntax: **bold**, - list, [link](url)
• R Markdown added R chunks, and Jupyter did the same for Python
• Both struggle with mixed languages or output formats
• Quarto (2022, Posit) rebuilt R Markdown for any language: Python, R, Julia, Observable JS
• One .qmd renders to HTML, PDF, Word, slides, websites, books and dashboards
• Pandoc does the final conversion
• These slides, the course website and every course PDF use Quarto

Part of a LaTeX template. You can find more here: https://github.com/danilofreire/quarto-templates

What Quarto can produce

• Clockwise from top left: a company report, an interactive dashboard, a website, and a book
• All four are plain text, so they work well with Git
• Only the format: line changes
• Click any image to zoom in

How does Quarto work?

%%{
  init: {
    "theme": "dark",
    "themeCSS": ".label foreignObject, .cluster-label foreignObject { font-size: 90%; overflow: visible; }"
  }
}%%
flowchart LR
  A1[qmd] --> C{"knitr<br>(R)"}
  A1[qmd] --> B{"Jupyter<br>(Python)"}
  A2[ipynb] --> B{"Jupyter<br>(Python)"}
  B --> D[md]
  C --> D[md]
  D --> E{Pandoc}
  E --> F[pdf]
  E --> G[docx]
  E --> H[html]
  E --> I[...]

  subgraph engine [Engine]
  B
  C
  end

• Quarto reads .qmd files and Jupyter notebooks (.ipynb). A YAML cell on top is optional
• Pipeline: .qmd → engine (Jupyter or knitr) → .md → Pandoc → output
• Pandoc turns Markdown into HTML, PDF, Word and dozens of other formats
• Quarto calls Pandoc for you, with the options from your YAML header

Getting Quarto running 🛠️

Installing Quarto

• Download the installer from the Quarto website, or use a package manager
• On macOS, with Homebrew:

Terminal

brew install --cask quarto

• On Windows, install Quarto inside WSL, next to your shell and Git:

WSL terminal

wget https://quarto.org/download/latest/quarto-linux-amd64.deb
sudo apt install ./quarto-linux-amd64.deb

• PDFs also need LaTeX. Install the lightweight TinyTeX:

Terminal

quarto install tinytex

Did it work? Run quarto check

Terminal

quarto check

Quarto 1.10.18
[✓] Checking environment information...
[✓] Checking versions of quarto binary dependencies...
      Pandoc version 3.10: OK
      Dart Sass version 1.101.0: OK
[✓] Checking Quarto installation......OK
      Version: 1.10.18
[✓] Checking tools....................OK
      TinyTeX: v2026.04
[✓] Checking LaTeX....................OK
[✓] Checking basic markdown render....OK
[✓] Checking Python 3 installation....OK
      Version: 3.13.13
      Jupyter: 5.9.1

Run this first when something breaks. Your versions may differ from mine

Write in whatever editor you like

• A .qmd file is plain text: VS Code, RStudio, Jupyter Lab, Neovim, even Notepad work
• We use VS Code with the Quarto extension: highlighting, completion, a render button and live preview
• Install it from the Extensions marketplace
• Its visual editor renders Markdown as you type. Try both modes

Quarto extension for VS Code

Rendering from the terminal

quarto --help lists every command. These do most of the work:

Terminal

## Render a file to its default format
quarto render report.qmd

## Render to a specific format
quarto render report.qmd --to pdf
quarto render report.qmd --to html
quarto render report.qmd --to docx

## Render to slides
quarto render report.qmd --to revealjs

## Convert a Jupyter notebook to .qmd (and back)
quarto convert notebook.ipynb
quarto convert report.qmd

## Render, open in the browser, and re-render on every save
quarto preview report.qmd

• render builds the document once, when you are done
• preview keeps a live version open while you write
• Output lands next to the source (report.qmd → report.html), unless a project sets output-dir

Your first Quarto document

Anatomy of a Quarto document

• A Quarto document has a YAML header, code chunks and Markdown text
• We look at each in turn. Markdown in depth comes next lecture

The YAML header

---
title: "Rainfall report"
author: "Your name"
date: 2026-10-06
format:
  html:
    toc: true
    code-fold: true
execute:
  warning: false
jupyter: python3
---

• YAML stands for “YAML Ain’t Markup Language” (a recursive joke). It holds the document’s metadata and settings
• It sits at the top, between two --- lines
• key: value pairs, where indentation creates nesting. Use spaces: tabs are illegal for indentation
• format: html is the short form. Indent under it to add options
• execute: sets defaults for every chunk
• Quote values that contain a colon: title: "Quarto: a first look"
• _quarto.yml, GitHub Actions and Docker Compose use the same syntax

Code chunks

```{python}
#| label: fig-rainfall
#| echo: false
#| warning: false

import matplotlib.pyplot as plt

plt.bar(rain["month"], rain["mm"])
plt.ylabel("Rainfall (mm)")
plt.show()
```

• Three backticks and {python} open a chunk; three backticks close it
• On render, the chunk runs and its output goes into the document
• #| lines are chunk options, in YAML syntax
• Here they label the figure and hide the code and warnings
• More here

The chunk options you will use most

Option	Effect
echo: false	hide the code, keep the output
eval: false	show the code, do not run it
include: false	run the code, show nothing
warning: false	hide warnings
label: fig-rain	name the chunk for cross-references
fig-cap: "..."	put a caption under the figure
layout-ncol: 2	arrange the outputs in columns

• #| options apply to one chunk
• Document-wide defaults go in the YAML header:

execute:
  echo: false
  warning: false

• Chunk options override these, so you can hide all code but show one chunk
• Cross-references need a prefixed label: fig- (figures), tbl- (tables), eq- (equations), sec- (sections), lst- (code listings)
• @fig-rain then prints “Figure 1”, numbered for you
• Full list in the Quarto documentation

Chunk options in action

layout-ncol: 2 puts two plots side by side; echo: false hides the code:

```{python}
#| echo: false
#| layout-ncol: 2

plt.bar(rain["month"], rain["mm"])
plt.show()

plt.plot(rain["month"], rain["mm"])
plt.show()
```

What the reader sees:

Both figures come from the code at render time. Change the data and they update

Engines: what actually runs your code

• Quarto runs no code itself. It hands chunks to an engine
• Jupyter runs Python and any language with a Jupyter kernel. knitr runs R
• This course uses Jupyter. These slides render with jupyter: python3
• Each render starts one fresh session and runs the chunks top to bottom
• Chunks share state like notebook cells: a variable from chunk one exists in chunk ten
• So every render is a restart-and-run-all test
• A document that renders has just run its analysis from scratch

---
title: "My report"
format: html
jupyter: python3
---

Quarto can usually detect the engine. Declaring it tells readers what to install.

To use R, write {r} chunks and Quarto picks knitr

A minimal Python document

---
title: "Palmer Penguins Demo"
format:
  html:
    code-fold: true
jupyter: python3
---

## Meet Quarto {#sec-introduction}

Quarto enables you to weave together content and
executable code into a finished document. To learn
more about Quarto see <https://quarto.org>.

```{python}
#| echo: false

import seaborn as sns
from palmerpenguins import load_penguins
sns.set_style('whitegrid')

penguins = load_penguins()

g = sns.lmplot(x="flipper_length_mm",
               y="body_mass_g",
               hue="species",
               height=7,
               data=penguins,
               palette=['#FF8C00','#159090','#A034F0'])
g.set_xlabels('Flipper Length')
g.set_ylabels('Body Mass')
```

The chunk in @sec-introduction plots body mass
against flipper length.

• Under forty lines, with everything from today’s lecture
• The YAML header sets the title, format and engine
• code-fold: true hides code behind a toggle, but echo: false removes this chunk’s code
• The prose is Markdown. {#sec-introduction} labels a heading and @sec-introduction cites it
• One chunk loads penguin data and draws a figure (needs pip install palmerpenguins)
• The next slide shows the result

The rendered page

• Title from the YAML, heading and prose from the Markdown, figure from the chunk
• echo: false wins over code-fold: true: only the figure shows
• @sec-introduction became a link reading “Section 1”, which jumps to the heading
• Change the data and render again: the figure and numbers follow
• The page can only show figures its own code produced

Try it yourself!

Create a Quarto document from scratch and render it to HTML.

1. Make a new file called weather.qmd
2. Start it with this YAML header:

---
title: "Rainfall report"
author: "Your name"
format: html
jupyter: python3
---

3. Add a ## heading and a short paragraph
4. Add a chunk that builds the table below and prints it
5. Add a second chunk that draws a bar chart, with its code hidden
6. Render the file and open the result

Use this data so everyone gets the same answer:

import pandas as pd

rain = pd.DataFrame({
    "month": ["Jan", "Feb", "Mar", "Apr"],
    "mm": [241, 215, 180, 96]
})

Stuck? See the chunk options slide. quarto check diagnoses a broken installation

Answer: Appendix 01

Summary

Summary

• The reproducibility crisis is real. Most computational results fail because the code does not run
• The ladder: re-runnable → reproducible → reusable → replicable. This course gets you to step two
• Code, data, environment and documentation must all be shared
• Habits: untouched raw data, cleaning in code, relative paths, seeds, recorded versions, a README
• Literate programming keeps text, code and results in one file
• Quarto is our tool: one plain-text .qmd, one render command, many formats

Next class

• Quarto in practice: Markdown, PDFs and automatic citations
• freeze: rendering without re-running everything
• Slides (like these) and websites, published from GitHub
• Parameterised reports: one file, one report per country
• Please bring a working Quarto installation. If something doesn’t work, let me know 🤓

Next class: websites like this one, built with Quarto

Additional materials

• Quarto documentation
• Quarto gallery, full example documents with their source code
• Awesome Quarto, a curated list of examples and extensions
• Introduction to Quarto (video)
• Quarto for academics (video)
• Beautiful reports and presentations with Quarto (video - worth watching)
• The Turing Way, a community handbook on reproducible data science
• Sandve et al. (2013), “Ten Simple Rules for Reproducible Computational Research”, still true a decade later
• Markowetz (2015), “Five selfish reasons to work reproducibly”
• “The student who caught out the profs” (BBC), the Reinhart and Rogoff story

And that’s all for today! 🥳

Thank you very much and see you soon! 😊 🙏🏼

Appendix 01: Solution to the exercise

The complete weather.qmd:

---
title: "Rainfall report"
author: "Your name"
format: html
jupyter: python3
---

## Rainfall in the first four months

Rainfall fell steadily from January to April, with April
receiving less than half of January's total.

```{python}
import pandas as pd

rain = pd.DataFrame({
    "month": ["Jan", "Feb", "Mar", "Apr"],
    "mm": [241, 215, 180, 96]
})

rain
```

```{python}
#| echo: false

import matplotlib.pyplot as plt

plt.bar(rain["month"], rain["mm"], color="#1B3A6B")
plt.ylabel("Rainfall (mm)")
plt.show()
```

Then render it:

Terminal

quarto render weather.qmd

#| echo: false hides the code and keeps the output, as task 5 asked.

The chart your second chunk produces:

Back to the main text