DATASCI 350 - Data Science Computing

Lecture 21 - Parallel Computing Fundamentals

Danilo Freire

danilo.freire@emory.edu

Department of Data and Decision Sciences
Emory University

Hello again, my friends! 😊

Brief recap 📚

Module 06: getting data from the web

Lectures 18 and 19

A web API is a contract between your code and a server
We built URLs, read status codes and followed JSON paths
curl and requests fetched the data
A for loop turned the reply into a DataFrame
A saved snapshot gives the same numbers every time

Module 07: what if the data get really big?
Three possible fixes:
- Parallel code: several cores at once
- A better algorithm
- A faster engine, like DuckDB (next lecture)

Lecture 19’s wdi_panel.parquet uses the parquet format we meet today

Tweet of the day

About Jensen Huang

Source: X

Today’s agenda

Lecture outline

Serial and parallel code, and the map function
joblib and concurrent.futures: parallel work on your laptop
Big O notation: what parallelism can and cannot fix
Amdahl’s law: the limit on speed-up
Dask for data bigger than your memory
- Lazy arrays and DataFrames
- CSV and parquet files
- dask.delayed for your own functions
Best practices: when to parallelise, and when not to

Serial vs parallel algorithms

Serial execution and the `map` function

Python runs one line after another
This loop calls bar on each number, one at a time:

import numpy as np

# An array of six numbers
foo = np.array([0, 1, 2, 3, 4, 5])

# A function that squares a number
def bar(x):
    return x * x

# Call bar on each element, in order
for element in foo:
    print(bar(element))

map applies one function to every element of a list
My own version shows the loop inside:

def my_map(function, array):
    output = []           # empty list for the results
    for element in array:
        output.append(function(element))
    return output

my_map(bar, foo)

[np.int64(0),
 np.int64(1),
 np.int64(4),
 np.int64(9),
 np.int64(16),
 np.int64(25)]

Python has a built-in map (faster than mine):

list(map(bar, foo))

[np.int64(0),
 np.int64(1),
 np.int64(4),
 np.int64(9),
 np.int64(16),
 np.int64(25)]

Using `joblib` for parallel computing

Each call to bar is independent, so the calls can run at once
Your CPU has several cores. Each runs one task at a time
joblib sends tasks to worker processes: separate copies of Python
Parallel(n_jobs=6) runs up to six tasks at once. n_jobs=-1 uses all cores
delayed(bar) wraps bar, so a worker runs it later
delayed(bar)(x) for x in foo is a generator expression: a for loop inside parentheses that makes one task per element
More information about joblib on their website: https://joblib.readthedocs.io/en/stable/

With bar and foo from before:

# Install joblib if you haven't done so yet
# !pip install joblib

from joblib import Parallel, delayed

results = Parallel(n_jobs=6)(
    delayed(bar)(x) for x in foo
)
results

[np.int64(0),
 np.int64(1),
 np.int64(4),
 np.int64(9),
 np.int64(16),
 np.int64(25)]

Each worker gets a different element of foo
The results match the serial version

Serial vs parallel execution

Our function calculation (below) runs ten heavy steps on ten million random numbers
The calls are independent, so the problem is embarrassingly parallel
%timeit averages a line over several runs (Jupyter only)
%%timeit (two % signs) times a whole cell
One call:

def calculation(size=10000000):
    # Create a large array and perform operations
    arr = np.random.rand(size)
    for _ in range(10):
        arr = np.sqrt(arr) + np.sin(arr)
    return np.mean(arr)

# Single run
%timeit calculation()

333 ms ± 5.25 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Four calls, one after another:
_ names a loop variable we do not use

# Sequential runs (4 times)
%timeit [calculation() for _ in range(4)]

1.34 s ± 8.59 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Four calls, in parallel:

%%timeit
# Parallel runs (4 times)
Parallel(n_jobs=4)(
    delayed(calculation)() for _ in range(4)
)

471 ms ± 4.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Not a full 4× speed-up, because starting workers and collecting results takes time

concurrent.futures

`concurrent.futures`: the standard-library option

concurrent.futures comes with Python: nothing to install
Computing (maths, model training): ProcessPoolExecutor
Waiting (downloads, API calls): ThreadPoolExecutor
Processes are separate copies of Python that compute at the same time
Threads share one copy of Python, so they start fast. While one waits, the others keep working

from concurrent.futures import ProcessPoolExecutor

with ProcessPoolExecutor(max_workers=4) as executor:
    results = list(executor.map(bar, foo))

In a notebook, import bar from a .py file. Workers cannot see functions defined in a cell, but joblib can

Threads start fast and need no extra core to wait
Each URL asks the World Bank API for one country’s population

from concurrent.futures import ThreadPoolExecutor
import requests

base = "https://api.worldbank.org/v2/country/"
end = "/indicator/SP.POP.TOTL?format=json&per_page=100"
urls = [base + "BRA" + end,
        base + "IND" + end,
        base + "USA" + end]

def fetch(url):
    return requests.get(url, timeout=30).json()

with ThreadPoolExecutor(max_workers=3) as executor:
    results = list(executor.map(fetch, urls))

All three requests wait at once (Lecture 19 sent one at a time)

When to use which?

joblib suits numeric loops, and scikit-learn uses it when you set n_jobs=-1
load_digits() loads 1,797 small images of handwritten digits
6 combinations × 3 folds (cv=3) = 18 fits shared across the cores

from sklearn.datasets import load_digits
from sklearn.model_selection import GridSearchCV
from sklearn.svm import SVC
import time

digits = load_digits()
param_grid = {"C": [0.1, 1, 10], "gamma": [0.001, 0.01]}

start = time.time()
search = GridSearchCV(SVC(), param_grid, cv=3, n_jobs=-1)
search.fit(digits.data, digits.target)
elapsed = time.time() - start

print(f"Best score: {search.best_score_:.3f}")
print(f"Time: {elapsed:.2f}s (all cores)")

Best score: 0.976
Time: 1.97s (all cores)

Scenario	Tool
sklearn / numeric loops	`joblib` (`n_jobs=-1`)
Download many files	`ThreadPoolExecutor`
CPU-heavy, no dependencies	`ProcessPoolExecutor`
Bigger than RAM, clusters	`Dask` (next section)

Every tool splits the work, runs it on separate workers, then collects the results

Big O notation

Big O notation describes how runtime grows with input size
O(1) is constant: reading one array element takes the same time at any size
A single loop is O(n), linear: double the input, double the time
A nested loop is O(n²), quadratic: double the input, four times the time
Calling calculation n times is O(n): 100 calls take 100 times as long as one

Code

import matplotlib.pyplot as plt
import numpy as np

# Simulate processing times
num_images = np.array([1, 10, 50, 100, 200])
sequential_time = num_images * 2  # 2 seconds per image
parallel_time = (num_images * 2) / 4  # 4 cores, ideal speedup

plt.figure(figsize=(8, 5))
plt.plot(num_images, sequential_time, 'o-', label='Serial O(n)', linewidth=2, markersize=8)
plt.plot(num_images, parallel_time, 's-', label='Parallel on 4 cores (still O(n))', linewidth=2, markersize=8)
plt.xlabel('Number of Images', fontsize=12)
plt.ylabel('Time (seconds)', fontsize=12)
plt.title('O(n) Scaling: Serial vs Parallel', fontsize=14, fontweight='bold')
plt.legend(fontsize=11)
plt.grid(True, alpha=0.3)
plt.show()

What parallelism does to complexity

With p cores, an O(n) task takes about n/p time plus overhead. It is still O(n)
Best candidates: embarrassingly parallel O(n) tasks, which share no data and do not wait for each other

Parallelism divides the time by a constant but never changes the Big O class. O(n²) on eight cores is still O(n²)

Parallelism cannot fix this:

# O(n^2): every pair of elements
def pairwise_sum(data):
    n = len(data)
    results = []
    for i in range(n):
        for j in range(n):
            results.append(data[i] + data[j])
    return results

data = list(range(1_000))
print(f"{len(pairwise_sum(data)):,} operations")

1,000,000 operations

1,000 elements give 1 million operations
n = 100_000 gives 10 billion. Eight cores still leave 1.25 billion each
Poor candidates for parallelism:
- Steps that need results from a previous step
- Tasks that share the same data
- Tasks limited by memory, not by the CPU

Try it yourself!

Run pip install joblib numpy if you have not already
Run the serial version below and write down the time it prints. np.arange(1000000) makes an array of the numbers 0 to 999,999

import numpy as np

def square(x):
    return x**2

# Create a large array to process
numbers = np.arange(1000000)

# Sequential version
%timeit [square(x) for x in numbers]

Run the parallel version and write down its time. It takes a few seconds

from joblib import Parallel, delayed

# Parallel version
%timeit Parallel(n_jobs=4)(delayed(square)(x) for x in numbers)

Compare the two times and decide which version you would use

Appendix 01

Amdahl’s law

Amdahl’s law: the speed-up ceiling

Some parts of a program cannot run in parallel: setup, merging results, or steps that wait for the one before
That part is the serial share \(s\). The rest, \(p = 1 - s\), can be split across cores
More cores only speed up \(p\), so \(s\) sets a ceiling (Amdahl’s law)
\(\text{speedup} = \dfrac{1}{s + \dfrac{p}{N}}\), with \(N\) = cores
Example: 90% parallel (\(s = 0.1\)) on 8 cores gives \(\frac{1}{0.1 + 0.9/8} = \frac{1}{0.2125} \approx 4.7\times\), not \(8\times\)
With infinite cores, the limit is \(1/0.1 = 10\times\)

Code

import matplotlib.pyplot as plt
import numpy as np

cores = np.arange(1, 65)
serial_fracs = [0.05, 0.2, 0.5]
labels = ["95% parallel", "80% parallel", "50% parallel"]
colours = ["#1B3A6B", "#E07A24", "#2CA02C"]

plt.figure(figsize=(8, 5))
for s, lab, c in zip(serial_fracs, labels, colours):
    speedup = 1 / (s + (1 - s) / cores)
    plt.plot(cores, speedup, label=lab, linewidth=2, color=c)

plt.xlabel("Number of cores", fontsize=12)
plt.ylabel("Speed-up (×)", fontsize=12)
plt.title("Amdahl's law: speed-up vs core count",
          fontsize=14, fontweight="bold")
plt.legend(fontsize=11)
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()

Ten people chop vegetables faster than one, but the oven still takes 10 minutes. Dinner can never be faster than the oven

Dask

Dask is a Python library for parallel computing
It handles data bigger than your RAM and runs on clusters of many machines
You write NumPy or pandas code, and Dask runs it in parallel
A task scheduler splits the work into small tasks and spreads them over your cores
Dask offers arrays, DataFrames and bags (parallel lists)
Latest release: 2026.8.0 (August 2026)
Next lecture times the same group-by in DuckDB, pandas and Dask

https://www.dask.org/

joblib splits the work into function calls. Dask also splits the data into chunks, so the data can be bigger than your RAM

Dask arrays

import dask.dataframe as dd
import dask.array as da

A Dask array splits a large array into chunks
Each chunk is a NumPy array, processed in parallel
Here each chunk is 100 by 100:

data = np.random.normal(size=100000).reshape(200, 500)
a = da.from_array(data, chunks=(100, 100))
a

	Array	Chunk
Bytes	781.25 kiB	78.12 kiB
Shape	(200, 500)	(100, 100)
Dask graph	10 chunks in 1 graph layer
Data type	float64 numpy.ndarray

Dask arrays are lazy: Dask plans the steps first, which saves memory
Slicing gives a plan, not numbers:

a[:10, 5] # first 10 rows of the 6th column

	Array	Chunk
Bytes	80 B	80 B
Shape	(10,)	(10,)
Dask graph	1 chunks in 2 graph layers
Data type	float64 numpy.ndarray

.compute() runs the plan:

a[:10, 5].compute()

array([-1.22254226, -0.20602161, -1.73760655, -0.47669143,  0.73397732,
        0.79288256,  0.36582556, -0.61382531, -0.66821034, -0.26960348])

Dask arrays

Maths (+, *, exp, log) and summaries (sum(), mean(), std()) work as in NumPy (full list)

a.sum().compute()

np.float64(-205.97997407969373)

sqrt plus sin on 100 million numbers, first in NumPy:

size = 100000000
np_arr = np.random.random(size)
%timeit np_result = np.sqrt(np_arr) + np.sin(np_arr)

395 ms ± 3.56 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Then in Dask, with automatic chunks:

da_arr = da.random.random(size, chunks='auto')
%timeit (da.sqrt(da_arr) + da.sin(da_arr)).compute()

239 ms ± 8.23 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

NumPy uses one core here and Dask uses several. On small data, Dask is often slower

Dask dataframes

A Dask DataFrame is many small pandas DataFrames, split by row
dask.datasets.timeseries() makes random example data: one row per second from 1 to 30 January 2000

import dask
df = dask.datasets.timeseries()

Computed, it would take about 120 MB of memory
Dask DataFrames are lazy, like Dask arrays
The table shows ..., because no rows have been computed yet

df

Dask DataFrame Structure:

	name	id	x	y
npartitions=30
2000-01-01	string	int64	float64	float64
2000-01-02	...	...	...	...
...	...	...	...	...
2000-01-30	...	...	...	...
2000-01-31	...	...	...	...

Dask Name: to_string_dtype, 2 expressions

Dask already knows the column names and dtypes (column types)

df.dtypes

name     string
id        int64
x       float64
y       float64
dtype: object

Dask dataframes

Dask DataFrames support most of the pandas API (full list)
head() is not lazy: it computes the first rows straight away

import pandas as pd

pd.options.display.precision = 2
pd.options.display.max_rows = 10

df.head()

	name	id	x	y
timestamp
2000-01-01 00:00:00	Quinn	974	-0.11	0.59
2000-01-01 00:00:01	Yvonne	1011	-0.12	-0.58
2000-01-01 00:00:02	Alice	970	-0.59	0.77
2000-01-01 00:00:03	Quinn	1016	0.29	0.05
2000-01-01 00:00:04	Michael	1023	0.88	0.50

Keep rows where y > 0, then take the standard deviation of x by name

df2 = df[df.y > 0]
df3 = df2.groupby("name").x.std()
df3

Dask Series Structure:
npartitions=1
    float64
        ...
Dask Name: getitem, 8 expressions
Expr=(((Filter(frame=ArrowStringConversion(frame=Timeseries(5afcdfc)), predicate=ArrowStringConversion(frame=Timeseries(5afcdfc))['y'] > 0))[['name', 'x']]).std(ddof=1, numeric_only=False, split_out=None, observed=True))['x']

df3 shows no numbers until .compute() runs the plan and returns a pandas Series

df3.compute()

name
Bob         0.58
Dan         0.58
Edith       0.58
George      0.58
Hannah      0.58
            ... 
Patricia    0.58
Tim         0.58
Ursula      0.58
Victor      0.58
Xavier      0.58
Name: x, Length: 26, dtype: float64

Dask dataframes

Aggregations work the same way: the sum of x and the maximum of y for each name

df4 = df.groupby("name").aggregate({"x": "sum", "y": "max"})
df4.compute()

	x	y
name
Kevin	233.65	1.0
Ingrid	2.00	1.0
Patricia	158.18	1.0
Alice	-310.16	1.0
Sarah	223.77	1.0
...	...	...
Jerry	192.04	1.0
Oliver	5.75	1.0
Norbert	23.32	1.0
Hannah	392.09	1.0
Wendy	-231.32	1.0

26 rows × 2 columns

.persist() computes a result once and keeps it in memory for later steps
Persist small results, not the full 120 MB table

df5 = df4.persist()
df5.head()

	x	y
name
Kevin	233.65	1.0
Ingrid	2.00	1.0
Patricia	158.18	1.0
Alice	-310.16	1.0
Sarah	223.77	1.0

Combining Dask and pandas

Dask does the heavy work on data too big for memory
Dask code reads like pandas, so switching is easy

Load and filter in Dask, aggregate to something small, call .compute(), then finish in pandas

import dask
import dask.dataframe as dd

df = dask.datasets.timeseries()

# Step 1: filter with Dask
positive = df[df.x > 0]

# Step 2: aggregate with Dask
summary = positive.groupby("name").agg({"x": "mean", "y": "std"})

# Step 3: bring the small result to pandas
pdf = summary.compute()

# Step 4: use pandas normally
pdf.sort_values("x", ascending=False).head(5)

	x	y
name
Patricia	0.5	0.58
Zelda	0.5	0.58
Ursula	0.5	0.58
Laura	0.5	0.58
Michael	0.5	0.58

Try it yourself!

Install Dask with pip install dask if you have not already
Import numpy and dask.array
Set size = 10_000_000 (smaller if your laptop is short on memory)
Build a Dask array of that many random numbers with chunks=100_000
Time da.sqrt(x**2).mean().compute() with %timeit
Repeat steps 4 and 5 with chunks=2_000_000, then with chunks=5_000_000
Write down the three timings before you look at the appendix

import numpy as np
import dask.array as da

size = 10_000_000

# Dask with SMALL chunks
da_data_small = da.random.random(size, chunks=100_000)  # 100 chunks
%timeit da.sqrt(da_data_small**2).mean().compute()

# Dask with MEDIUM chunks
da_data_medium = da.random.random(size, chunks=2_000_000)  # 5 chunks
%timeit da.sqrt(da_data_medium**2).mean().compute()

# Dask with LARGE chunks
da_data_large = da.random.random(size, chunks=5_000_000)  # 2 chunks
%timeit da.sqrt(da_data_large**2).mean().compute()

Appendix 02

Read and write data with Dask

Reading and writing data

CSV is still the most common data format
pandas loads the whole file into memory, even if you need one column
Dask reads and writes CSV in partitions, pieces of the table
Here, one partition per day. Dask replaces the * in 'data/*.csv' with each name
name(i) turns partition i into a date, giving 30 files (1 to 30 January 2000)
:02d pads the day with a zero, so 1 becomes 01
The ; hides the file list that to_csv returns

df = dask.datasets.timeseries()
df

Dask DataFrame Structure:

	name	id	x	y
npartitions=30
2000-01-01	string	int64	float64	float64
2000-01-02	...	...	...	...
...	...	...	...	...
2000-01-30	...	...	...	...
2000-01-31	...	...	...	...

Dask Name: to_string_dtype, 2 expressions

import os

# Create the data folder (no error if it already exists)
os.makedirs("data", exist_ok=True)

# Partition i becomes the file for day i + 1
# (2000-01-01.csv, 2000-01-02.csv, ...)
def name(i):
    return f"2000-01-{i + 1:02d}"

df.to_csv("data/*.csv", name_function=name);

Reading and writing data

The data folder now has 30 CSV files, 182 MB in total
dd.read_csv accepts a glob pattern (the wildcards from Lecture 04): 2000-*-*.csv matches them all
Dask reads them as one DataFrame, a few files at a time and in parallel, so memory use stays low

df = dd.read_csv('data/2000-*-*.csv')
df

Dask DataFrame Structure:

	timestamp	name	id	x	y
npartitions=30
	string	string	int64	float64	float64
	...	...	...	...	...
...	...	...	...	...	...
	...	...	...	...	...
	...	...	...	...	...

Dask Name: to_string_dtype, 2 expressions

A group-by across all 30 files:

%timeit df.groupby('name').x.mean().compute()

724 ms ± 7.05 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Reading and writing data

Parquet is the usual format for large analytical data
It stores data by column, so a query can skip the columns it does not need
It compresses too: 182 MB of CSV becomes 86 MB of parquet (53% smaller)

df.to_parquet('data/2000-01.parquet')

Apache Parquet on GitHub · Documentation

Read back only the columns you need:

df = dd.read_parquet('data/2000-01.parquet',
                     columns=['name', 'x'])
df

Dask DataFrame Structure:

	name	x
npartitions=30
	string	float64
	...	...
...	...	...
	...	...
	...	...

Dask Name: read_parquet, 1 expression

The same group-by, on parquet:

%timeit df.groupby('name').x.mean().compute()

68.8 ms ± 1.52 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

Why parquet?

CSV is row-based: reading one column still scans every row
Parquet is column-based: it reads only the columns you name

Feature	CSV	Parquet
Storage	Row-based	Column-based
Compression	None	Snappy/gzip
Column selection	Reads all	Reads only needed
Data types	Text only	Typed (int, float, date)
Our January 2000 data	182 MB	86 MB

pandas reads parquet files too:

import pandas as pd

# CSV: reads every column, guesses the types
one_day = pd.read_csv("data/2000-01-01.csv")
print(one_day.shape)

# Parquet: reads only the columns you name
month = pd.read_parquet("data/2000-01.parquet",
                        columns=["name", "x"])
print(month.shape)

(86400, 5)
(2592000, 2)

CSV loads one day with all 5 columns. Parquet loads 30 days but only the 2 columns we named

Use parquet for files over 100 MB, or with many columns when you need only a few. Keep CSV for small files and anything a colleague will open in Excel

Dask delayed

dask.delayed parallelises functions you already have
No Dask DataFrame or array needed
First, four calls in a row:

def calculation(size=10000000):
    arr = np.random.rand(size)
    for _ in range(10):
        arr = np.sqrt(arr) + np.sin(arr)
    return np.mean(arr)

%timeit [calculation() for _ in range(4)]

1.37 s ± 46.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

A decorator is an @ line above a def. It changes what the function does
With @dask.delayed, calling the function creates a task instead of running it

@dask.delayed
def delayed_calculation(size=10000000):
    arr = np.random.rand(size)
    for _ in range(10):
        arr = np.sqrt(arr) + np.sin(arr)
    return np.mean(arr)

results = []
for _ in range(4):
    results.append(delayed_calculation())

# Compute all results at once
%timeit final_results = dask.compute(*results)

510 ms ± 11 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

dask.compute(*results) runs all four tasks. * unpacks the list into four arguments

Dask delayed

Three functions on three input sizes: nine tasks in three branches

@dask.delayed
def generate_data(size):
    return np.random.rand(size)

@dask.delayed
def transform_data(data):
    return np.sqrt(data) + np.sin(data)

@dask.delayed
def aggregate_data(data):
    return np.mean(data)

# Build the tasks (nothing runs yet)
dask_results = []
for size in [1_000_000, 2_000_000, 3_000_000]:
    data = generate_data(size)
    transformed = transform_data(data)
    dask_results.append(aggregate_data(transformed))

%timeit dask.compute(*dask_results)

23.1 ms ± 636 μs per loop (mean ± std. dev. of 7 runs, 10 loops each)

dask.visualize() draws the plan without running it

# Needs graphviz: pip install graphviz
# plus brew/apt install graphviz
dask.visualize(*dask_results)

Dask sees all nine tasks first. The branches are independent, so they can run on separate workers

The Dask dashboard

from dask.distributed import Client, then client = Client(), opens a dashboard at localhost:8787
Install it first: pip install "dask[distributed]" bokeh
The picture shows our three-step pipeline, run in a loop
Each row is one worker thread, and each coloured bar is one task
Purple is generate_data, teal is transform_data, and green is aggregate_data
Bars stacked on top of each other ran at the same time
Use it to check that your work runs in parallel

Dashboard at localhost:8787, running on my laptop

White space means an idle thread. Big gaps usually mean the chunks are too big, so there are fewer tasks than threads

Best practices

Measure first with %timeit. Amdahl’s law says what fixing the slow part is worth
Check that tasks are independent and big enough to be worth a worker
A better algorithm often beats parallel code on large inputs
A faster engine like DuckDB may beat parallel pandas (next lecture)
NumPy or pandas may already have a fast built-in function
Do you need all the data? Try a sample first

Our parquet group-by was about 10× faster than CSV
Call .compute() and .persist() rarely: each call runs the whole plan
Try a few chunk sizes, or start with chunks='auto'
More tips in the Dask best practices

The most common mistake is parallelising before measuring. More cores will not fix a slow disk or an \(O(n^2)\) loop

Summary

What we learned today

Serial and parallel code: map, embarrassingly parallel problems, joblib
concurrent.futures: processes for CPU work, threads for waiting
Big O notation: parallelism divides the time, but the growth rate stays the same
Amdahl’s law: the serial share caps the speed-up
Dask: lazy arrays and DataFrames for data bigger than RAM
dask.delayed parallelises your own functions, and dask.visualize shows the plan
Parquet: our data shrank from 182 MB to 86 MB, and the group-by ran about 10× faster
Best practices: measure before you parallelise

Next lecture, SQL Revision with DuckDB: SQL on the WDI panel, and the same group-by in DuckDB, pandas and Dask

Engine	Language	Parallel?	Bigger than RAM?
pandas	Python/C	Single-threaded	No
Polars	Rust	All cores	Yes (streaming)
DuckDB	C++	All cores	Yes
Dask	Python	All cores + clusters	Yes

Polars is covered in tutorial 07
SQL on DataFrames before then: tutorial 04

Thank you! 😊

Appendix 01

def square(x):
    return x**2

# Create a large array to process
numbers = np.arange(1000000)

# Sequential version
%timeit [square(x) for x in numbers]

44.6 ms ± 604 μs per loop (mean ± std. dev. of 7 runs, 10 loops each)

from joblib import Parallel, delayed

# Parallel version
%timeit Parallel(n_jobs=4)(delayed(square)(x) for x in numbers)

1.9 s ± 55.8 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

The parallel version is slower here
square(x) is so fast that sending a million tiny tasks to the workers costs more than the work
Parallel computing pays off when each task is heavy

Back to exercise

Appendix 02

Ten million random numbers, computing mean(sqrt(x^2)) at four chunk sizes
Medium chunks (2 million) win on my laptop

size = 10_000_000

# Dask with SMALL chunks  # 100 chunks
da_data_small = da.random.random(size, chunks=100_000)
%timeit da.sqrt(da_data_small**2).mean().compute()

39.5 ms ± 800 μs per loop (mean ± std. dev. of 7 runs, 10 loops each)

# Dask with MEDIUM chunks  # 5 chunks
da_data_medium = da.random.random(size, chunks=2_000_000)
%timeit da.sqrt(da_data_medium**2).mean().compute()

13.6 ms ± 179 μs per loop (mean ± std. dev. of 7 runs, 100 loops each)

# Dask with LARGE chunks  # 2 chunks
da_data_large = da.random.random(size, chunks=5_000_000)
%timeit da.sqrt(da_data_large**2).mean().compute()

20.4 ms ± 826 μs per loop (mean ± std. dev. of 7 runs, 10 loops each)

# Dask with AUTO chunks  # Dask picks
da_data_auto = da.random.random(size, chunks='auto')
%timeit da.sqrt(da_data_auto**2).mean().compute()

31.6 ms ± 379 μs per loop (mean ± std. dev. of 7 runs, 10 loops each)

Back to exercise

Appendix 02: why medium chunks win

Every chunk becomes a few tasks, and each task has a fixed overhead: Dask plans it, sends it to a worker and collects the result
Small chunks (100): hundreds of tiny tasks, so most of the time goes on overhead
Large chunks (2): little overhead, but only 2 cores have work
Medium chunks (5): 5 cores busy, few tasks to manage
Auto aims for about 128 MB per chunk. Our 80 MB array becomes one chunk on one core
Best size: big enough that overhead is small, small enough that every core gets work
My laptop has 10 cores, so about 10-20 chunks would probably be faster still

Back to exercise

DATASCI 350 - Data Science Computing

Hello again, my friends! 😊

Brief recap 📚

Module 06: getting data from the web

Lectures 18 and 19

Tweet of the day

Today’s agenda

Lecture outline

Serial vs parallel algorithms

Serial execution and the map function

Using joblib for parallel computing

Serial vs parallel execution

concurrent.futures

concurrent.futures: the standard-library option

When to use which?

Big O notation

Big O notation

What parallelism does to complexity

Try it yourself!

Amdahl’s law

Amdahl’s law: the speed-up ceiling

Dask

Dask

Dask arrays

Dask arrays

Dask dataframes

Dask dataframes

Dask dataframes

Combining Dask and pandas

Try it yourself!

Read and write data with Dask

Reading and writing data

Reading and writing data

Reading and writing data

Why parquet?

Dask delayed

Dask delayed

Dask delayed

The Dask dashboard

Best practices

Best practices

Summary

What we learned today

Thank you! 😊

Appendix 01

Appendix 02

Appendix 02: why medium chunks win

Serial execution and the `map` function

Using `joblib` for parallel computing

`concurrent.futures`: the standard-library option