• Launched an EC2 t3.micro running Ubuntu Server 26.04, on the free plan ($100 in credits, up to $100 more)
• Created a key pair, ran chmod 400 on the private key, and connected with ssh -i
• Met AWS CloudShell, the browser terminal
• Installed software with apt
• Moved files with scp and wget, then forwarded port 8888 so Jupyter on the instance opened in our own browser
• Stopping an instance pauses billing but releases the public IP; terminating erases the machine
• A t3.micro left running costs about $7.50 a month

• Check that no instance you are not using is still running
• Terminate your EC2 instances and delete your S3 buckets
• Log in and search for Billing and Cost Management
• Open the Billing and Cost Management Dashboard to see every charge on the account
• We terminated everything together last class, so the total should read zero

• A charge you did not expect can be refunded, so ask
• Go to the AWS Support Center
• Click on Create case
• Select Account and billing support
• Fill in the form and explain what happened
• Submit the case and wait for a reply (it is usually quick)
• Email me if you get stuck at any of these steps

• Pull data from a web API with requests, then process it with DuckDB or Polars
• Your pull script saves a snapshot in data/raw/; the report reads only that snapshot
• Report in Quarto: 1,500 to 2,500 words, at least two visualisations, visible code
• Ship the whole thing in a Docker container that anyone can build and run, code on GitHub
• Track A (the default): the World Bank Indicators API, no key, no registration
• Track B: any public API you clear with me first, free, no OAuth, documented well enough for a classmate to use
• Groups of three to four; email me the names by Thursday 5 November or I assign you at random

• The project is worth 20% of your course grade

• Every member contributes to the shared repository, and I read the commit history as the evidence
• A member with no meaningful contribution receives a zero for the project
• Where the history shows clearly unequal shares, I adjust individual grades
• Your report ends with a short contribution statement, which I check against the repository

• Think back to Lecture 14
• You created an API key and saved it in a .env file
• A few lines of Python sent your prompt to a language model somewhere on the internet
• A reply came back, and you printed it
• That was an API call, so you have already done this
• Today we take a closer look at what actually happened
• Once you can see the machinery, you can point it at weather, economic indicators, census records, Emory’s course catalogue, or anything else that is public!

1. What an API is

• The contract between two programs
• The client-server model you met on EC2
• Why data scientists care

2. HTTP

• URLs and query strings
• GET and POST
• Status codes (headers wait for Lecture 19, and Appendix 05 has a preview)

3. JSON

• The format almost every API answers in
• How it maps onto Python dictionaries and lists

4. The requests library

• Three lines of code do (most of) what we need
• Some exercises along the way

• API stands for Application Programming Interface
• It is just a contract between two programs
• The contract says: if you ask me in exactly this way, I will answer in exactly that way
• For example, the World Bank promises to turn a URL shaped like this into economic indicators shaped like that
• Neither side needs to know how the other works internally
• Their database, their programming language, and the location of the machine are all none of your business

• The tired metaphor is a restaurant, and it is tired because it works 😅
• The menu is the documentation, telling you what you are allowed to ask for
• Your order is the request, written in terms the kitchen recognises
• The dish is the response, arriving in a predictable form
• You never enter the kitchen
• Order something off the menu and you get an error, but never a surprise dish

• You have played both roles this semester
• The client asks: your laptop, your script, your browser
• The server answers: a machine in a data centre, waiting for requests
• Last week you launched a server on EC2 that sat there waiting for connections
• The World Bank API is the same idea, run by people with a bigger budget and a better uptime record (hopefully!) 😂
• Again, there’s nothing magical about it. It is someone else’s computer, and you already know what to do 🤓

The data is already out there

• The World Bank, the IMF, the US Census Bureau, the ECB, Eurostat, NOAA, and most national statistics offices publish through APIs
• So do GitHub, Wikipedia, and most platforms you use
• Emory’s course catalogue (Atlas) is served this way too
• The World Bank API alone exposes 29,544 indicators for 218 countries and territories, going back decades
• Most of this data is free, and a script can assemble exactly the panel you need

Scripts are always better than clicking

• A manual download is a story you tell about your data. A script is your data’s history
• Re-run it next year and you get the updated numbers
• Send it to a colleague and they get your exact dataset
• Push it to GitHub and a reviewer can verify every step
• This is ingredient two from Lecture 10, “the raw inputs, or a script that fetches them”, made concrete

• The query string starts at the ? and never appears before it
• Each option is key=value, and options are joined by &

?format=json&date=2014:2025&per_page=100
 ^^^^^^^^^^ ^^^^^^^^^^^^^^ ^^^^^^^^^^^^
 option 1   option 2       option 3

• Order does not matter. ?a=1&b=2 and ?b=2&a=1 are the same request
• The path says what you want; the query string says how you want it
• A value can never hold a raw ?, &, =, / or space, because those characters already mean something. They are percent-encoded instead
• Hand-writing that is miserable, so requests does it for you

GET: “please send me this”

• Asking for data, changing nothing
• The whole request fits in the URL
• Your browser sends one every time you visit a page
• Safe to repeat: fetching twice gives you the same thing
• This is roughly 95% of what you will do in this course

POST: “here is some data”

• Sending something to the server
• The data travels in the request body, away from the URL
• Used for logins, uploads, and long inputs
• You already sent one in Lecture 14
• Your prompt went to the model as a POST body, because a paragraph of text does not belong in a URL

• Before writing any code, paste the URL into your browser’s address bar (click here to try it)
• Your browser sends the GET for you and shows you the raw response
• JSON on screen means your URL is right and the problem is in your Python
• An error on screen means your URL is wrong, and no amount of Python will fix it
• This one habit will save you hours of debugging this semester
• The terminal does the same job, and you already know curl from Module 02:

curl -s \
  "https://api.open-meteo.com/v1/forecast\
?latitude=33.75&longitude=-84.39\
&current=temperature_2m" \
  | python3 -m json.tool

• -s hides the progress meter, and python3 -m json.tool pretty-prints the result

• JSON means JavaScript Object Notation, and the name records its origin, nothing about where it is used today
• It is plain text, so any editor opens it
• It is built from key-value pairs and lists, nested as deeply as the data needs
• Almost every web API answers in it, because every language can read it
• JSON is limited to pure data only: no programming logic, functions or classes

{
  "city": "Atlanta",
  "temperature": 29.6,
  "raining": false,
  "sensors": ["a1", "b2"],
  "last_checked": null
}

The json module ships with Python, so there is nothing to install

import json

text = '''{"city": "Atlanta",
 "temperature": 29.6, "raining": false}'''

# json.loads() reads a string
# The 's' in 'loads' is for 'string'
data = json.loads(text)

print(type(data))
print(data["city"], data["temperature"])
print(data["raining"], type(data["raining"]))

<class 'dict'>
Atlanta 29.6
False <class 'bool'>

The saved reply, printed with indentation so the nesting is visible

import json

with open("data/openmeteo_atlanta.json") as f:
    weather = json.load(f)

print(json.dumps(weather, indent=2)[:560])

{
  "latitude": 33.759865,
  "longitude": -84.39586,
  "generationtime_ms": 0.06091594696044922,
  "utc_offset_seconds": -14400,
  "timezone": "America/New_York",
  "timezone_abbreviation": "GMT-4",
  "elevation": 316.0,
  "current_units": {
    "time": "iso8601",
    "interval": "seconds",
    "temperature_2m": "\u00b0C",
    "relative_humidity_2m": "%",
    "wind_speed_10m": "km/h"
  },
  "current": {
    "time": "2026-08-21T16:00",
    "interval": 900,
    "temperature_2m": 33.2,
    "relative_humidity_2m": 42,
    "wind_speed_10m": 4.4
  }
}

Nested data means chained lookups, and every step on the way is printable

# One step at a time
current = weather["current"]
print(current)

# The whole path at once
print(weather["current"]["temperature_2m"],
      weather["current_units"]["temperature_2m"])

{'time': '2026-08-21T16:00', 'interval': 900,
 'temperature_2m': 33.2, 'relative_humidity_2m': 42,
 'wind_speed_10m': 4.4}
33.2 °C

Not every API answers with a tidy dict at the top

with open("data/wb_gdp_bra.json") as f:
    wb = json.load(f)

print("Top level is a", type(wb).__name__,
      "with", len(wb), "elements")
print("Element 0 (metadata):", wb[0])
print("Element 1 is a", type(wb[1]).__name__,
      "with", len(wb[1]), "records")
print("First record:", json.dumps(wb[1][0], indent=2))

Top level is a list with 2 elements
Element 0 (metadata): {'page': 1, 'pages': 1,
'per_page': 100, 'total': 12, 'sourceid': '2',
'lastupdated': '2026-07-13'}
Element 1 is a list with 12 records
First record: {
  "indicator": {
    "id": "NY.GDP.PCAP.KD",
    "value": "GDP per capita (constant 2015 US$)"
  },
  "country": {
    "id": "BR",
    "value": "Brazil"
  },
  "countryiso3code": "BRA",
  "date": "2025",
  "value": 9747.99557762692,
  [...]
}

Assume a plain list of records, write wb[0]["value"], and you get a confusing error or a wrong number

print(wb[1][0]["country"]["value"],
      wb[1][0]["date"],
      round(wb[1][0]["value"], 2))

Brazil 2025 9748.0

requests is not in the standard library, so install it once with pip install requests

More information here: https://github.com/psf/requests

import requests

url = ("https://api.open-meteo.com/v1/forecast"
       "?latitude=33.75&longitude=-84.39"
       "&current=temperature_2m")

r = requests.get(url)

print(r.status_code)
data = r.json()           # JSON into Python objects
print(data["current"]["temperature_2m"])

200
33.2

Hand requests a params dictionary and it writes the query string for you

import requests

params = {
    "latitude": 33.75,
    "longitude": -84.39,
    "current": "temperature_2m,wind_speed_10m",
    "timezone": "America/New_York",
}

r = requests.get("https://api.open-meteo.com/v1/forecast",
                 params=params)
print(r.url)

https://api.open-meteo.com/v1/forecast?latitude=
33.75&longitude=-84.39&current=temperature_2m%2C
wind_speed_10m&timezone=America%2FNew_York

r.json() on an error page raises a confusing exception, because an error page is not JSON

r = requests.get(url, params=params, timeout=10)
r.raise_for_status()     # raises on 4xx and 5xx
data = r.json()

A friendlier version for scripts other people will run:

try:
    r = requests.get(url, params=params, timeout=10)
    r.raise_for_status()
    data = r.json()
except requests.exceptions.HTTPError as e:
    print(f"The server said no: {e}")
except requests.exceptions.RequestException as e:
    print(f"Could not reach the server: {e}")

• Always pass timeout. Without it a silent server hangs your script indefinitely
• HTTPError means a status you did not want; RequestException means you never got one

The World Bank retired the old CO2 indicator, and this is how it says so:

url = ("https://api.worldbank.org/v2/country/BRA"
       "/indicator/EN.ATM.CO2E.PC")

r = requests.get(url, params={"format": "json"},
                 timeout=10)
print(r.status_code)
print(r.text[:110])

200
[{"message":[{"id":"175","key":"Invalid format",
"value":"The indicator was not found. It may have
been deleted

Twelve years of Brazilian GDP per capita, in one request

import requests, json

country   = "BRA"
# GDP per capita, constant 2015 US$
indicator = "NY.GDP.PCAP.KD"
url = ("https://api.worldbank.org/v2"
       f"/country/{country}/indicator/{indicator}")

params = {"format": "json", "date": "2014:2025",
          "per_page": 100}

r = requests.get(url, params=params, timeout=10)
r.raise_for_status()
wb = r.json()

with open("data/wb_gdp_bra.json", "w") as f:
    json.dump(wb, f)

print(r.status_code, wb[0]["total"], "records saved")

200 12 records saved

Unpack the two elements, then sort the records oldest first

with open("data/wb_gdp_bra.json") as f:
    wb = json.load(f)

meta, records = wb[0], wb[1]
print(f"{meta['total']} records, "
      f"page {meta['page']} of {meta['pages']}\n")

# Build a list of (year, value) pairs, oldest first
series = [(int(r["date"]), r["value"]) for r in records]
series.sort()

for year, value in series:
    print(f"  {year}  {value:>10,.0f} US$")

12 records, page 1 of 1

  2014       9,338 US$
  2015       8,936 US$
  2016       8,578 US$
  2017       8,628 US$
  2018       8,722 US$
  2019       8,771 US$
  2020       8,435 US$
  2021       8,799 US$
  2022       9,032 US$
  2023       9,288 US$
  2024       9,567 US$
  2025       9,748 US$

You already know pandas, so the analysis starts here

import pandas as pd

df = pd.DataFrame(series, columns=["year", "gdp_per_capita"])
df["change_pct"] = df["gdp_per_capita"].pct_change() * 100

print(df.round(2).to_string(index=False))

 year  gdp_per_capita  change_pct
 2014         9338.34         NaN
 2015         8936.20       -4.31
 2016         8577.84       -4.01
 2017         8628.25        0.59
 2018         8722.34        1.09
 2019         8771.44        0.56
 2020         8435.01       -3.84
 2021         8799.23        4.32
 2022         9032.08        2.65
 2023         9288.03        2.83
 2024         9566.75        3.00
 2025         9748.00        1.89

Concepts

• An API is a contract: ask this way, get that back. The server is someone else’s computer
• An endpoint is one URL pattern that contract offers
• HTTP is the transport: a path, a query string, a GET for reading, and a status code in reply
• JSON is the format: key-value pairs and lists, mapping onto Python dicts and lists
• requests is the tool, and three lines cover the whole journey

r = requests.get(url, params=p, timeout=10)
r.raise_for_status()
data = r.json()

Habits worth keeping

• Paste the URL into your browser before you debug your Python
• Print the shape of the response before you parse it
• Let params build your query string, and read r.url to check it
• Always set a timeout
• Call raise_for_status(), then look at the body anyway
• Fetch once, save to disk, then work from the file
• Read the documentation’s example request first, then adapt it

Open-Meteo needs latitude and longitude, and current asks for present conditions

https://api.open-meteo.com/v1/forecast?latitude=33.75
&longitude=-84.39&current=temperature_2m

Adding a timezone makes the timestamp readable:

https://api.open-meteo.com/v1/forecast?latitude=33.75
&longitude=-84.39&current=temperature_2m
&timezone=America%2FNew_York

The same request in Python:

import requests

r = requests.get(
    "https://api.open-meteo.com/v1/forecast",
    params={"latitude": 33.75, "longitude": -84.39,
            "current": "temperature_2m",
            "timezone": "America/New_York"},
    timeout=10,
)
print(r.url)
print(r.json()["current"])

https://api.open-meteo.com/v1/forecast?latitude=
33.75&longitude=-84.39&current=temperature_2m
&timezone=America%2FNew_York
{'time': '2026-08-21T16:00', 'interval': 900,
 'temperature_2m': 33.2}

• Both coordinates are required, and the request fails without either one
• requests encoded the / in the timezone as %2F, which is params earning its keep
• Without timezone the time comes back in GMT, which reads oddly for Atlanta
• The reading is live, so your number will differ from this one

• data["courses"] is a list, so [1] picks the second course
• The office needs three steps down: list, then course, then instructor
• The sum runs over the list, and no course is named in the code
• null became None, never the string "null", which is the JSON-to-Python mapping doing its job

Only the two path pieces change

import requests, json

country   = "URY"
indicator = "SP.POP.TOTL"
url = ("https://api.worldbank.org/v2"
       f"/country/{country}/indicator/{indicator}")

r = requests.get(url,
                 params={"format": "json",
                         "date": "2014:2025",
                         "per_page": 100},
                 timeout=10)
r.raise_for_status()

with open("data/wb_pop_ury.json", "w") as f:
    json.dump(r.json(), f)

print(r.status_code, r.json()[0]["total"], "records")

200 12 records

• country and indicator are the only two edits, and the rest of the script is the Brazilian one
• The query string is identical, because the same options apply to every indicator
• Saving first means the next slide runs with no network
• A wrong indicator code here would return 200 with an error message, so read the count

• wb[1] is the data list, exactly as in the Brazilian response
• The loop compares r["date"] with the string "2020", because the API sends years as text
• Sorting on the same string works here, since all twelve years have four digits
• Uruguay’s population peaks in 2020 and falls every year after it
• {:,} adds the thousands separators, which makes seven digits readable

• Alongside the URL, every request carries headers: metadata about the request itself

• Responses carry headers too: content type, cache rules, and your remaining rate-limit quota