DATASCI 350 - Data Science Computing

Lecture 18 - Web APIs and JSON

Danilo Freire
danilo.freire@emory.edu

Department of Data and Decision Sciences
Emory University

Hey, there! 😊
I hope you’re all doing well!

Brief recap 📚

Lecture 17: a machine in the cloud

What we did on AWS

  • Launched an EC2 t3.micro running Ubuntu 26.04
  • Connected with a key pair and ssh -i
  • Installed software with apt
  • Moved files with scp and wget
  • Checked the instance details in the AWS Console
  • Met AWS CloudShell, the browser terminal
  • Queried AWS with the aws command line
  • Stopping pauses compute, but the disk still charges. Terminating deletes the machine

Source: K21

Check your AWS account!

Confirm that the bill reads zero

Billing and Cost Management Home

The final project

A quick reminder

  • Groups of four to five. Email me the names by Thursday 5 November, or I assign you at random 😉
  • Due Tuesday 8 December. Send the repository link on Canvas
  • Today’s lecture covers the first step: pulling data from a web API
  • Instructions and starter repository

Today’s lecture 📚

You have already used an API

  • In Lecture 14, we saved an API key in a .env file
  • We used it to send prompts to a model on OpenRouter and get a reply
  • That exchange was an API call
  • The openai library sent the request for us. Today we write the requests ourselves
  • Other APIs give you weather, economic data, census records and more
# Lecture 14
client = OpenAI(
    base_url="https://openrouter.ai/api/v1",
    api_key=os.environ["OPENROUTER_API_KEY"],
)

response = client.chat.completions.create(
    model="google/gemma-4-31b-it:free",
    messages=[{"role": "user", "content": "Hi!"}],
)
print(response.choices[0].message.content)

base_url is a web address, and the library sends our request there. Today you will see the request and the reply

Lecture overview

What we will cover today

1. What an API is

  • A contract between two programs
  • The client and the server
  • Why data scientists care

2. URLs and status codes

  • The parts of a URL, and building one step by step
  • The codes a server sends back

3. JSON

  • The format almost every API uses
  • How to find one value inside a nested reply

4. APIs from the terminal

  • curl, a browser for your terminal
  • Saving a reply to a file

5. A first taste of Python

  • A few lines that do what curl did. Lecture 19 takes them apart

What is an API?

An API is a contract

Client and server

  • You have played both roles this semester
  • The client asks: your laptop, script, or browser
  • The server answers: a machine in a data centre, waiting for requests
  • Last class you launched a server on EC2
  • The World Bank API is the same idea, with a bigger budget and (hopefully!) better uptime 😂
  • There is nothing magical about it. It is someone else’s computer 🤓

What makes an API a web API

  • “API” is a broad word. pandas has an API, and so does your operating system
  • We focus on web APIs:
    • The request is an HTTP request to a URL, like the ones your browser sends
    • The response is usually JSON, a text format we cover shortly
  • So you can use them from any language that fetches URLs: Python, R, JavaScript, or curl
  • Terms you will see in documentation:

Why this matters for data science

The data are already out there

A script instead of a download button

api.worldbank.org/v2/country/BRA;ARG/
  indicator/NY.GDP.PCAP.CD
  ?date=2000:2023
  • A CSV does not record what you selected
  • Data get revised. Rerun the script for the latest figures
  • Save a dated copy in data/raw/
  • A short loop can fetch data for all 200+ countries

Three kinds of API

Some are open, some need a key, and some charge you

Kind What it costs you Examples
Open and keyless Nothing. Just fetch the URL Open-Meteo, World Bank, National Weather Service, USGS earthquakes, GitHub
Free but key-required A signup form, then a key in every request NASA, OpenWeatherMap, OpenRouter’s free models, GitHub for higher limits
Paid A key and a bill OpenRouter’s paid models, most commercial data vendors
  • An API key is a password for programs. It tells the server who is asking
  • GitHub answers keyless requests, up to 60 per hour per IP address
  • Today we stay in the first row, so nobody needs an account

api.github.com/users/danilofreire, fetched with no key at all (click image to enlarge)

URLs and status codes 🔗

Anatomy of a URL

Four parts you will see in every request

  • Every API call starts with a URL, and every URL has the same parts
  • Here is a real one: Brazil’s GDP per person, 2014 to 2025

https://api.worldbank.org/v2/country/BRA/indicator/NY.GDP.PCAP.KD?format=json&date=2014:2025

Part Name What it does
https scheme Which protocol. Almost always https
api.worldbank.org host Which machine to ask
/v2/country/BRA/indicator/… path Which data on that machine
? separator Ends the path. Everything after it is the query string
format=json&date=2014:2025 query string Options, as key=value pairs joined by &

Building a URL

Step Add this What it means
1 https://api.worldbank.org/v2 The API’s base URL, version 2
2 /country/BRA Brazil, by its three-letter ISO code
3 /indicator/NY.GDP.PCAP.KD GDP per person, in constant US dollars
4 ?format=json Start the options. Send JSON, not XML
5 &date=2014:2025 Only these years
  • The path (steps 1 to 3) says what you want
  • The query string (steps 4 and 5) says how you want it
  • Swap BRA for URY and you get Uruguay
  • Swap the indicator for SP.POP.TOTL and you get population
  • Indicator codes are listed at data.worldbank.org/indicator
  • In NY.GDP.PCAP.KD:
    • NY = national accounts (income)
    • GDP = gross domestic product
    • PCAP = per capita
    • KD = constant US dollars

Reading the query string

Everything after the question mark

  • The query string starts at the ?
  • Each option is key=value, and & joins them
?format=json&date=2014:2025&per_page=100
 ^^^^^^^^^^^ ^^^^^^^^^^^^^^ ^^^^^^^^^^^^
 option 1    option 2       option 3
  • Order does not matter: ?a=1&b=2 and ?b=2&a=1 are the same request
  • Leave out an option and the API uses its default
  • Misspell a key and the API usually ignores it without telling you
  • Some characters already have a job in a URL: ?, &, =, #, and the space
  • To put one inside a value, you percent-encode it
  • A space becomes %20, and a / becomes %2F
  • So America/New_York travels as America%2FNew_York
  • Python’s requests library does this for you, so you rarely type it
  • The full table is in Appendix 04

Status codes

  • Every response carries a three-digit number saying how it went
  • The first digit sums it up: 2xx worked, 4xx is your mistake, 5xx is theirs
Code Name What it means
200 OK It worked. Go ahead and read the data
301 / 302 Moved The data lives elsewhere now. Browsers and requests follow these for you
400 Bad Request Your URL is malformed. Check the query string
401 Unauthorized You need a key, or yours is wrong
403 Forbidden Your key works, but it does not give you access to this
404 Not Found Nothing at that path. Usually a typo
429 Too Many Requests You are asking too fast. Slow down
500 Internal Server Error Their problem. Wait and retry

Some APIs, like the World Bank, answer errors with 200 and hide the failure in the reply. A status check alone is not enough. We will see this in the terminal soon

GET and POST

Two main verbs

GET: “please send me this”

  • Asks for data and changes nothing
  • The whole request fits in the URL
  • Your browser sends one for every page you visit
  • Safe to repeat: fetching twice changes nothing on the server
  • About 95% of what you will do in this course

POST: “here is some data”

  • Sends data to the server
  • The data travels in the request body, not the URL
  • Used for logins, uploads, and long inputs
  • In Lecture 14, the openai library sent your prompt as a POST
  • A paragraph of text does not belong in a URL

Other methods exist (PUT, DELETE, PATCH), but you rarely need them to read public data

Try it in the browser first

The address bar is an HTTP client you already know

  • Before writing any code, paste the URL into your browser (try this one)
  • The browser sends a GET and shows the raw reply
  • JSON on screen: the URL is right. If your code fails later, the problem is in the code
  • An error on screen: the URL is wrong
  • It’s a good habit to check the URL in the browser before writing code! 😉

Open-Meteo answering a plain GET from the address bar, no code involved

Try it yourself! 🤓

Build a URL by hand in your browser

  1. Open the Open-Meteo docs and start from https://api.open-meteo.com/v1/forecast
  2. Ask for the current temperature (current=temperature_2m) in Atlanta (latitude 33.75, longitude -84.39). You need three parameters
  3. Write the query string yourself. Do not copy it from the docs
  4. Paste the URL into the address bar and press Enter
  5. Delete longitude. What happens?
  6. To covert the temperature to Fahrenheit, add &temperature_unit=fahrenheit to the URL

What to look for

  • A temperature and its units in the JSON
  • A different number for the other city
  • An error message without longitude

Stuck, or want to compare your URL with mine?

Appendix 01

JSON 📦

What JSON is

  • JSON means JavaScript Object Notation. The name records its origin, not where it is used today
  • It is plain text, so any editor opens it
  • Almost every web API uses it, because every language can read it
  • It holds data only: no code, functions, or classes
  • Two building blocks, nested as deep as needed:
    • An object { } holds "key": value pairs
    • An array [ ] holds values in order
{
  "city": "Atlanta",
  "temperature": 29.6,
  "raining": false,
  "sensors": ["a1", "b2"],
  "last_checked": null
}

Each JSON type has a Python equivalent

JSON Python
{...} object dict (dictionary)
[...] array list
"text" str (string)
42, 3.14 int, float
true / false True / False
null None
  • Keys are always text in double quotes
  • No comma after the last item
  • In the example, "sensors" is an array inside an object

A real reply

{
  "latitude": 33.759865,
  "longitude": -84.39586,
  "timezone": "America/New_York",
  "elevation": 316.0,
  "current_units": {
    "time": "iso8601",
    "temperature_2m": "°C",
    "wind_speed_10m": "km/h"
  },
  "current": {
    "time": "2026-08-21T16:00",
    "temperature_2m": 33.2,
    "wind_speed_10m": 4.4
  }
}

Shortened from the saved reply, openmeteo_atlanta.json

  • The outer { } is an object. It has nine keys, and six are shown here
  • Two of them, current and current_units, hold objects of their own
  • That is what “nested” means: a box inside a box
  • The temperature is inside current. Its unit is inside current_units
  • time is text, because JSON has no date type

Finding a value: follow the path

  • A path lists the steps to a value, from the outside in
  • In an object, each step is a key name
  • In an array, each step is a position, starting at 0, as in Python
  • In Python, each step becomes a pair of square brackets
  • Get the path right first

Two replies: the Open-Meteo one, and {"sensors": ["a1", "b2"]}

Path Value
current → temperature_2m 33.2
current_units → temperature_2m °C
sensors → 0 “a1”
sensors → 1 “b2”

The same paths in Python:

data["current"]["temperature_2m"]   # 33.2
data["sensors"][0]                  # "a1"

The World Bank reply

An array with metadata, then data

  • The reply is an array with two items
  • Position 0: metadata about the request
  • Position 1: the data, one object per year, newest first
[
  {"page": 1, "pages": 1, "total": 12, ...},
  [
    {"country": {"id": "BR", "value": "Brazil"},
     "date": "2025", "value": 9747.99557762692},
    ... 11 more years
  ]
]

Shortened from wb_gdp_bra.json, the reply to our Brazil URL

import json
data = json.load(open("wb_gdp_bra.json"))

data[0]["total"]                 # 12
data[1][0]["date"]               # "2025"
data[1][0]["value"]              # 9747.99...
data[1][0]["country"]["value"]   # "Brazil"
  • Every path starts with a position, because the reply is an array
  • A number picks an item from a [ ], a name picks one from a { }
  • Look at the shape before you write any code

More about the World Bank API reply, and its official documentation: Appendix 07

Try it yourself! 🤓

Write the paths in Python

import json

text = """
{"term": "Fall 2026",
 "courses": [
   {"code": "DATASCI 350", "enrolled": 40,
    "instructor": {"name": "Danilo Freire"}},
   {"code": "DATASCI 101", "enrolled": 65,
    "instructor": {"name": "Danilo Freire"}}],
 "updated": null}
"""
reply = json.loads(text)

print(reply["term"])   # Fall 2026
  1. Copy the code into a notebook or a .py file and run it
  2. Print the code of the second course
  3. Print the name of the first course’s instructor
  4. Print reply["updated"]. What do you see, and why?

Stuck, or want to compare your paths with mine?

Appendix 02

APIs from the terminal 💻

curl: a browser for your terminal

  • curl sends a GET to a URL and prints the reply, as one long line
  • It comes with macOS, Linux and WSL. Missing in WSL? Run sudo apt install curl
  • Put the URL in double quotes. Otherwise the shell reads & and ? as its own symbols
curl "https://api.open-meteo.com/v1/forecast?latitude=33.75&longitude=-84.39&current=temperature_2m"
  • That is hard to read and easy to mistype. Split it into pieces:
    • BASE= stores the base URL, and Q= the query string
    • $BASE and $Q paste them back
BASE="https://api.open-meteo.com/v1/forecast"
Q="latitude=33.75&longitude=-84.39&current=temperature_2m"
curl "$BASE?$Q"

Readable replies and status codes

  • Pipe (|) the reply into Python’s JSON tool to indent it
  • -s means “silent”: no progress bar
curl -s "$BASE?$Q" | python3 -m json.tool
{
    "latitude": 33.759865,
    ...
    "current": {
        "time": "2026-10-26T16:15",
        "interval": 900,
        "temperature_2m": 18.9
    }
}
  • -i adds the headers (Appendix 05). The first line is the status
  • | head -n 1 keeps only that line
curl -si "$BASE?$Q" | head -n 1
HTTP/1.1 200 OK

Without longitude:

curl -si "$BASE?latitude=33.75" | head -n 1
HTTP/1.1 400 Bad Request

The World Bank answers 200 even for a country that does not exist (XYZ), and puts the error inside the JSON. Check the status and the data

Saving a reply to a file

  • -o means “output”: save the reply to a file instead of printing it
BASE="https://api.worldbank.org/v2/country/BRA/indicator/NY.GDP.PCAP.KD"
Q="format=json&date=2014:2025&per_page=100"
curl -s -o wb_gdp_bra.json "$BASE?$Q"
ls -lh wb_gdp_bra.json                              # 2.7K
python3 -m json.tool wb_gdp_bra.json | head -n 12   # first 12 lines
  • A saved file does not change. The live API does
  • Your analysis then runs offline and gives the same numbers every time
  • The files in this lecture’s data/ folder were made this way
  • Your project does the same: save a snapshot in data/raw/, and read only that file
  • In Lecture 19 we do it in Python

Try it yourself! 🤓

Uruguay’s population, from the terminal

  1. Type these four lines:
BASE="https://api.worldbank.org/v2/country/URY/indicator/SP.POP.TOTL"
Q="format=json&date=2014:2025"
curl -s -o wb_pop_ury.json "$BASE?$Q"
python3 -m json.tool wb_pop_ury.json
  1. Scroll the output and find the population in 2020
  2. Bonus: now print the population in 2020 from Python

What to look for

  • Twelve records, newest first
  • About 3.4 million people

Hint for step 3

  • Start Python in the folder where you ran curl, then load the file:
import json
data = json.load(open("wb_pop_ury.json"))
  • The records are in position 1, newest first

Stuck, or want to compare with mine? Appendix 03

A first taste of Python 🐍

A few lines of Python

The same request, from a script

  • requests is the Python library for web APIs. Install it once: pip install requests
import requests

base = "https://api.open-meteo.com/v1/forecast"
q = "latitude=33.75&longitude=-84.39&current=temperature_2m"

r = requests.get(base + "?" + q)          # send a GET, like curl
print(r.status_code)                      # 200 means OK

data = r.json()                           # JSON text -> Python dictionary
print(data["current"]["temperature_2m"])  # the path, one [] per step
200
19.2

Conclusion 📚

What we learned today

Concepts

  • An API is a contract: ask this way, get that back
  • A URL has a path (what) and a query string (how)
  • A status code says how it went: 2xx, 4xx, 5xx
  • JSON values are reached by a path of keys and positions

Good habits

  • Test the URL in your browser first
  • Quote URLs in the terminal
  • Check the status and the data
  • Save a copy, then work from the file

Next class

  • Next class we do all of this in Python
  • Reading JSON from a reply and from a file
  • Building requests with params, and checking them for errors
  • From the World Bank reply to a pandas DataFrame
  • Your project’s pull script, line by line

Before then

  1. Install the packages you will need next class (see below)
  2. Check that your AWS bill reads zero
  3. Read the project instructions and form a group of four to five
  4. Finish Exercise 03 if you did not have time in class
pip install requests pandas pyarrow

A list of keyless APIs to explore is in Appendix 06

Group names are due by Thursday 5 November

And that’s all for today! 😊

Appendix 01

Exercise 01 solution

Open-Meteo needs latitude and longitude, and current asks for present conditions

https://api.open-meteo.com/v1/forecast?latitude=33.75
&longitude=-84.39&current=temperature_2m

Adding a timezone makes the time readable:

https://api.open-meteo.com/v1/forecast?latitude=33.75
&longitude=-84.39&current=temperature_2m
&timezone=America%2FNew_York

Without longitude, the page shows:

{"error":true,"reason":"Parameter 'latitude' and
'longitude' must have the same number of elements"}
  • Both coordinates are required. The request fails without either one
  • The / in the timezone is written %2F, because / already has a job in a URL
  • Without timezone, the time comes back in GMT, which reads oddly for Atlanta
  • The reading is live, so your number will differ from mine

Appendix 02

Exercise 02 solution

  1. courses → 1 → code → “DATASCI 101”
  2. courses → 0 → instructor → name → “Danilo Freire”
  3. updated → null, which becomes None in Python

The same paths in Python:

print(reply["courses"][1]["code"])                  # DATASCI 101
print(reply["courses"][0]["instructor"]["name"])    # Danilo Freire
print(reply["updated"])                             # None
  • courses is an array, so the second course is position 1, not 2
  • The name is four steps down: courses, then the course, then instructor, then name
  • null is JSON’s word for “no value”. It is not the text "null"

Appendix 03

Exercise 03 solution

Part of the output, around 2020:

        {
            ...
            "countryiso3code": "URY",
            "date": "2020",
            "value": 3398968,
            ...
        },
  • The 2020 population is 3,398,968
  • Its path: 1 → 5 → value. Records run newest first, so 2025 is position 0 and 2020 is position 5

Step 3, in Python:

import json
data = json.load(open("wb_pop_ury.json"))
print(data[1][5]["value"])   # 3398968
  • [1] is the list of records, [5] the 2020 record, and ["value"] its population
  • Only the country and the indicator differ from the Brazil example
  • The saved file is also in data/wb_pop_ury.json

Got a FileNotFoundError?

  • open looks in the folder where you started Python, and it does not understand ~
  • Use pwd and ls to find the file, then cd there before python3
  • Or give the full path: open("/Users/yourname/wb_pop_ury.json")

Appendix 04: percent-encoding

Characters that need a code inside a value

You write It travels as
space %20 or +
/ %2F
? %3F
& %26
= %3D
: %3A
# %23
% %25
  • The pattern is always the same: % and the character’s byte in hexadecimal (remember them from Lecture 02? 😉)
  • America/New_York becomes America%2FNew_York
  • requests encodes values for you, so you rarely write these by hand
  • Full tables and rules: MDN, W3Schools, and RFC 3986 §2.1

Appendix 05: headers

  • Every request and every reply carries headers: notes about the message itself
Header Purpose
User-Agent Who is asking. Browsers set this, and polite scripts should too
Accept What format you want back (application/json)
Authorization Your API key, for APIs that need one
Content-Type The format of the data being sent 
curl -sI "https://api.worldbank.org/v2/country/BRA/indicator/NY.GDP.PCAP.KD?format=json&date=2014:2025"
HTTP/2 200
content-type: application/json;charset=utf-8
cache-control: public, max-age=86400
server: cloudflare
...
  • -I asks for the headers only, without the data
  • content-type confirms the reply is JSON
  • max-age=86400 says the reply can be reused for a day (86,400 seconds)

Appendix 06: free APIs worth exploring

All of these answered a keyless request on 21 August 2026

API What it gives you Documentation
Open-Meteo Weather forecasts and history, anywhere https://open-meteo.com/en/docs
World Bank 29,544 development indicators, all countries https://datahelpdesk.worldbank.org/knowledgebase/topics/125589-developer-information
National Weather Service US forecasts and alerts, from api.weather.gov/points/{lat},{lon} https://www.weather.gov/documentation/services-web-api
GitHub Repositories, users, commits (keyless with low limits) https://docs.github.com/en/rest
Open Library Books, authors, covers, by ISBN (occasionally flaky) https://openlibrary.org/developers/api
USGS Earthquakes Every recorded earthquake, live https://earthquake.usgs.gov/fdsnws/event/1/

Pick one this week and fetch something from it, in the browser or with curl

Appendix 07: the World Bank reply in detail

Position 0: the metadata

Key Meaning
page Which page this is
pages How many pages there are
per_page Records per page: 50, unless you set per_page
total Records across all pages
lastupdated When the indicator was last updated 
data[0]["pages"]   # 1
  • If pages is above 1, you only have part of the data
  • Add &page=2 to the URL, or raise per_page
  • country/all for 2023 returns 266 records, so 6 pages of 50
  • The 266 include regions and income groups, not only countries
  • Everything else is in the World Bank’s developer documentation

Position 1: one full record from wb_gdp_bra.json

{"indicator": {"id": "NY.GDP.PCAP.KD",
   "value": "GDP per capita (constant 2015 US$)"},
 "country": {"id": "BR", "value": "Brazil"},
 "countryiso3code": "BRA",
 "date": "2025",
 "value": 9747.99557762692,
 "unit": "",
 "obs_status": "",
 "decimal": 1}
  • country.id is the two-letter code, countryiso3code the three-letter one
  • date is text, and value is null (Python’s None) when a year has no data
for record in data[1]:
    year = int(record["date"])         # date is text
    if record["value"] is not None:    # null becomes None
        print(year, round(record["value"]))