DATASCI 350 - Data Science Computing

Lecture 05 - Git and GitHub

Danilo Freire

danilo.freire@emory.edu

Department of Data and Decision Sciences
Emory University

Nice to see you all again! 😊

Recap and lecture overview 📚

Recap of our last lecture

In our last class, we covered

Essential navigation commands: ls, cd, pwd
Useful directory shortcuts: ~ (home), . (current), .. (parent directory)
File manipulation commands: touch, mkdir, rm, rmdir
File content commands: cat, head, tail
Wildcards for efficient file matching: * (any characters), ? (any single character)
Batch operations with {}
Chaining commands with && and ||
Text manipulation tools:
- wc, grep, sed

Recap of our last lecture

Regular expressions

Regular expressions search for patterns rather than exact text
They work in most languages and tools, and in the shell through grep -E

Symbol	Meaning	Example	Matches
`.`	Any single character	`d.g`	`dog`, `dig`, `d@g`
`^`	Start of a line	`^Hello`	`Hello world`
`$`	End of a line	`end$`	`This is the end`
`[]`	Any character inside	`[aeiou]`	`a`, `e`, `i`, `o`, `u`
`*`	Zero or more of the previous	`ab*c`	`ac`, `abc`, `abbc`
`+`	One or more of the previous	`ab+c`	`abc`, `abbc`
`?`	Zero or one of the previous	`colou?r`	`color`, `colour`
`\\|`	Either one or the other	`cat\\|dog`	`cat`, `dog`
`()`	Group, to apply a symbol to all of it	`(ha)+`	`ha`, `haha`

Today’s lecture

Thinking in projects, and how to set one up
What version control is, and why it will save you
Git and GitHub, and why they are not the same thing
Installing Git and introducing yourself to it
Repositories, the staging area, and commits
git init, add, commit, log, and checkout
Branches, merging, and conflicts come next class

Project management 📂

Taming chaos

In the data science workflow, there are two sorts of surprises and cognitive stress:

Analytical (often good)
Infrastructural (almost always bad)

Analytical surprise: you learn something from the data
Infrastructural surprise: you cannot find what you did last month, the code breaks, the report will not compile, or your collaborator cannot run any of it
Good project management lets you focus on the right kind of stress

Keeping future-you happy

It is tempting to set up a project assuming you will be its only reader
That is almost never true. Coauthors and collaborators happen to the best of us 😅
And even alone, there is always future-you, who has forgotten everything
Future-you either enjoys the fruits of your work, or fights through your chaos
So be kind to future-you. Set up a good workflow now 😊

Project setup 👷🏻‍♀️👷🏽‍♂️

You should always think in terms of projects!

A project is a self-contained unit of data science work that can be

Shared (e.g., with collaborators)
Recreated by others (including future-you)
Packaged (e.g., as a report)
Dumped (exported or archived)

A project contains:

Content, e.g., raw data, processed data, scripts, functions, documents and other output
Metadata, e.g., information about tools for running it (required libraries, compilers), version history

For a Python project:

The project is a folder
The metadata is .git, plus files like pyproject.toml and uv.lock that record which packages it needs

Project setup: directories

Recommendations

One parent folder contains everything inside it
You decide what goes in the project folder. The project dictates the structure
Keep input separate from output. Definitely separate raw from processed data!
Keep every internal path relative. Never write "/Users/me/data/thing.csv", write "../data/thing.csv" instead
An absolute path works on exactly one machine, and often not on that one either after a few months!

A workable starting point:

my-project/
├── data/
│   ├── raw/
│   └── processed/
├── scripts/
│   ├── 00-setup.py
│   ├── 01-import-data.py
│   └── 02-analysis.py
├── output/
│   ├── figures/
│   └── tables/
├── README.md
└── pyproject.toml

Project setup: scripts

Scripts are the glue that holds your project together
They should be readable and reproducible
Names should only include letters and numbers with dashes - or underscores _ to separate words
Use numbering to indicate the order in which files should be run:
- 00-setup.py
- 01-import-data.py
- 02-preprocess-data.py
Write short, modular scripts. Every script serves a purpose in your pipeline
Put the setup first (e.g., your import lines)
Always comment more than you usually do

Version control systems 🔄

What is version control?

Version control tracks every change you make to a set of files
It stores them as snapshots and branches you can move between
So you can see how a project grew, and go back to any point in it
It gives you a backup, a history, and a way for several people to work at once
It scales from a two-file assignment to the Linux kernel

Why version control?

More reasons to use version control

Have you ever…

Changed your code, realised it was a mistake and wanted to revert back?
Lost code or had a backup that was too old?
Wanted to see the difference between different versions of your code?
Wanted to submit a change to someone else’s code?
Wanted to see how much work is being done, when, and by whom?

Version control can help with all of these!

Git and GitHub 🐙

What is Git?

Git is a distributed version control system
Imagine Dropbox and Word’s “Track changes” had a baby, except built for code
There is a learning curve. It is worth it
Other version control systems exist, but Git won
Around 94% of developers used it in 2022. Later surveys stopped asking, because the answer is everyone
Data scientists will assume you know it

Git and GitHub are not the same thing
Git is the software that tracks your changes, and it runs on your machine
GitHub hosts your repositories online and adds tools for working with other people
You do not need GitHub to use Git, just as you do not need VS Code to write Python
But it makes life considerably easier

Git: some background

Where does Git come from?

Git was created in 2005 by Linux creator Linus Torvalds
The initial motivation was to have a non-proprietary version control system to manage Linux kernel development
Check out this (quite opinionated) talk on Git by Linus Torvalds from 2007, two years after its creation
There are many Git GUIs, giving you the option to use git without the shell (often with reduced functionality). Popular choices are the GitHub Desktop, and the integration with VS Code

GitHub: some background

Where does GitHub come from?

Created in 2008, and owned by Microsoft since 2018
It does far more than host code: documentation, websites, issue tracking, and automation
Think of it as a social network for developers
I am a big fan. My website and this course are both hosted on it

Please create an educational account to get free access to GitHub Pro. Use your Emory email address
You will receive a GitHub Student Developer Pack with many freebies, including GitHub Copilot, which is an AI tool that helps you write code
We will meet Copilot and other AI coding tools in future lectures, but you can start using it now

First step: install Git

Git is separate software. You need it installed to use it from the command line or VS Code
It is probably there already. Check where it is and which version you have:

which git
git --version

To install or update it on a Mac, use Homebrew, “the missing package manager for macOS (or Linux)”:

brew install git

On WSL:

sudo apt-get install git

Git for Windows (not WSL): see happygitwithr.com

Second step: introduce yourself to Git

Git labels every commit with your name and email, so others know it came from you. This matters most when you use Git without GitHub
Check your current settings (opens a pager, q quits):

git config --list

Set your name and email:

git config --global user.name 'danilofreire'
git config --global user.email 'danilo.freire@emory.edu'

The name can be your GitHub user name, but does not have to be. The email should match your GitHub account
Name the first branch main. Otherwise Git still uses master, and later commands in this lecture fail:

git config --global init.defaultBranch main

More options: Software Carpentry’s setup guide

Git from the shell 🐚

Repositories

Repositories, usually called repos, store the full history and source control of a project
They can either be hosted locally, or on a shared server, such as GitHub
Most repositories are stored on GitHub, while core contributors make copies of the repository on their machine and update the repository using the push/pull system
Any repository stored somewhere other than locally is called a remote repository

Repos vs directories

A repository is the whole timeline of a project, every change ever made
A working directory is the project as it looks right now
A folder on your machine that talks to a repository is itself a repository. We call it a local repository, to distinguish it from the remote one

Workflow diagram

Work moves through four places:

The working directory: your files, as they are now
The staging area: the changes you have picked out to save next
The local repository: the full history, on your machine
The remote repository: the same history, on a server, usually GitHub

A commit is a snapshot: it takes what is staged and adds it to the timeline

Main Git commands

git init: initialises a new Git repository
git clone: copies an existing repository
git add: adds changes to the staging area
git commit: commits (saves) changes
git push: pushes (uploads) changes
git pull: pulls (downloads) changes
git status: shows the status of the working directory
git diff: shows what changed, line by line
git log: shows the commit history
git branch: shows the branches in the repository
git checkout: switches branches (newer Git also has git switch)

Git documentation

Hands-on: Git and GitHub 🤲🏽

Let’s create a repository together

Let’s create a new repository on GitHub from scratch!
We will create one in our local machine and make some changes
Next class, I will show you how to clone it and modify it
We will also learn how to create branches, merge them, and resolve conflicts

Let’s do it! 🚀

Creating a new repository

I will create a new folder/directory in my computer: my-project
Open the bash/zsh Terminal and go to the my-project directory
I will check where I am with pwd
I will try to add this folder to the staging area

mkdir my-project
cd my-project
git add .

Error! 😟

We need to initialise the repository. Do not do that in your home directory!

git init

Adding and removing files from the repo

Check the staging area status

git status

Let’s add a 01-data-cleaning.py file to the staging area. Then, check its status

touch 01-data-cleaning.py
git add 01-data-cleaning.py
git status

Adding and removing files from the repo

Now let’s create more files and add them to the staging area

touch 02-exploratory-data-analysis.py
touch 03-modelling.py
touch 04-visualisation.py
git add .
git status

Adding and removing files from the repo

In the directory, let’s delete the 04-visualisation.py file and check the status:

rm -f 04-visualisation.py
git status

First commit

Now our “initial commit” :

git commit -m "initial commit"

We deleted 04-visualisation.py but never staged the deletion, so the file is still part of this commit
That is why it comes back when we travel to this commit later

First commit

To check all commits in your repo:

git log

Most important things here are:
- commit id
- date/time
- branch
- commit message

Git checkout

First, let’s make new commits in our repo:

touch 04-new-visualisation.py # new file
touch 05-comments.py # new file
git add . 
git commit -m "adding files" 
echo "Hello you" >> 05-comments.py # edit file
git add . 
git commit -m "editing 05-comments.py" 
rm -f 05-comments.py 
git commit -am "deleting 05-comments.py" 
git log

Hands on! Git checkout

We use checkout to go back in time to a given commit. Let’s go back to the “initial commit”:

git checkout 470636f38e409f4f322c48183e19633ebb550625

Your hash will be different. Copy yours from git log, and the first 5-6 characters are enough
Git warns you about a detached HEAD. That is normal here: you are looking at an old commit, not working on a branch
Check the folder!
04-visualisation.py is back!
Important: doing this does not delete our commits. We just move back in time!

Git checkout

To “go back to the future”, the most recent commit, we just need to go back to the main branch:

git checkout main
git log

Check the folder/directory again!
04-visualisation.py is gone and 04-new-visualisation.py is back!

Next steps 🙂

Phew! That was a lot of work! 😅
Next class we will see how we can create branches and merge them
We will also learn how to resolve conflicts and push our changes to GitHub
For now, let’s keep our my-project folder and we will use it in our next class
If you want to run ahead, create a repository on GitHub and try pushing to it. We will cover the details next class 🤓
Questions?

Summary 📝

Today we learned about:

The importance of project management
The benefits of version control systems
What is Git and why use it
Why GitHub is a great tool for collaborative projects
The main Git commands
The difference between repositories and directories
How to add, remove, commit, and check changes in a repository

Thank you very much and see you soon! 😊🙏🏻