DATASCI 350 - Data Science Computing

Lecture 05 - Git and GitHub

Danilo Freire
danilo.freire@emory.edu

Department of Data and Decision Sciences
Emory University

Nice to see you all again! 😊

Recap and lecture overview 📚

Recap of our last lecture

In our last class, we covered

  • Essential navigation commands: ls, cd, pwd
  • Useful directory shortcuts: ~ (home), . (current), .. (parent directory)
  • File manipulation commands: touch, mkdir, rm, rmdir
  • File content commands: cat, head, tail
  • Wildcards for efficient file matching: * (any characters), ? (any single character)
  • Batch operations with {}
  • Chaining commands with && and ||
  • Text manipulation tools:
    • wc, grep, sed

Recap of our last lecture

Regular expressions

  • Regular expressions search for patterns rather than exact text
  • They work in most languages and tools, and in the shell through grep

Symbol Meaning Example Matches
. Any single character d.g dog, dig, d@g
^ Start of a line ^Hello Hello world
$ End of a line end$ This is the end
[] Any character inside [aeiou] a, e, i, o, u
* Zero or more of the previous ab*c ac, abc, abbc
+ One or more of the previous ab+c abc, abbc
? Zero or one of the previous colou?r color, colour
\| Either one or the other cat\|dog cat, dog
() Group, to apply a symbol to all of it (ha)+ ha, haha

Today’s lecture

  • Thinking in projects, and how to set one up
  • What version control is, and why it will save you
  • Git and GitHub, and why they are not the same thing
  • Installing Git and introducing yourself to it
  • Repositories, the staging area, and commits
  • git init, add, commit, log, and checkout
  • Branches, merging, and conflicts come next class

Project management 📂

Taming chaos

In the data science workflow, there are two sorts of surprises and cognitive stress:

  1. Analytical (often good)
  2. Infrastructural (almost always bad)
  • Analytical surprise: you learn something from the data
  • Infrastructural surprise: you cannot find what you did last month, the code breaks, the report will not compile, or your collaborator cannot run any of it
  • Good project management lets you focus on the right kind of stress

Keeping future-you happy

  • It is tempting to set up a project assuming you will be its only reader
  • That is almost never true. Coauthors and collaborators happen to the best of us 😅
  • And even alone, there is always future-you, who has forgotten everything
  • Future-you either enjoys the fruits of your work, or fights through your chaos
  • So be kind to future-you. Set up a good workflow now 😊

Project setup 👷🏻‍♀️👷🏽‍♂️

You should always think in terms of projects!

A project is a self-contained unit of data science work that can be

  • Shared (e.g., with collaborators)
  • Recreated by others (including future-you)
  • Packaged (e.g., as a report)
  • Dumped (exported or archived)

A project contains:

  • Content, e.g., raw data, processed data, scripts, functions, documents and other output

  • Metadata, e.g., information about tools for running it (required libraries, compilers), version history

For a Python project:

  • The project is a folder
  • The metadata is .git, plus files like pyproject.toml and uv.lock that record which packages it needs

Project setup: directories

Recommendations

  • One parent folder contains everything inside it
  • You decide what goes in the project folder. The project dictates the structure
  • Keep input separate from output. Definitely separate raw from processed data!
  • Keep every internal path relative. Never write "/Users/me/data/thing.csv"
  • That path works on exactly one machine, and often not on that one either, six months later

A workable starting point:

my-project/
├── data/
│   ├── raw/
│   └── processed/
├── scripts/
│   ├── 00-setup.py
│   ├── 01-import-data.py
│   └── 02-analysis.py
├── output/
│   ├── figures/
│   └── tables/
├── README.md
└── pyproject.toml

Project setup: scripts

  • Scripts are the glue that holds your project together
  • They should be readable and reproducible
  • Names should only include letters and numbers with dashes - or underscores _ to separate words
  • Use numbering to indicate the order in which files should be run:
    • 00-setup.py
    • 01-import-data.py
    • 02-preprocess-data.py
  • Write short, modular scripts. Every script serves a purpose in your pipeline
  • Put the setup first (e.g., library() and source())
  • Always comment more than you usually do

Version control systems 🔄

What is version control?

  • Version control tracks every change you make to a set of files
  • It stores them as snapshots and branches you can move between
  • So you can see how a project grew, and go back to any point in it
  • It gives you a backup, a history, and a way for several people to work at once
  • It scales from a two-file assignment to the Linux kernel

Why version control?

Source: PhD Comics

More reasons to use version control

Have you ever…

  • Changed your code, realised it was a mistake and wanted to revert back?
  • Lost code or had a backup that was too old?
  • Wanted to see the difference between different versions of your code?
  • Wanted to review the history of some code?
  • Wanted to submit a change to someone else’s code?
  • Wanted to see how much work is being done, when, and by whom?

Version control can help with all of these!

Git and GitHub 🐙

What is Git?

  • Git is a distributed version control system
  • Imagine Dropbox and Word’s “Track changes” had a baby, except built for code
  • There is a learning curve. It is worth it
  • Other version control systems exist, but Git won
  • Around 94% of developers used it by 2021. Recent surveys stopped asking, because the answer is everyone
  • Data scientists will assume you know it

  • Git and GitHub are not the same thing
  • Git is the software that tracks your changes, and it runs on your machine
  • GitHub hosts your repositories online and adds tools for working with other people
  • You do not need GitHub to use Git, just as you do not need VS Code to write Python
  • But it makes life considerably easier

Git: some background

Where does Git come from?

  • Git was created in 2005 by Linux creator Linus Torvalds
  • The initial motivation was to have a non-proprietary version control system to manage Linux kernel development
  • Check out this (quite opinionated) talk on Git by Linus Torvalds from 2007, two years after its creation
  • There are many Git GUIs, giving you the option to use git without the shell (often with reduced functionality). Popular choices are the GitHub Desktop, and the integration with VS Code

GitHub: some background

Where does GitHub come from?

  • Created in 2008, and owned by Microsoft since 2018
  • It does far more than host code: documentation, websites, issue tracking, and automation
  • Think of it as a social network for developers
  • I am a big fan. My website and this course are both hosted on it

First step: install Git

Again, Git is an independent piece of software. You need to have it installed on your machine to call it from the command line or VS Code

Chances are that that’s already the case. Here’s how you can check using the command line:

which git

And here’s how you can check the version:

git --version

If you want to install (or update) Git on your Mac, I recommend using Homebrew, “the missing package manager for macOS (or Linux)”:

brew install git

WSL users can install Git using:

sudo apt-get install git

To install/update Git for Windows (not WSL), check out happygitwithr.com

Second step: introduce yourself to Git

This is particularly important when you work with Git but without the GitHub overhead. The idea is to define how your commits are labelled. Others should easily identify your commits as coming from you.

Have you already introduced yourself to Git? Find it out:

git config --list

Still have to introduce yourself? To that end, we set our user name and email address like this:

git config --global user.name 'danilofreire'
git config --global user.email 'danilo.freire@emory.edu'

The user name can be (but does not have to be) your GitHub user name. The email address should definitely be the one associated with your GitHub account.

Check out these setup instructions from Software Carpentry to learn about more configuration options.

Git from the shell 🐚

Repositories

  • Repositories, usually called repos, store the full history and source control of a project
  • They can either be hosted locally, or on a shared server, such as GitHub
  • Most repositories are stored on GitHub, while core contributors make copies of the repository on their machine and update the repository using the push/pull system
  • Any repository stored somewhere other than locally is called a remote repository

Repos vs directories

  • A repository is the whole timeline of a project, every change ever made
  • A working directory is the project as it looks right now
  • A folder on your machine that talks to a repository is itself a repository. We call it a local repository, to distinguish it from the remote one

Workflow diagram

Work moves through four places:

  • The working directory: your files, as they are now
  • The staging area: the changes you have picked out to save next
  • The local repository: the full history, on your machine
  • The remote repository: the same history, on a server, usually GitHub

A commit is a snapshot: it takes what is staged and adds it to the timeline

Main Git commands

  • git init: initialises a new Git repository
  • git clone: copies an existing repository
  • git add: adds changes to the staging area
  • git commit: commits changes
  • git push: pushes changes
  • git pull: pulls changes
  • git status: shows the status of the working directory
  • git log: shows the commit history
  • git branch: shows the branches in the repository
  • git checkout: switches branches

Git documentation

Hands-on: Git and GitHub 🤲🏽

Let’s create a repository together

  • Let’s create a new repository on GitHub from scratch!
  • We will create one in our local machine and make some changes
  • Next class, I will show you how to clone it and modify it
  • We will also learn how to create branches, merge them, and resolve conflicts

Let’s do it! 🚀

Creating a new repository

  1. I will create a new folder/directory in my computer: my-project
  2. Open the bash/zsh Terminal and go to the my-project directory
  3. I will copy the my-project directory path into a text document
  4. I will try to add this folder to the staging area
mkdir my-project
cd my-project
git add .
  1. Error! 😟
  1. We need to initialise the repository. Do not do that in your home directory!
git init

Adding/Removing files from the repo

  1. Check the staging area status
git status
  1. Let’s add a 01-data-cleaning.py file to the staging area. Then, check its status
touch 01-data-cleaning.py
git add 01-data-cleaning.py
git status

Adding/Removing files from the repo

  1. Now let’s create more files and add them to the staging area
touch 02-exploratory-data-analysis.py
touch 03-modelling.py
touch 04-visualisation.py
git add .
git status

Adding/Removing files from the repo

  1. In the directory, let’s delete the 04-visualisation.py file and check the status:
rm -f 04-visualisation.py
git status

First commit

  1. Now our “initial commit” :
git commit -m "initial commit"

First commit

  1. To check all commits in your repo:
git log
  • Most important things here are:
    • commit id;
    • date/time;
    • branch;
    • commit message;

Git checkout

  1. First, let’s make new commits in our repo:
touch 04-new-visualisation.py # new file
touch 05-comments.py # new file
git add . 
git commit -m "adding files" 
echo "Hello you" >> 05-comments.py # edit file
git add . 
git commit -m "editing 05-comments.py" 
rm -f 05-comments.py 
git commit -am "deleting 05-comments.py" 
git log

Hands on! Git checkout

  1. We use checkout to go back in time to a given commit. Let’s go back to the “initial commit”:
git checkout 470636f38e409f4f322c48183e19633ebb550625

  • Your hash will be different. Copy yours from git log, and the first 5-6 characters are enough

  • Check the folder!

  • 04-visualisation.py is back!

  • Important: doing this does not delete our commits. We just move back in time!

Git checkout

  1. To “go back to the future”, the most recent commit, we just need to go back to the main branch:
git checkout main
git log
  • Check the folder/directory again!
  • 04-visualisation.py is gone and 04-new-visualisation.py is back!

Next steps 🙂

  • Phew! That was a lot of work! 😅
  • Next class we will see how we can create branches and merge them
  • We will also learn how to resolve conflicts and push our changes to GitHub
  • For now, let’s keep our my-project folder and we will use it in our next class
  • If you want to run ahead, create a repository on GitHub and try pushing to it. We will cover the details next class 🤓
  • Questions?

Summary 📝

Today we learned about:

  • The importance of project management
  • The benefits of version control systems
  • What is Git and why use it
  • Why GitHub is a great tool for collaborative projects
  • The main Git commands
  • The difference between repositories and directories
  • How to add, remove, commit, and check changes in a repository

Thank you very much and see you soon! 😊🙏🏻