• Computers went from people to mechanical calculators to silicon chips
• The Von Neumann architecture keeps instructions and data in one memory
• Everything is stored in binary, base 2, using only 0s and 1s
• One digit is a bit. Eight bits are a byte
• Hexadecimal is base 16, and one hex digit covers exactly four bits
• Abstraction is what lets numbers stand for images and text

• An image is a grid of pixels
• RGB gives each pixel three numbers from 0 to 255: red, green, and blue
• That is 8 bits per channel, so 16.7 million colours
• Text is split into characters, and each character gets a number
• ASCII is a lookup table for 128 characters
• Unicode extends it to accents, other alphabets, and emoji

• Konrad Zuse built the first programmable computers in the 1940s
• Assembly gives machine code readable names, one line per instruction
• Low-level languages are fast, but hard to read and write
• High-level languages like Python hide the hardware and run anywhere
• Compiled languages translate first. Interpreted ones translate as they run

• The command line is a text interface to your computer
• We will navigate the file system, and create, move, and delete files
• We will write shell scripts to automate work you would rather not repeat
• It is still everywhere: remote servers, cloud computing, and automation
• Let’s get started! 🤓

• The operating system sits between your programs and the hardware, and manages access to it
• It has two halves: the kernel and user space
• The kernel talks to the hardware, runs the drivers, and hands out memory and CPU time
• Code running in the kernel can break everything, which is why users are kept out of it
• User space is where your programs run, each in its own sandbox, asking the kernel for whatever it needs
• Curiosity: the Linux kernel source is on GitHub

• A shell is a user space program that lets you reach the system’s resources
• Use it interactively by typing at a terminal, or run a script, a file full of commands
• Most people use a graphical interface instead. Same idea, prettier surface
• Why “kernel” and “shell”? A nut has a soft kernel protected by a hard shell. Useful metaphor, innit?

• You never talk to the shell directly. You talk to a terminal
• A terminal reads your keystrokes, passes them to a program, and shows the results
• A shell cannot do that on its own. It needs a terminal as its interface
• Why “terminal”? Before screens, a terminal was hardware: a keyboard wired to a mainframe, often many terminals to one machine
• Working on a data centre or cloud machine today is much the same, with a nicer screen 😉

• A terminal is simply a text-based interface. The first thing it does is start a shell
• The shell takes input two ways: interactively from the command line, or from a script
• It sends output back to the terminal, or to a file
• The command line is the line you type on. You can customise it with colours and shortcuts

• There are many shells, and they differ in what they can do
• bash is the default on most Unix-like systems. The name stands for Bourne-Again Shell
• zsh is the default on macOS
• Windows has cmd.exe and PowerShell. There is also C Shell, and many more
• Your terminal starts your preferred shell automatically, and you can change which one

• Our tools come from Unix, built at Bell Labs in the 1970s
• They still follow the Unix philosophy:

Do One Thing And Do It Well

• Each tool is small and does a single job
• Chain the small tools together and you can build something large
• Hence the description: minimalist and modular

Of course, it’s always possible to open up the shell directly if you prefer. It’s your turn!

• Linux

• Mac

• Windows: install WSL first, using our tutorial. It gives you the same shell as everyone else

Feel free to check our class tutorial on how to set up your shell in VS Code

Open your terminal and type the following commands (without the $):

$ echo $SHELL 
$ whoami 
$ pwd 
$ mkdir new-folder
$ rm -rf new-folder
$ cd ..
$ ls

Share your results with a colleague (or the class)!

All bash commands have the same basic syntax:

command option(s) argument(s)

Examples:

$ # list files in the Documents directory 
$ # in long format and human-readable sizes
$
$ ls -lh ~/Documents 

$ # sort the file and display 
$ # only unique lines
$
$ sort -u file.txt 

Commands

• You don’t always need options or arguments

• For example:

  • ls ~/Documents/ and ls -lh ~/Documents are both valid commands that will yield (different) output

• However, you always need a command!

All Bash commands have the same basic syntax:

command option(s) argument(s)

Examples:

$ # list files in the Documents directory 
$ # in long format and human-readable sizes
$
$ ls -lh ~/Documents 

$ # sort the file and display 
$ # only unique lines
$
$ sort -u file.txt 

Options (also called flags)

• Start with a dash, usually one letter
• Chain several under a single dash

$ ls -l -a -h /var/log  # this works
$ ls -lah /var/log      # so does this

• Longer, spelled-out names take two dashes

$ ls --human-readable /var/log

• -l long listing, -a all files including hidden ones, -h readable sizes
• -u on sort keeps only unique lines
• Hard to remember what each letter means? You’re totally right!

All Bash commands have the same basic syntax:

command option(s) argument(s)

Examples:

$ # list files in the Documents directory 
$ # with human-readable sizes
$ ls -lh ~/Documents/

$ # sort the file and remove duplicates
$ sort -u file.txt

Arguments

• Tell the command what to operate on

• What counts as a valid argument depends on the command

• Can be a file, path, a set of files and folders, a string, and more

• Sometimes more than just one argument is needed:

$ mv figs/cat.png best-figs/cat02.png

• man ls opens the manual page for ls. Press q to quit

• Manual pages come in numbered sections. Run man man to see them

• cht.sh gives short, practical answers straight in the terminal (you may need to install curl first):

$ curl cht.sh/ls

• Or you can use an LLM (or Google) to search for help on a command
• There’s a great chance that someone else has already asked the same question you have 😉

• You can also install the tldr tool (GitHub repository) which provides simplified help pages for common commands. Run it like this: tldr command

$ tldr ls

Successfully updated local database

ls

List directory contents.
More information: <https://www.gnu.org/software/coreutils/manual/html_node/ls-invocation.html>.

- List files one per line:
    ls -1

- List all files, including hidden files:
    ls [-a|--all]

- List files with a trailing symbol to indicate file type (directory/, symbolic_link@, executable*, ...):
    ls [-F|--classify]

... (the manual continues; press q to quit)

• For more info on how to get help, see here

man shows the page in a pager, a small viewer for long text. The keys you need:

• j and k, or the arrow keys, move one line
• d and u move half a page, space and b a whole one
• g and G jump to the very start and the very end
• /pattern searches forwards, ?pattern backwards. Then n and N for the next and previous hit
• q quits
• h lists every other key

• Phones and tablets hide the file system from you completely
• In data science you need to find your way around it quickly
• The questions that keep coming up:
  • Where am I right now?
  • Where is my home directory?
  • Where are my files, and where do I want to put them?
  • How do I get from here to there?

Key navigation commands are:

• pwd to print (the current) working directory
• cd to change directory

$ pwd

/Users/dafreir/Documents/github/datasci350/lectures/lecture-03

Paths can be absolute (from the top) or relative (from where you are). Prefer relative ones, and use these shortcuts:

• ~ your home folder
• . the current directory
• .. the directory above

Beware of spaces in names. Say you have a folder called “My Documents” (I’m looking at you, Windows)

• cd My Documents fails. Bash reads My and Documents as two separate arguments
• Two ways around it:
  • Quote the name: cd "My Documents"
  • Escape the space: cd My\ Documents. Tab completion does this for you
• Better still: never put spaces in names
• Use lowercase with dashes or underscores, like assignment-05 or assignment_05

Getting around

Working with files

A fuller list is here

• Managing files: create, copy, move, and delete them from the terminal
• Wildcards and find, so you can act on many files at once
• Working with text: cat, head, grep, and sed
• Pipes and redirects, chaining small tools into bigger ones. The Unix philosophy in action
• Loops and shell scripts, to automate what you would otherwise repeat
• Bring your laptop. We will be typing 🤓