HPC Carpentry - Conestoga College

May 26-27, 2018

10am - 5pm

Instructors: Pawel Pomorski, Tyson Whitehead, Isaac Ye

Helpers: Ed Armstrong

General Information

HPC Carpentry is a new initiative designed to teach researchers the basics of scientific computing, in particular accessing and using high-performance computing installations that have become commonplace in modern research. This hands-on workshop will cover basic concepts and tools, including using a cluster, programming in Python, and pipeline/task automation. Participants will be encouraged to help one another and to apply what they have learned to their own research problems. For more information on what we teach and why, please see the HPC Carpentry website.

Who: The course is aimed at graduate students and other researchers looking to take advantage of high-performance compute clusters and perform reproducible research. You don't need to have any previous knowledge of the tools that will be presented at the workshop. This workshop is a fantastic chance to get started with scientific computing.

Where: DMB-1E05, 299 Doon Valley Dr, Kitchener. Get directions with OpenStreetMap or Google Maps.

When: May 26-27, 2018. Add to your Google Calendar.

Requirements: Participants must bring a laptop with a Mac, Linux, or Windows operating system (not a tablet, Chromebook, etc.) that they have administrative privileges on. They should have a few specific software packages installed (listed below). They are also required to abide by HPC Carpentry's Code of Conduct.

Accessibility: We are committed to making this workshop accessible to everybody. The workshop organizers have checked that:

Contact: Please email ddvorski@conestogac.on.ca for more information.

Surveys

Please be sure to complete these surveys before and after the workshop.

Pre-workshop Survey

Post-workshop Survey

Schedule

Day 1

10:00 Introduction to the UNIX shell
13:00 Lunch break
14:00 Using a compute cluster
17:00 Wrap-up

Day 2

10:00 Programming with Python
13:00 Lunch break
14:00 Analysis pipelines with Snakemake
17:00 Wrap-up

We will use this collaborative document for chatting, taking notes, and sharing URLs and bits of code.

Syllabus

Introduction to High Performance Computing

  • Connecting to a cluster
  • Shell basics: files, directories, and text editing
  • Writing and running shell scripts
  • Environment variables
  • Scheduling jobs on a cluster
  • Accessing and compiling software
  • Transferring files
  • Reference...

Programming in Python

  • Basic Python programming and syntax
  • Writing and executing Python scripts
  • Storing data using lists, dicts, and Numpy arrays
  • Functions and conditions
  • Parallel programming concepts
  • Writing an analysis pipelines with Snakemake
  • Executing a pipeline across a cluster or local computer
  • Reference...

Setup

To participate in a HPC Carpentry workshop, you will need access to the software described below. In addition, you will need an up-to-date web browser.

We maintain a list of common issues that occur during installation as a reference for instructors that may be useful on the Configuration Problems and Solutions wiki page.

SSH Client

A large portion of this workshop will involve connecting to and using a remote computing cluster. To do this, we will use a SSH client.

Windows

You will need to download and install MobaXterm, a widely-used SSH client for Windows. MobaXterm can be downloaded from the following link: installer.

macOS

macOS ships with an SSH client by default, so there is no need to install anything. You access Bash and an SSH client from the Terminal (found in /Applications/Utilities). See the Git installation video tutorial for an example on how to open the Terminal. You may want to keep Terminal in your dock for this workshop.

Linux

Linux comes with an SSH client installed by default - there is no need to install anything.

Python

Python is a popular language for research computing, and great for general-purpose programming as well. Installing all of its research packages individually can be a bit difficult, so we recommend Anaconda, an all-in-one installer.

Regardless of how you choose to install it, please make sure you install Python version 3.x (e.g., 3.6 is fine).

We will teach Python using the Jupyter notebook, a programming environment that runs in a web browser. For this to work you will need a reasonably up-to-date browser. The current versions of the Chrome, Safari and Firefox browsers are all supported (some older browsers, including Internet Explorer version 9 and below, are not).

Windows

Video Tutorial
  1. Open https://www.anaconda.com/download/#windows with your web browser.
  2. Download the Python 3 installer for Windows.
  3. Install Python 3 using all of the defaults for installation except make sure to check Make Anaconda the default Python.

macOS

Video Tutorial
  1. Open https://www.anaconda.com/download/#macos with your web browser.
  2. Download the Python 3 installer for OS X.
  3. Install Python 3 using all of the defaults for installation.

Linux

  1. Open https://www.anaconda.com/download/#linux with your web browser.
  2. Download the Python 3 installer for Linux.
    (The installation requires using the shell. If you aren't comfortable doing the installation yourself stop here and request help at the workshop.)
  3. Open a terminal window.
  4. Type 
    bash Anaconda3-
    and then press tab. The name of the file you just downloaded should appear. If it does not, navigate to the folder where you downloaded the file, for example with: 
    cd Downloads
    Then, try again.
  5. Press enter. You will follow the text-only prompts. To move through the text, press the space key. Type yes and press enter to approve the license. Press enter to approve the default location for the files. Type yes and press enter to prepend Anaconda to your PATH (this makes the Anaconda distribution the default Python).
  6. Close the terminal window.