Summit ML Training

OLCF/IBM Machine Learning Training Materials

Tutorial information posted on github

• IBM Github ML Training

Training Documents

• Directory of ML Training Docs

Running Jupyter Notebooks on Summit

The training involved cloning a github repo into your account on Summit, creating an anaconda virtual environment, then doing an anaconda install of all the required libraries. Those instructions are covered in the documentation introductory materials.

The git repo contains a number of jupyter notebooks that can be run remote. Basic instructions are given on wiki pages for the IBM Github ML training, but a few steps are required to control the notebooks from your personal computers.

Two main steps include: 1) ssh into summit, activate the right conda envs, then run the commands below to identify an open port that the webbrowser can connect to on summit. 2) ssh into summit again using a version of ssh that redirects that terminal session to a port on your laptop that your browser can connect to. This will allow a browser on you laptop to run jobs on summit.

The details steps look include the following.

First, using your two factor login, ssh into summit into your own accoun there. Then, run these commands on Summit from your terminal window to find an unused port.

%for myport in {6818..11845}; do ! nc -z localhost ${myport} && break; done
%echo "ssh -NL $myport:$(hostname):$myport $USER@summit.ornl.olcf.gov"

#The commands above returned on Summit:
%ssh -NL 6818:login1:6818 pmaech@summit.ornl.olcf.gov

#Start a jupyter notebook on Summit with this:
%jupyter-notebook --no-browser --port=$myport --ip='0.0.0.0'

#This prints out http:links that you can post into browser on mac once
# the ssh forwarding is setup

#Now in another terminal window, on my mac, connect to summit again with this
ssh -NL 6818:login1.summit.olcf.ornl.gov:6818 pmaech@summit.ornl.olcf.gov

# This kind of hangs in the terminal window 2
#Now in a browwer on the mac, type the url that is given in the cluster 
# terminal window 1 when the jupyter notebooks was started
% http://127.0.0.1:6818/?token=461443ad9156ee01996321c23d62bab7831a2425ae35a6de

The browser then shows the directories on the cluster and you can run iphynb notebooks step by step from the browser

Related Entries

• Machine Learning
• Cloud Computing