Summit ML Training - Revision history

Maechlin: /* Training Documents */

2020-10-13T03:34:56Z

Training Documents

Maechlin: /* Running Jupyter Notebooks on Summit */

2020-10-13T02:44:08Z

Running Jupyter Notebooks on Summit

Maechlin at 02:19, 13 October 2020

2020-10-13T02:19:08Z

Maechlin: Created page with "OLCF/IBM Machine Learning Training Materials == Tutorial information posted on github == *[https://github.com/dustinvanstee/aicoc-ai-immersion/tree/master/ORNL IBM Github ML T..."

2020-10-13T02:08:10Z

Created page with "OLCF/IBM Machine Learning Training Materials == Tutorial information posted on github == *[https://github.com/dustinvanstee/aicoc-ai-immersion/tree/master/ORNL IBM Github ML T..."

New page

OLCF/IBM Machine Learning Training Materials
== Tutorial information posted on github ==
*[https://github.com/dustinvanstee/aicoc-ai-immersion/tree/master/ORNL IBM Github ML Training]

== Training Documents ==
*[http://hypocenter.usc.edu/research/OLCF_ML_Training Directory of ML Training Docs]

== Running Jupyter Notebooks on Summit ==
The training involved cloning a github repo into your account on Summit, creating an anaconda virtual environment, then doing an anaconda install of all the required libraries. Those instructions are covered in the documentation introductory materials.

The git repo contains a number of jupyter notebooks that can be run remote. Basic instructions are given on wiki pages for the IBM Github ML training, but a few steps are required to control the notebooks from your personal computers.

Two main steps include:
1) ssh into summit and run the commands below to identify an open port that the webbrowser can connect to on summit.
2) ssh into summit again using a version of ssh that redirects that terminal session to a port on your laptop that your browser can connect to. This will allow a browser on you laptop to run jobs on summit.

The details steps look include the following.

<pre>
First, using your two factor login, ssh into summit into your own accoun there. Then, run these commands on Summit from your terminal window to find an unused port.

%for myport in {6818..11845}; do ! nc -z localhost ${myport} && break; done
%echo "ssh -NL $myport:$(hostname):$myport $USER@summit.ornl.olcf.gov"

#The commands above returned on Summit:
%ssh -NL 6818:login1:6818 pmaech@summit.ornl.olcf.gov

#Start a jupyter notebook on Summit with this:
%jupyter-notebook --no-browser --port=$myport --ip='0.0.0.0'

#This prints out http:links that you can post into browser on mac once
# the ssh forwarding is setup

#Now in another terminal window, on my mac, connect to summit again with this
ssh -NL 6818:login1.summit.olcf.ornl.gov:6818 pmaech@summit.ornl.olcf.gov

# This kind of hangs in the terminal window 2
#Now in a browwer on the mac, type the url that is given in the cluster
# terminal window 1 when the jupyter notebooks was started
% http://127.0.0.1:6818/?token=461443ad9156ee01996321c23d62bab7831a2425ae35a6de

The browser then shows the directories on the cluster and you can run iphynb notebooks step by step from the browser

</pre>

== Related Entries ==
*[[Machine Learning]]
*[[Cloud Computing]]

@@ Line 4: / Line 4: @@
 == Training Documents ==
-*[http://hypocenter.usc.edu/research/OLCF_ML_Training Directory of ML Training Docs]
+*[http://hypocenter.usc.edu/research/OLCF_ML_Training/ OLCF ML Training Docs]
 == Running Jupyter Notebooks on Summit ==

@@ Line 22: / Line 22: @@
 First, using your two factor login, ssh into summit into your own accoun there. Then, run these commands on Summit from your terminal window to find an unused port.
 %echo "ssh -NL $myport:$(hostname):$myport $USER@summit.ornl.olcf.gov"
 #The commands above returned on Summit:
-%ssh -NL 6818:login1:6818 pmaech@summit.ornl.olcf.gov
+%ssh -NL 7046:login1:7046 pmaech@summit.ornl.olcf.gov
 #Start a jupyter notebook on Summit with this:
 %jupyter-notebook --no-browser --port=$myport --ip='0.0.0.0'
-#This prints out http:links that you can post into browser on mac once
+which returns something like this:
-# the ssh forwarding is setup
+(wmlce17-ornl) [pmaech@login4.summit ~]$ jupyter-notebook --no-browser --port=$myport --ip='0.0.0.0'
 #Now in another terminal window, on my mac, connect to summit again with this
-ssh -NL 6818:login1.summit.olcf.ornl.gov:6818 pmaech@summit.ornl.olcf.gov
+ssh -NL 7046:login4.summit.olcf.ornl.gov:7046 pmaech@summit.olcf.ornl.gov
 # This kind of hangs in the terminal window 2
 #Now in a browwer on the mac, type the url that is given in the cluster
 # terminal window 1 when the jupyter notebooks was started
-% http://127.0.0.1:6818/?token=461443ad9156ee01996321c23d62bab7831a2425ae35a6de
+% http://127.0.0.1:7046/?token=654efaa66e39e2dadb80868ceafc02583b3685c386c2404c
 The browser then shows the directories on the cluster and you can run iphynb notebooks step by step from the browser
 </pre>

@@ Line 13: / Line 13: @@
 Two main steps include:
-) ssh into summit and run the commands below to identify an open port that the webbrowser can connect to on summit.
+) ssh into summit, activate the right conda envs, then run the commands below to identify an open port that the webbrowser can connect to on summit.
 ) ssh into summit again using a version of ssh that redirects that terminal session to a port on your laptop that your browser can connect to. This will allow a browser on you laptop to run jobs on summit.