Running Jupyter notebooks on luria cluster nodes

Setup - Connect to your luria account and run the following commands to create a conda environment called jupyter, activate that environment and install the necessary software

module add miniconda3/v4

source /home/software/conda/miniconda3/bin/condainit

conda create --name jupyter

conda activate jupyter

conda install -c anaconda jupyter

Running jupyter notebooks -

On both mac and PC:

login to luria and run:

srun --pty -n 16 bash

to get all of the resources of a single cluster node in the normal queue. You need to note the cluster node being used. It will be something like: c##

then cd into your data directory: cd /net/host/data/share/users/kerberosID

NOTE: the path will vary by user and PI. The necessary information for host, share and kerberosID should be visible by running the following command from inside your luria home:

ls -lat | grep data

activate the jupyter conda environment with: module add miniconda3/v4

source /home/software/conda/miniconda3/bin/condainit

conda activate jupyter

From this cluster node with an active jupyter coda environment the execute the command: jupyter notebook --ip=0.0.0.0 --port=12123

That 12123 is a randomish 5 digit number, you could change to most anything between 10000 and 60000, you just need to keep track of what you use.

You should see a lot of output stream across the screen, leave that window open and running. There will be information in the output that you will need for the next step.

For Macs, you need to open up a new terminal window and run:

ssh -t [e-mail] -L 12123:localhost:12123 ssh cXX -L 12123:localhost:12123

the cXX needs to be replaced with the node that you are running on

this will create a 2nd “tunneled” connection to the luria cluster node that is running your jupyter notebook server.

For PCs running secureCRT, you just need to modify the existing connection. This is done by selecting OptionsàSessionOptions and the Port Forwarding item

In there, you need to click Add and add a tunnel according to slide 9 of: https://docs.google.com/presentation/d/1lP8ttdnWJYJ1pZuFgsh1WXlRaPemIMFtVAOiVmB0wCc/edit?usp=sharing

your port and your socket value will both be 5 digit number you select and you need to get the node correct.

Once those connections are set up, you can open a web browser on your mac and go to one of the URLs provided in the output of the notebook run.

There will be a couple of options for URLS, use the one that starts with something like:

http://127.0.0.1:12123...

If all is well, a jupyter notebook that is actually running on the cluster node will appear in your browser and the directories in your storage will be available.

In addition, I also want to point out another computational resource at MIT called engaging on demand: https://engaging-web.mit.edu/eofe-wiki/logging_in/engaging-ood/

This is a different computer that you have access to and it also offers jupyter notebooks with more substantial computational resources.