Running Jupyter Notebook on AWS instances

My Wife is pursuing her Masters in Business Analytics course and in her final term project on building Recommender systems , wanted to use Jupyter Notebook that can be run from a cloud instance.

Lets run it on the AWS EC2 instances was my immediate answer. As a long time user of AWS, I knew it was the first thing to test. Lucky for her, her institution gave her credits to running AWS workloads.

While there are several in-depth articles available on the web on running Jupyter on AWS , this is a simplified step-by-step version that makes it easy to follow and requires no prior expertise at all.

Step 1:

As with all other setup, sign-in to your AWS account to access the console to launch your EC2 instance to choose the tight type of instance. For the current requirement , we launched using Amazon Linux AMI.

Amazon Linux AMI

Step 2:

You will need to choose your instance type to suit your requirement. For this setup, we chose a m4.xlarge instance type and created a 32GB storage space.

we initially chose the t2.micro instance to run test Jupyter , however , had issues with instance memory as it only offers 1GB and this project needed more with SciPy library.

Step 3 :

This step will be to select a Security group to allow who, what and where access to the instance can happen.

I have created a security group to allow SSH (with ICMP and also default Jupyter port 8888) to reach the instance and install the necessary packages.

Step 4 :

Access to Linux instance is via password-less key based authentication. We need to create a key pair at this stage and can use an existing or create a new pair.

For this setup, we created a new keypair and downloaded the mykeypair.pem to access the instance. It is very important to secure your private key.

Upon completion, this will launch the instance and you will see the details on the EC2 Dashboard. You will need note down the Public IP address of your newly launched instance.

Step 5 :

Follow the instructions to SSH into instances when using Linux or Windows. Once you have successfully logged into the instance via its Public IP address , you can now download and install Anaconda 5.x.y using the following command

$ wget https://repo.continuum.io/archive/Anaconda2-5.0.1-Linux-x86_64.sh
$ bash Anaconda2–5.0.1-Linux_x86_64.sh 

Follow the prompts and install Anaconda, then

$ source .bashrc 

Upon completion, traverse to the path where Jupyter is installed, usually at ~/anaconda2/bin/ and start the notebook via the following command

$ ./jupyter notebook --ip instance-internal-ip-address --no-browser --port=8888

Jupyter notebook is now ready for access. There are users who can use Local port forwarding to allow tunnel access via SSH and present Jupyter in the local port. In our case, i have opened the port 8888 on AWS security groups as shown in Step 3 and wanted to add password authentication to access the notebook.

To prepare a hashed password, you need to run

$ jupyter notebook password
Enter password:  ************
Verify password: ************
[NotebookPasswordApp] Wrote hashed password to /home/ec2-user/.jupyter/jupyter_notebook_config.json

Its a good practice to setup a notebook configuration file to make changes when required as documented here.

When you access your notebook via the Public IP address of the instance, you will be prompted to enter the password and you can now start working on your desired project.

There are several improvements that can be added to this setup :

1. TLS/SSL certificate via LetsEncrypt to make the site secure,
2. Assigning a domain name to rememeber the site easier than via IP,
3. Embedding notebook on a website etc.,

Please send me your comments / questions for improvement on this topic.