This document provides an overview of using Amazon Web Services (AWS) for scientific computing. It demonstrates how to launch an AWS server instance, install software like IPython Notebook, and configure security settings to access a Jupyter notebook remotely. Instructions are given for networking, storage, cost optimization techniques, and best practices for managing AWS resources.

1. Amazon Web Services for scientific computing
Alexander Novikov
June 18, 2015
Alexander Novikov AWS for scientific computing June 18, 2015 1 / 21

2. 1 Overview
2 Ipython Notebook example
3 Other stuff
4 Cleaning up
5 Troubleshooting
Alexander Novikov AWS for scientific computing June 18, 2015 2 / 21

3. Summary
1 Start 1 or 1000 configured servers in 5 minutes;
2 Run anything you want (you control OS and software);
3 Vary the size of your server (from micro instance to 32 cores 244 Gb
of RAM, to 4 GPUs onboard);
4 Pay per hour;
5 1 server for 100 hours cost the same as 100 servers for 1 hour.
Alexander Novikov AWS for scientific computing June 18, 2015 3 / 21

4. Amazon sell books don’t they?
Alexander Novikov AWS for scientific computing June 18, 2015 4 / 21

5. Amazon Web Services – AWS
We focus on Amazon Elastic Compute Cloud – EC2.
Alexander Novikov AWS for scientific computing June 18, 2015 5 / 21

6. Ipython in a cloud
You will run Ipython Notebook on AWS server in Frankfurt.
If you don’t have an AWS account but want to follow the tutorial, go to
the spreadsheet from my email and claim one of the servers.
Alexander Novikov AWS for scientific computing June 18, 2015 6 / 21

7. Running a server
1 Log into the AWS console;
2 Click on EC2;
3 Choose a region (Frankfurt is the closest; N. Virginia and Oregon are
cost effective);
4 Click Launch Instance (server);
5 Choose Ubuntu 14.04 (HVM);
6 Choose t2.micro (it’s free);
7 Next;
8 Choose the size of the root drive;
9 Name your instance;
10 Click Review and Launch;
11 Now review and click Launch.
Alexander Novikov AWS for scientific computing June 18, 2015 7 / 21

8. SSH keys
AWS use ssh keys to protect your connection to the server.
Name your key, click download, and save it in a safe location.
On Linux, you will need to restrict the access to the file:
$ cd Downloads/
$ chmod 400 key_name.pem
Alexander Novikov AWS for scientific computing June 18, 2015 8 / 21

9. Connect & update
On Linux and Max, connect with ssh tool
$ ssh -i tutorial.pem ubuntu@52.28.135.168
On Windows, convert your key to ppk and connect with PuTTY (user
name is ubuntu, ssh key file is key_name.ppk).
Update the server
$ sudo apt-get update
$ sudo apt-get upgrade
$ sudo reboot
Alexander Novikov AWS for scientific computing June 18, 2015 9 / 21

10. Install miniconda
Download the installer
$ wget https://repo.continuum.io/miniconda/...
Make it runnable and install (use tab for autocompletion)
$ chmod +x ./Miniconda-latest-Linux-x86_64.sh
$ ./Miniconda-latest-Linux-x86_64.sh
Now reconnect to the server (ctr + d to disconnect) and install conda
packages
$ conda install ipython ipython-notebook
Alexander Novikov AWS for scientific computing June 18, 2015 10 / 21

11. Networking
Generate an HTTPS certificate (you can leave the defaults for all the
questions)
$ ipython profile create nbserver
$ openssl req -x509 -nodes -days 365 -newkey rsa:1024
-keyout mycert.pem -out mycert.pem
Hash your password
$ ipython
from IPython.lib import passwd
passwd()
Alexander Novikov AWS for scientific computing June 18, 2015 11 / 21

12. Configure Ipython Notebook
Let’s find and edit the configuration file
$ find ./ -name "*notebook_config*"
$ nano
.ipython/profile_nbserver/ipython_notebook_config.py
Add the following lines after c = get_config()
c.NotebookApp.certfile = u'/home/ubuntu/mycert.pem'
c.NotebookApp.ip = '*'
c.NotebookApp.open_browser = False
c.NotebookApp.password = u'sha1:207eb1f4671f:92af6...'
c.NotebookApp.port = 8888
Alexander Novikov AWS for scientific computing June 18, 2015 12 / 21

13. Nano
Nano is a command line editor.
Essential commands (like ctr + o for saving and ctr + x to exit) are listed
on the bottom of the interface.
Alexander Novikov AWS for scientific computing June 18, 2015 13 / 21

14. Almost there
Run the notebook
$ ipython notebook --profile=nbserver
Find the public URL of your server in AWS console (e.g. https://
ec2-52-28-135-168.eu-central-1.compute.amazonaws.com:8888)
and open it in a browser.
Aaand it doesn’t work.
Alexander Novikov AWS for scientific computing June 18, 2015 14 / 21

15. Security groups
Go to the Security Groups tab and open incoming traffic on the 8888 port.
Alexander Novikov AWS for scientific computing June 18, 2015 15 / 21

16. Tmux
If you close the terminal now, the program will be killed.
Use tmux to run it in the background and be able to see the output.
$ tmux # Create new terminal.
$ ctr + b d # Detach from the current terminal.
$ tmux attach # Attach to an existing terminal.
Alexander Novikov AWS for scientific computing June 18, 2015 16 / 21

17. Terminating a server
When you terminate your server it’s hard drive is erased.
1 Next time you have to configure the server from scratch;
2 You cannot store any data (experiment results).
Alexander Novikov AWS for scientific computing June 18, 2015 17 / 21

18. Amazon Machine Image – AMI
Template from AWS instances are launched;
OS + software;
Launch different instance types from the same AMI;
You customize (install custom software, services, config, etc) and
make your own AMI;
AWS Marketplace makes it easy to launch a number of popular
applications (e.g, MongoDB, Cafee, OpenCV, etc.).
Alexander Novikov AWS for scientific computing June 18, 2015 18 / 21

19. Elastic Block Storage – EBS
Create a permanent magnetic or SSD volume of any size;
Connect (disconnect) the volume to any server in its data center
(availability group).
Alexander Novikov AWS for scientific computing June 18, 2015 19 / 21

20. Spot instances
Bid $X per hour;
If current price > bid, instance terminates;
You pay the market price, not the bids.
Alexander Novikov AWS for scientific computing June 18, 2015 20 / 21

21. Workflow
Create a small instance, set up all the software, download all the data;
Make an AMI;
Start a powerful spot instance (or a dozen) from your AMI;
Create a permanent volume and connect it to the instance;
Process the data (in tmux!) and write the results into the volume;
Terminate the instance.
Alexander Novikov AWS for scientific computing June 18, 2015 21 / 21

22. Other stuff
Run a program on a server and see its GUI on your laptop (ssh -X);
Use scp to download data to and from your server;
Use df -h to see all the disk mounted to the server;
Use htop to see running processes;
Set up a monitoring so when your server stops working you get an
email;
Explore AWS bill https://console.aws.amazon.com/billing;
Apply for an AWS academic grant.
Alexander Novikov AWS for scientific computing June 18, 2015 22 / 21

23. Cleaning up
Got to the console and terminate your server;
Go to the volume tab and remove all your volumes;
Alexander Novikov AWS for scientific computing June 18, 2015 23 / 21

24. Unprotected private key
If you see this message when trying to connect
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: UNPROTECTED PRIVATE KEY FILE! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0660 for ’tutorial.pem’ are too open.
It is required that your private key files are NOT
accessible by others.
This private key will be ignored.
bad permissions: ignore key: tutorial.pem
Permission denied (publickey).
Just set more restrictive permissions for your key file
$ chmod 400 key_name.pem
Alexander Novikov AWS for scientific computing June 18, 2015 22 / 21

25. Full volume
Your program says «Cannot write a file»?
Check your volumes, they are probably full. To list all mounted volumes
and their sizes use
$ df -h
Alexander Novikov AWS for scientific computing June 18, 2015 21 / 21

26. Slow down
Everything start beeing slow?
You probably burnt all your volumes I/O capacity. Wait so it can restore
or order a bigger volume (see https://docs.aws.amazon.com/AWSEC2/
latest/UserGuide/EBSVolumeTypes.html for details).
Alexander Novikov AWS for scientific computing June 18, 2015 21 / 21