1) The document discusses using the Python boto library to programmatically build cloud infrastructure on Amazon Web Services (AWS). 2) It provides examples of using boto to create an EC2 instance in Singapore with SSH access only, and to create an S3 bucket in Singapore configured to allow image uploads. 3) The document then explains how to put these components together into a photo uploading application that runs on an EC2 instance and stores photos in the S3 bucket, initializing the instance using user data to automatically deploy the application.

1. Building your Cloud
Infrastructure with Python
George Goh
14 June 2013
PyCon SG
1
+
1Friday, 14 June, 13

2. Agenda
• Python boto
• Examples
• Putting it together
2
2Friday, 14 June, 13

3. Using the infrastructure cloud,
we the programmers:
• see infrastructure as code.
• are able to version our infrastructure
configs, just like software.
• can experiment with different deployment
topologies as easily as experimenting with
different algorithms.
3
3Friday, 14 June, 13

4. 4
boto
4Friday, 14 June, 13

5. A Simple Example 1
5
1. Create an AWS EC2 instance in Singapore.
2. Using SSH-only login.
3. With firewall closing all ports except port
22 (for SSH).
5Friday, 14 June, 13

6. A Simple Example 1
6
import time
from boto.ec2 import connect_to_region
ec2_conn = connect_to_region("ap-southeast-1")
# create security group allowing SSH.
demo_sg = ec2_conn.create_security_group("demo-sg", "Demo Sec Group")
demo_sg.authorize("tcp", 22, 22, "0.0.0.0/0")
# create and save a public key for SSH login.
demo_key = ec2_conn.create_key_pair("demo-key")
demo_key.save(".")
# create the an instance using AMZN Linux.
resrv = ec2_conn.run_instances("ami-fade91a8",
key_name="demo-key",
instance_type="t1.micro",
security_groups=["demo-sg"])
# get the demo instance IP address.
demo = resrv.instances[0]
while demo.update() != 'running':
time.sleep(1)
print demo.ip_address
6Friday, 14 June, 13

7. boto
• http://docs.pythonboto.org/
• https://github.com/boto/boto
• It’s a python interface to Amazon Web
Services (AWS).
7
7Friday, 14 June, 13

8. Helicoptor view of boto
• Compute
• Elastic Compute Cloud (EC2)
• Elastic MapReduce (EMR)
• Auto Scaling
• Data Pipeline
• Elastic Transcoder
• Content Delivery
• CloudFront
• Database
• SimpleDB
• DynamoDB2
• DynamoDB
• Relational Data Services (RDS)
• ElastiCache
• Redshift
• Deployment and Management
• CloudFormation
• ElasticBeanstalk
• Identity & Access
• Identity and Access Management (IAM)
• Security Token Service (STS)
8
• Application Services
• Simple Workflow Service (SWF)
• Simple Queue Service (SQS)
• Simple Notification Service (SNS)
• Simple Email Service (SES)
• Cloudsearch
• Monitoring
• CloudWatch
• Networking
• Route 53
• Virtual Private Cloud (VPC)
• Elastic Load Balancer (ELB)
• Payments & Billing
• Flexible Payments Service
• Storage
• Simple Storage Service (S3)
• Amazon Glacier
• Google Cloud Storage
• Workforce
• Mechanical Turk
• Other
• Marketplace Web Services
8Friday, 14 June, 13

9. A Simple Example 2
9
1. Create an S3 bucket in Singapore.
2. Make it open for putting images.
9Friday, 14 June, 13

10. A Simple Example 2
10
import boto
from boto.s3.cors import CORSConfiguration
s3_conn = boto.connect_s3()
# create bucket with _globally_unique_ name.
bucket = s3_conn.create_bucket("unique_name", location="ap-southeast-1")
# create CORS config.
cors = CORSConfiguration()
cors.add_rule(allowed_method="GET", allowed_origin="*", allowed_header="*")
cors.add_rule(allowed_method="POST", allowed_origin="*", allowed_header="*")
cors.add_rule(allowed_method="PUT", allowed_origin="*", allowed_header="*")
# apply CORS config to our bucket.
bucket.set_cors(cors)
10Friday, 14 June, 13

11. 11
Putting it together
11Friday, 14 June, 13

12. 12
Photo Uploader
12Friday, 14 June, 13

13. 13
Photo Uploader
• Flask application on EC2 instance
• https://github.com/georgegoh/FlaskDirectUploader
• S3 bucket to store photos
13Friday, 14 June, 13

14. Instance initialization
• Do the following when an instance is
created:
• install software prerequisites
• clone the code from github
• initialize environment variables
• run the server
14
14Friday, 14 June, 13

15. Instance init - init.sh
15
#!/bin/sh
# Install software prerequisites.
yum install -y git
easy_install flask
# Clone the code from github.
git clone https://github.com/georgegoh/FlaskDirectUploader
# Initialize environment variables. BTW, these creds will be deleted
export AWS_ACCESS_KEY_ID=AKIAJH6C25DY6YS3Q7UQ
export AWS_SECRET_ACCESS_KEY=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
export S3_BUCKET=george.goh-pyconsg2013
# Run the server.
cd FlaskDirectUploader
python application.py
15Friday, 14 June, 13

16. Instance Initialization
16
• Attach init.sh to EC2 instance as user-data.
• init.sh runs once when the instance is
created.
16Friday, 14 June, 13

17. Modified from Simple Example 1
17
import time
import base64
from boto.ec2 import connect_to_region
ec2_conn = connect_to_region("ap-southeast-1")
# create security group allowing SSH and WWW.
demo_sg = ec2_conn.create_security_group("demo-sg", "Demo Sec Group")
demo_sg.authorize("tcp", 22, 22, "0.0.0.0/0")
demo_sg.authorize("tcp", 80, 80, "0.0.0.0/0")
# create and save a public key for SSH login.
demo_key = ec2_conn.create_key_pair("demo-key")
demo_key.save(".")
# Base64 encode the userdata.
with open('init.sh') as f:
userdata = f.read()
# create the an instance using AMZN Linux.
resrv = ec2_conn.run_instances("ami-fade91a8",
key_name="demo-key",
instance_type="t1.micro",
security_groups=["demo-sg"],
user_data=userdata)
# get the demo instance IP address.
demo = resrv.instances[0]
while demo.update() != 'running':
time.sleep(1)
print demo.ip_address
17Friday, 14 June, 13

18. Demo
18
18Friday, 14 June, 13

19. Recap
• boto
• createVM instances
• create storage
• use user-data to initializeVM instances
• too much to cover here
19
19Friday, 14 June, 13

20. What to do next
• fabric - http://docs.fabfile.org
• saltstack - http://saltstack.com/
• CloudFormation - http://aws.amazon.com/
cloudformation/
20
20Friday, 14 June, 13

21. Thank you
21
@georgegoh
21Friday, 14 June, 13