2013 PyCon SG - Building your cloud infrastructure with Python

1,364 views

Published on

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,364
On SlideShare
0
From Embeds
0
Number of Embeds
10
Actions
Shares
0
Downloads
31
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

2013 PyCon SG - Building your cloud infrastructure with Python

  1. 1. Building your CloudInfrastructure with PythonGeorge Goh14 June 2013PyCon SG1+1Friday, 14 June, 13
  2. 2. Agenda• Python boto• Examples• Putting it together22Friday, 14 June, 13
  3. 3. Using the infrastructure cloud,we the programmers:• see infrastructure as code.• are able to version our infrastructureconfigs, just like software.• can experiment with different deploymenttopologies as easily as experimenting withdifferent algorithms.33Friday, 14 June, 13
  4. 4. 4boto4Friday, 14 June, 13
  5. 5. A Simple Example 151. Create an AWS EC2 instance in Singapore.2. Using SSH-only login.3. With firewall closing all ports except port22 (for SSH).5Friday, 14 June, 13
  6. 6. A Simple Example 16import timefrom boto.ec2 import connect_to_region ec2_conn = connect_to_region("ap-southeast-1") # create security group allowing SSH.demo_sg = ec2_conn.create_security_group("demo-sg", "Demo Sec Group")demo_sg.authorize("tcp", 22, 22, "0.0.0.0/0") # create and save a public key for SSH login.demo_key = ec2_conn.create_key_pair("demo-key")demo_key.save(".") # create the an instance using AMZN Linux.resrv = ec2_conn.run_instances("ami-fade91a8",                               key_name="demo-key",                               instance_type="t1.micro",                               security_groups=["demo-sg"]) # get the demo instance IP address.demo = resrv.instances[0]while demo.update() != running:    time.sleep(1)print demo.ip_address6Friday, 14 June, 13
  7. 7. boto• http://docs.pythonboto.org/• https://github.com/boto/boto• It’s a python interface to Amazon WebServices (AWS).77Friday, 14 June, 13
  8. 8. Helicoptor view of boto• Compute• Elastic Compute Cloud (EC2)• Elastic MapReduce (EMR)• Auto Scaling• Data Pipeline• Elastic Transcoder• Content Delivery• CloudFront• Database• SimpleDB• DynamoDB2• DynamoDB• Relational Data Services (RDS)• ElastiCache• Redshift• Deployment and Management• CloudFormation• ElasticBeanstalk• Identity & Access• Identity and Access Management (IAM)• Security Token Service (STS)8• Application Services• Simple Workflow Service (SWF)• Simple Queue Service (SQS)• Simple Notification Service (SNS)• Simple Email Service (SES)• Cloudsearch• Monitoring• CloudWatch• Networking• Route 53• Virtual Private Cloud (VPC)• Elastic Load Balancer (ELB)• Payments & Billing• Flexible Payments Service• Storage• Simple Storage Service (S3)• Amazon Glacier• Google Cloud Storage• Workforce• Mechanical Turk• Other• Marketplace Web Services8Friday, 14 June, 13
  9. 9. A Simple Example 291. Create an S3 bucket in Singapore.2. Make it open for putting images.9Friday, 14 June, 13
  10. 10. A Simple Example 210import botofrom boto.s3.cors import CORSConfiguration s3_conn = boto.connect_s3() # create bucket with _globally_unique_ name.bucket = s3_conn.create_bucket("unique_name", location="ap-southeast-1") # create CORS config.cors = CORSConfiguration()cors.add_rule(allowed_method="GET", allowed_origin="*", allowed_header="*")cors.add_rule(allowed_method="POST", allowed_origin="*", allowed_header="*")cors.add_rule(allowed_method="PUT", allowed_origin="*", allowed_header="*") # apply CORS config to our bucket.bucket.set_cors(cors)10Friday, 14 June, 13
  11. 11. 11Putting it together11Friday, 14 June, 13
  12. 12. 12Photo Uploader12Friday, 14 June, 13
  13. 13. 13Photo Uploader• Flask application on EC2 instance• https://github.com/georgegoh/FlaskDirectUploader• S3 bucket to store photos13Friday, 14 June, 13
  14. 14. Instance initialization• Do the following when an instance iscreated:• install software prerequisites• clone the code from github• initialize environment variables• run the server1414Friday, 14 June, 13
  15. 15. Instance init - init.sh15#!/bin/sh # Install software prerequisites.yum install -y giteasy_install flask # Clone the code from github.git clone https://github.com/georgegoh/FlaskDirectUploader # Initialize environment variables. BTW, these creds will be deletedexport AWS_ACCESS_KEY_ID=AKIAJH6C25DY6YS3Q7UQexport AWS_SECRET_ACCESS_KEY=fvmq8a/oW7AtzD64tBdFYteMmgR7maSZkTFuNySZexport S3_BUCKET=george.goh-pyconsg2013 # Run the server.cd FlaskDirectUploaderpython application.py15Friday, 14 June, 13
  16. 16. Instance Initialization16• Attach init.sh to EC2 instance as user-data.• init.sh runs once when the instance iscreated.16Friday, 14 June, 13
  17. 17. Modified from Simple Example 117import timeimport base64from boto.ec2 import connect_to_region ec2_conn = connect_to_region("ap-southeast-1") # create security group allowing SSH and WWW.demo_sg = ec2_conn.create_security_group("demo-sg", "Demo Sec Group")demo_sg.authorize("tcp", 22, 22, "0.0.0.0/0")demo_sg.authorize("tcp", 80, 80, "0.0.0.0/0") # create and save a public key for SSH login.demo_key = ec2_conn.create_key_pair("demo-key")demo_key.save(".") # Base64 encode the userdata.with open(init.sh) as f:    userdata = f.read() # create the an instance using AMZN Linux.resrv = ec2_conn.run_instances("ami-fade91a8",                               key_name="demo-key",                               instance_type="t1.micro",                               security_groups=["demo-sg"],                               user_data=userdata) # get the demo instance IP address.demo = resrv.instances[0]while demo.update() != running:    time.sleep(1)print demo.ip_address17Friday, 14 June, 13
  18. 18. Demo1818Friday, 14 June, 13
  19. 19. Recap• boto• createVM instances• create storage• use user-data to initializeVM instances• too much to cover here1919Friday, 14 June, 13
  20. 20. What to do next• fabric - http://docs.fabfile.org• saltstack - http://saltstack.com/• CloudFormation - http://aws.amazon.com/cloudformation/2020Friday, 14 June, 13
  21. 21. Thank you21@georgegoh21Friday, 14 June, 13

×