Your SlideShare is downloading. ×

Moving to the Cloud


Published on

"The Cloud" is a buzzword these days, but in this talk Nate Aune and Sally Kleinfeldt describe what you need to know about public and private clouds, why it's important, and why you might want to …

"The Cloud" is a buzzword these days, but in this talk Nate Aune and Sally Kleinfeldt describe what you need to know about public and private clouds, why it's important, and why you might want to consider it for your company/organization. We'll discuss the pros/cons of public cloud providers such as Amazon Web Services and Rackspace Cloud, and how you can build your own private cloud using OpenStack or Eucalpytus, or even a hybrid that leverages both public and private clouds. Lastly, we'll look at how Plone can be deployed onto the cloud, and best practices for setting up your Plone site in a flexible and scalable way to take full advantage of all that these cloud providers have to offer. In particular, we'll examine the Connexions/Rhaptos project as a case study, and how they benefited from an Amazon-backed infrastructure.

Video of the presentation is available here:

Published in: Technology
1 Like
  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. Moving to the Cloud Nate Aune(Presented by Sally Kleinfeldt) Plone Symposium East 2011 Penn State University
  • 2. Agenda• Benefits of cloud computing• What services does Amazon provide?• 5 minute launch of Plone on EC2• Case study: Rhaptos hosting on EC2• Questions?
  • 3. What makes the Cloud so attractive?
  • 4. Abstract resources Focus on your needs, not on hardware specs.As your needs change, so should your resources.
  • 5. On-demand provisioning Ask for what you need, exactly when you need it. Get rid of it when you don’t need it.
  • 6. Scalability in minutes Scale out or in depending on usage needs.
  • 7. Pay per consumption No contracts or long-term commitments. Pay only for what you use.
  • 8. Efficiency of ExpertsUtilize the skills, knowledge and resources of experts.
  • 9. Most applications need:• Compute• Storage• Messaging• Payment• Distribution• Scale• Analytics
  • 10. Amazon delivers this
  • 11. Characteristics of Truly Scalable Service• Increasing resources results in a proportional increase in performance• capable of handling heterogeneity, loosely coupled• operationally efficient• resilient• becomes more cost effective when it grows
  • 12. Flexible infrastructure• Elastic Computing Cloud (EC2) • Amazon Machine Images (AMI) • Elastic IPs • Elastic Block Storage (EBS) • EBS bootable AMI • Auto-scaling • Elastic Load Balancing (ELB)
  • 13. Other services• Simple Storage Solution (S3)• CloudFront (CDN)• CloudWatch (monitoring)
  • 14. Compared to Rackspace• Rackspace is still catching up to AWS (just announced load balancing a few weeks ago)• Rackspace is a bit easier to get started with • sends you the root password • machines already have a static IP • the disk storage is persisted• Prices are comparable to AWS • Cheapest Rackspace VM is $14/mo
  • 15. Private clouds• Provision virtual machines on your own hardware• Pros: you control your own machines.• Cons: you have to manage your own machines• Examples: Eucalyptus, OpenStack
  • 16. Text
  • 17. mr.awsomeBuildout recipe to manage EC2 instances
  • 18. Add an AWS part to buildout[buildout]parts = aws[aws]recipe = zc.recipe.eggeggs = mr.awsomeentry-points = aws=mr.awsome:aws assh=mr.awsome:aws_ssharguments = configpath="${buildout:directory}/etc"
  • 19. Make a etc/aws.conf file[securitygroup:demo-server]description = Bristol Demo Serverconnections = tcp 22 22 tcp 80 80 tcp 8080 8080[instance:demo-server]keypair = bristol-keypairsecuritygroups = demo-serverregion = us-east-1placement = us-east-1ainstance_type = t1.microimage = ami-480df921startup_script = startup-demo-serverfabfile =
  • 20. Make a bash script: etc/startup-demo-server#!/bin/bashset -e -xexport DEBIAN_FRONTEND=noninteractivewget add ./pubkey.gpgecho "deb luciduniverse" >> /etc/apt/sources.listapt-get update && apt-get upgrade -yapt-get install plone-default -y/etc/init.d/plone-default
  • 21. Set your access keysFile:!/bin/bashexport AWS_ACCESS_KEY_ID="<your_access_key_id>"export AWS_SECRET_ACCESS_KEY="<your_secret_access_key>"Set these environment variables.$ source$ env
  • 22. Start the demo server$ ./bin/aws start demo-serverINFO: Instance demo-server unavailableINFO: Creating instance demo-serverINFO: Instance created, waiting until its available....INFO: Instance demo-server availableINFO: Instance running.INFO: Instances DNS name ec2-50-16-25-92.compute-1.amazonaws.comINFO: Instances public DNS name
  • 23. Benefits of EC2• Seamless disaster recovery with repeatable deployments using buildout and mr.awsome• Easy launching of testing, staging and QA servers that are isolated from your production environment (stop when not used to avoid paying for them)• Pay-for-what-you-use metered pricing to launch servers on-demand during peak periods• Auto-scaling to launch new Zeo client machines and shut them down when no longer needed
  • 24. Connexions & Rhaptos A case study in deploying OSS to the cloud
  • 25. Log In Contact Us Report a Bug Search Home Content Lenses About Us Help MyCNXYou are here: Home MY ACCOUNTConnexions is: FIND CONTENTa place to view and share educational material made of Username 16547 reusable modules woven into 1007 collections.small knowledge chunks called modules that can be Passwordorganized as courses, books, reports, etc. Anyone may view Search Content Go Log inor contribute: or browse by ... Arts Get an account authors create and collaborate Subject Business Forgot your password? s instructors rapidly build and share custom collections Language Humanities Mathematics and Statistics learners find and explore content Popularity Science and TechnologyMore about us ... New features Title, author, etc. Social Sciences FEATURED CONTENT CREATE CONTENT Advanced Algebra II Creating content in Connexions is as easy as 1, 2, 3: The world of second-year SPOTLIGHT algebra comes alive in Kenny Felders Advanced Algebra II. User feedback Felder, a high school math "I enjoy using teacher in North Carolina, the designed his course using a Connexions nontraditional approach to a very site and the traditional subject. Centered around a Homework and 1 Get an account and log in to 2 Make a module from scratch or 3 Publish your works, sharing philosophy behind it Activities book, along with your workspace. convert it from a them with the world. encourages accompanying Conceptual Word doc. one to learn Explanations and Teachers Guide collections, new fields and Advanced Algebra II tackles topics by focusing on to network with others. Your Jump right in Guides and tutorials comprehension rather than straight lecture delivery. site is my new homeschool on Get an account New author guide the internet. It is helping me Advanced Algebra II was selected by Californias Free to grow as a writer of Digital Textbook Initiative to be included in a list of How to create a module in Connexions Tutorial and academic materials as I read free textbooks available to California schools starting minutes Reference what others have written so as in Fall 2009. How to create a collection to shape my own. I am also amazed at the range of with existing modules institutions where your writers Collaborative Statistics come from. Thanks for the use of these great facilities." Collaborative Statistics was written by two faculty members
  • 26. Search Login Preferences Help/Guide About Trac Wiki Timeline Roadmap Browse Source View Tickets New Ticket Search Start Page Index History Last ChangeWelcome to Connexions and Rhaptos Software Development Connexions ( is legos for education. It is a world-wide repository and publishing system to build textbooks, courses, lessons, and journals. Individual authors, major content publishers, and educational organizations all contribute. Lenses provide a way for high-quality material to be endorsed, recommended, categorized, and aligned to standards. Rhaptos is the open source software that runs the Connexions site and Enterprise Rhaptos is the software for running private content installations. Read more about Connexions and Rhaptos mission and architecture here. Enterprise Rhaptos Connexions and Rhaptos Development Table of Contents Welcome to Connexions and Rhaptos Software Development Enterprise Rhaptos APIs for Accessing Connexions Data Accessing content, feeds, and statistics APIs for Accessing Data Quality Assurance and Testing Downloading, transforming and branding content Software Architecture and Application Programming Interfaces Connexions Rhaptos Development Developer Resources (Getting involved) Roadmap, Designs, Desires Starting Points for Connexions and Rhaptos Software Development Quality Assurance and Testing Test, Release, and Configuration Docs Using the Connexions Rhaptos Trac Bug System Release and Configuration Details for Connexions and RhaptosEnterprise Rhaptos More about TRAC Enterprise Rhaptos enables the creation, sharing, modification, and vetting of educational materials accessible to anyone, anywhere, anytime via the World Wide Web. Rhaptos unique features of permanent versions of all materials, a simple, semantic document language, and a powerful lensing system for post-publication quality control, customized tagging, and community-based search and discovery are the engine behind a truly reusable repository of knowledge and learning. Enterprise Rhaptos allows companies and individual organizations to host their own private authoring platform and publishing repository. Installation Instructions for Enterprise RhaptosAPIs for Accessing Connexions Data
  • 27. Virtualization of Rhaptos Two primary reasons to move to the cloud: 1) Hurricane territory (disaster recovery) 2) Increase adoption by other universities Cost was also a consideration but not a primary reason for carrying out the project.
  • 28. Deploy to Amazon EC2 with Python scripts• Define server profiles with OS dependencies and turn into an AMI (Amazon Machine Image)• Install & configure application on EBS volume• Take snapshot of volume• Launch new AMIs and attach EBS volume created from saved snapshot.
  • 29. Distributed Architecture Internet s2 frontend1 Webserver Cache Sys s3 frontend2 Load Balancer backends1 proxy s4 frontend3 s5
  • 30. aws.conf - define security groups[macro:base-securitygroup]connections = tcp 22 22 tcp 80 80[securitygroup:backend]<= macro:base-securitygroupdescription = Bristol backend machine running Zeo server[securitygroup:frontend1]<= macro:base-securitygroupdescription = Bristol frontend machine running Zeo client 1connections = tcp 8080 8080[securitygroup:frontend2]<= macro:base-securitygroupdescription = Bristol frontend machine running Zeo client 2connections = tcp 8080 8080[securitygroup:proxy]<= macro:base-securitygroupdescription = Bristol proxy machine running Apache, Varnish, HAProxy
  • 31. [macro:base-instance]image = ami-480df921keypair = bristol-keypairregion = us-east-1placement = us-east-1cinstance_type = t1.microfabfile =[instance:backend]<= macro:base-instancesecuritygroups = backendstartup_script = backend-startup.ship =[instance:frontend1]<= macro:base-instancesecuritygroups = frontend1startup_script = frontend-startup.ship =[instance:frontend2]<= macro:base-instancesecuritygroups = frontend2startup_script = frontend-startup.ship =[instance:proxy]<= macro:base-instancesecuritygroups = proxystartup_script = proxy-startup.ship =
  • 32. [buildout]parts = backend-script frontend-script proxy-script[backend-script]recipe = collective.recipe.templateinput = deployment/templates/zope-startup-script.shoutput = deployment/backend-startup.shsvn_url = = backend.cfg[frontend-script]recipe = collective.recipe.templateinput = deployment/templates/zope-startup-script.shoutput = deployment/frontend-startup.shsvn_url = = frontend.cfg[proxy-script]recipe = collective.recipe.templateinput = deployment/templates/proxy-startup-script.shoutput = deployment/proxy-startup.shsvn_url = = proxy.cfg
  • 33. #!/bin/bashset -e -xexport DEBIAN_FRONTEND=noninteractiveapt-get update && apt-get upgrade -yecho "----- Install system packages for Plone"apt-get -y install python2.6-dev subversion build-essentialecho "----- Create an unprivileged user zope with password disabled"adduser zope --disabled-password --gecos ""echo "----- Check out the buildout"cd /home/zopeecho pn | svn co buildoutecho "----- Make the zope user the owner of this directory"chown -R zope:zope buildoutecho "----- Run the buildout as user zope"cd buildoutsudo -u zope python2.6 -c backend.cfgsudo -u zope ./bin/buildout -v -c backend.cfgecho "----- Start Zeo processes"sudo -u zope ./bin/supervisordsudo -u zope ./bin/supervisorctl statusecho "----- Make sure that Supervisor starts on reboot"ln -s /home/zope/buildout/etc/supervisor /etc/init.d/supervisorchmod +x /etc/init.d/supervisorupdate-rc.d supervisor defaults
  • 34. Must define the FQDN of the Zeo clients and Zeo servers [buildout] backend-dns-name = frontend1-dns-name = frontend2-dns-name = ec2-184-73-240-104.compute-1.amazonaws.comWhen these names are resolved *on* the server, the DNSserver returns the private IP, which means you avoid paying bandwidth charges for outgoing/incoming traffic.
  • 35. Start up the servers with one command$ ./bin/aws start backend$ ./bin/aws start frontend1$ ./bin/aws start frontend2$ ./bin/aws start proxy
  • 36. Fault tolerantEliminating single points of failure.
  • 37. Distributed Architecture Internet s2 Webserver Cache Sys s3 Load Balancers1 s4 s5
  • 38. Heartbeatmonitors Zeoserver and moves tobackup in case offailureDRDBDistributed storagesystem similar inprinciple in RAID,but which worksover the network.
  • 39. Improvements to the deployment process Before After• time to launch a new site was days• brittle non-repeatable installation process• delay in procuring hardware and installing OS dependencies
  • 40. Improvements to the deployment process Before After• time to launch a new site • time to launch a new site was days was minutes• brittle non-repeatable • streamlined and installation process automated installation process• delay in procuring hardware and installing • virtual machines are OS dependencies launched instantly and OS dependencies are already installed on the disk image.
  • 41. FabricScripts to deploy to remote hosts.
  • 42. Fabricset(fab_user=plone, fab_hosts=[localhost], local=/home/plone/prod/, remote=/opt/zope/prod/, site=budapest.buildout)def deploy(): local(cd $(local)$(site)) local(tar cvfz $(site).tgz buildout --exclude=.svn --exclude=*.pyc) run(cd $(remote)$(site); ./bin/supervisorctl shutdown) run(rm -rf $(remote)$(site)) put($(site).tar.gz, $(remote)$(site).tar.gz) run(cd $(root)$(site) && tar zxf $(site).tar.gz) start()def start(): run(cd $(remote)$(site)) run(./bin/supervisord)
  • 43. Deploy $ fab deployWhat does this one command do?• Provides a local packaging of the source into a tarball• Removes the old remotely deployed code• Puts the source on the remote host• Restarts the site
  • 44. Benefits of Fabric• Chain local-command, remote-command, and remote-copy commands• Substitute variables, and the splitting of distinct operations (like in the case of the restart command) so you run them separately, with a fab restart, for example.
  • 45. Example Fabric file• See Alex Clark’s Fabric file:• Plone/blob/master/
  • 46. Boto• Python library to script AWS services• Programmatically: • Launch new EC2 instances • Attach EBS volumes • Create S3 buckets
  • 47. collective.hostoutBuildout-based recipe for deploying to remote host
  • 48. [buildout]extends = buildout.cfgparts += hostout prodauto-checkout += collective.hostout[sources]collective.hostout = svn[hostout]recipe = collective.hostoutuser = root#password = blahidentity-file = /Users/nateaune/id_rsa-gsg-keypairpre-commands =# ${buildout:directory}/bin/supervisorctl shutdown || echo Unable to shutdown# post-commands =# ${buildout:directory}/bin/supervisord#effective-user = zope[prod]recipe = collective.hostoutextends = hostouthost = ec2-75-101-211-135.compute-1.amazonaws.combuildout = buildout.cfgpath = /opt/zope/prod
  • 49. Run the deploy commands$ ./bin/buildout -NInstalling prod.Generated script /bin/hostout.$ bin/hostout deployInvalid hostout hostouts are: prod$ bin/hostout deploy prod...
  • 50. Chef & Puppet• Configuration management tools to prepare machines to behave according to their role.• Unlike Fabric which “pushes” config to the server, with Chef/Puppet, the server “pulls” it’s configuration from a master node.• Usually overkill for a simple deployment, but with a multi-server deployment can be useful, especially to keep machines up-to-date.
  • 51. Questions?•••••