Scalable Plone hosting with Amazon EC2 for Rice University’s Rhaptos open learning platform

Scalable Plone hosting
with Amazon EC2
for Rice University’s
Rhaptos open learning
platform
Nate Aune - Jazkarta.com
Plone Conference 2010
Bristol, UK
1Sunday, November 7, 2010

Agenda
• Beneﬁts of cloud computing
• What services does Amazon provide?
• 5 minute launch of Plone on EC2
• Case study: Rhaptos hosting on EC2
• Questions?

What makes the Cloud
so attractive?

Abstract resources
Focus on your needs, not on hardware specs.
As your needs change, so should your resources.

On-demand provisioning
Ask for what you need, exactly when you need it.
Get rid of it when you don’t need it.

Scalability in minutes
Scale out or in depending on usage needs.

Pay per consumption
No contracts or long-term commitments.
Pay only for what you use.

Efﬁciency of Experts
Utilize the skills, knowledge and resources of experts.

Most applications need:
• Compute
• Storage
• Messaging
• Payment
• Distribution
• Scale
• Analytics

Flexible infrastructure
• Elastic Computing Cloud (EC2)
• Amazon Machine Images (AMI)
• Elastic IPs
• Elastic Block Storage (EBS)
• Auto-scaling
• Elastic Load Balancing (ELB)

Other services
• Simple Storage Solution (S3)
• CloudWatch (monitoring)
• CloudFront (CDN)

http://uec-images.ubuntu.com/
Text

mr.awsome
Buildout recipe to manage EC2 instances

Add an AWS part to buildout
[buildout]
parts = aws
[aws]
recipe = zc.recipe.egg
eggs = mr.awsome
entry-points =
aws=mr.awsome:aws
assh=mr.awsome:aws_ssh
arguments = configpath="${buildout:directory}/etc"

Make a etc/aws.conf ﬁle
[securitygroup:demo-server]
description = Bristol Demo Server
connections =
tcp 22 22 0.0.0.0/0
tcp 80 80 0.0.0.0/0
tcp 8080 8080 0.0.0.0/0
[instance:demo-server]
keypair = bristol-keypair
securitygroups = demo-server
region = us-east-1
placement = us-east-1a
instance_type = t1.micro
image = ami-480df921
startup_script = startup-demo-server
fabfile = fabfile.py

Make a bash script:
etc/startup-demo-server
#!/bin/bash
set -e -x
export DEBIAN_FRONTEND=noninteractive
wget http://www.enfoldsystems.com/pubkey.gpg
apt-key add ./pubkey.gpg
echo "deb http://dist.clients.enfoldsystems.com/ubuntu lucid
universe" >> /etc/apt/sources.list
apt-get update && apt-get upgrade -y
apt-get install plone-default -y
/etc/init.d/plone-default

Set your access keys
#!/bin/bash
export AWS_ACCESS_KEY_ID="<your_access_key_id>"
export AWS_SECRET_ACCESS_KEY="<your_secret_access_key>"
Set these environment variables.
$ source setkeys.sh
$ env
File: setkeys.sh

Start the demo server
$ ./bin/aws start demo-server
INFO: Instance 'demo-server' unavailable
INFO: Creating instance 'demo-server'
INFO: Instance created, waiting until it's available
....
INFO: Instance 'demo-server' available
INFO: Instance running.
INFO: Instances DNS name ec2-50-16-25-92.compute-1.amazonaws.com
INFO: Instances public DNS name ec2-50-16-25-92.compute-1.amazonaws.com

Beneﬁts of EC2
• Seamless disaster recovery with repeatable
deployments using buildout and mr.awsome
• Easy launching of testing, staging and QA
servers that are isolated from your
production environment (stop when not
used to avoid paying for them)
• Pay-for-what-you-use metered pricing to
launch servers on-demand during peak
periods
• Auto-scaling to launch new Zeo client
machines and shut them down when no
longer needed

Connexions & Rhaptos
A case study in deploying OSS to the cloud

s
Log In Contact Us Report a Bug
Search
Search Site
Home Content Lenses About Us Help MyCNX
FEATURED CONTENT
New features
1 2 3
MY ACCOUNT
SPOTLIGHT
Skip
ConnexionsSections
You are here: Home
Connexions is:
a place to view and share educational material made of
small knowledge chunks called modules that can be
organized as courses, books, reports, etc. Anyone may view
or contribute:
authors create and collaborate
instructors rapidly build and share custom collections
learners find and explore content
More about us ...
Advanced Algebra II
The world of second-year
algebra comes alive in Kenny
Felder's Advanced Algebra II.
Felder, a high school math
teacher in North Carolina,
designed his course using a
nontraditional approach to a very
traditional subject. Centered
around a Homework and
Activities book, along with
accompanying Conceptual
Explanations and Teacher's Guide collections,
Advanced Algebra II tackles topics by focusing on
comprehension rather than straight lecture delivery.
Advanced Algebra II was selected by California's Free
Digital Textbook Initiative to be included in a list of
free textbooks available to California schools starting
in Fall 2009.
Collaborative Statistics
Collaborative Statistics was
written by two faculty members
FIND CONTENT
16547 reusable modules woven into 1007 collections.
or browse by ...
Language
Popularity
Title, author, etc.
CREATE CONTENT
Creating content in Connexions is as easy as 1, 2, 3:
Get an account
and log in to
your workspace.
Make a module
from scratch or
convert it from a
Word doc.
Publish your
works, sharing
them with the world.
Jump right in
Get an account
How to create a module in
minutes
How to create a collection
with existing modules
Guides and tutorials
New author guide
Connexions Tutorial and
Reference
Username
Password
Log in
Get an account
Forgot your password?
User feedback
"I enjoy using
the
Connexions
site and the
philosophy
behind it
encourages
one to learn
new fields and
to network with others. Your
site is my new homeschool on
the internet. It is helping me
to grow as a writer of
academic materials as I read
what others have written so as
to shape my own. I am also
amazed at the range of
institutions where your writers
come from. Thanks for the use
of these great facilities."
Go
Arts
Business
Humanities
Mathematics and Statistics
Science and Technology
Social Sciences
Search Content
Subject

Search
Login Preferences Help/Guide About Trac
Wiki Timeline Roadmap Browse Source View Tickets New Ticket Search
Start Page Index History Last Change
Welcome to Connexions and Rhaptos Software Development
Connexions ( cnx.org) is legos for education. It is a world-wide repository and publishing system to build textbooks, courses, lessons, and
journals. Individual authors, major content publishers, and educational organizations all contribute. Lenses provide a way for high-quality material
to be endorsed, recommended, categorized, and aligned to standards. Rhaptos is the open source software that runs the Connexions site and
Enterprise Rhaptos is the software for running private content installations. Read more about Connexions and Rhaptos mission and architecture
here.
Enterprise Rhaptos Connexions and Rhaptos Development
API's for Accessing Data Quality Assurance and Testing
Enterprise Rhaptos
Enterprise Rhaptos enables the creation, sharing, modification, and vetting of educational materials accessible to anyone, anywhere, anytime via
the World Wide Web. Rhaptos' unique features of permanent versions of all materials, a simple, semantic document language, and a powerful
lensing system for post-publication quality control, customized tagging, and community-based search and discovery are the engine behind a truly
reusable repository of knowledge and learning. Enterprise Rhaptos allows companies and individual organizations to host their own private
authoring platform and publishing repository.
Installation Instructions for Enterprise Rhaptos
API's for Accessing Connexions Data
Table of Contents
Welcome to Connexions and Rhaptos Software Development
Enterprise Rhaptos
API's for Accessing Connexions Data
Accessing content, feeds, and statistics
Downloading, transforming and branding content
Software Architecture and Application Programming Interfaces
Connexions Rhaptos Development
Developer Resources (Getting involved)
Roadmap, Designs, Desires
Starting Points for Connexions and Rhaptos Software Development
Quality Assurance and Testing
Test, Release, and Configuration Docs
Using the Connexions Rhaptos Trac Bug System
Release and Configuration Details for Connexions and Rhaptos
More about TRAC

Virtualization of Rhaptos
Two primary reasons to move to the cloud:
1) Hurricane territory (disaster recovery)
2) Increase adoption by other universities
Cost was also a consideration but not a
primary reason for carrying out the project.

Deploy to Amazon EC2
with Python scripts
• Define server profiles with OS dependencies
and turn into an AMI (Amazon Machine Image)
• Install & configure application on EBS volume
• Take snapshot of volume
• Launch new AMIs and attach EBS volume
created from saved snapshot.

Distributed Architecture
Internet
Webserver
Cache Sys
Load Balancer
s1
s3
s4 s5
s2
proxy
frontend1
frontend2
frontend3
backend

[macro:base-securitygroup]
connections =
tcp 22 22 0.0.0.0/0
tcp 80 80 0.0.0.0/0
[securitygroup:backend]
<= macro:base-securitygroup
description = Bristol backend machine running Zeo server
[securitygroup:frontend1]
description = Bristol frontend machine running Zeo client 1
connections =
tcp 8080 8080 0.0.0.0/0
[securitygroup:frontend2]
description = Bristol frontend machine running Zeo client 2
connections =
tcp 8080 8080 0.0.0.0/0
[securitygroup:proxy]
description = Bristol proxy machine running Apache, Varnish, HAProxy
aws.conf - deﬁne security groups

[macro:base-instance]
image = ami-480df921
keypair = bristol-keypair
region = us-east-1
placement = us-east-1c
instance_type = t1.micro
fabfile = fabfile.py
[instance:backend]
<= macro:base-instance
securitygroups = backend
startup_script = backend-startup.sh
ip = xxx.xxx.xxx.xxx
[instance:frontend1]
securitygroups = frontend1
startup_script = frontend-startup.sh
[instance:frontend2]
securitygroups = frontend2
startup_script = frontend-startup.sh
[instance:proxy]
securitygroups = proxy
startup_script = proxy-startup.sh

[buildout]
parts =
backend-script
frontend-script
proxy-script
[backend-script]
recipe = collective.recipe.template
input = deployment/templates/zope-startup-script.sh
output = deployment/backend-startup.sh
svn_url = http://svn.jazkarta.com/.../bristol-buildout/trunk
buildout_config = backend.cfg
[frontend-script]
input = deployment/templates/zope-startup-script.sh
output = deployment/frontend-startup.sh
buildout_config = frontend.cfg
[proxy-script]
input = deployment/templates/proxy-startup-script.sh
output = deployment/proxy-startup.sh
buildout_config = proxy.cfg

#!/bin/bash
set -e -x
export DEBIAN_FRONTEND=noninteractive
apt-get update && apt-get upgrade -y
echo "----- Install system packages for Plone"
apt-get -y install python2.6-dev subversion build-essential
echo "----- Create an unprivileged user zope with password disabled"
adduser zope --disabled-password --gecos ""
echo "----- Check out the buildout"
cd /home/zope
echo 'pn' | svn co http://svn.jazkarta.com/.../.../trunk buildout
echo "----- Make the zope user the owner of this directory"
chown -R zope:zope buildout
echo "----- Run the buildout as user zope"
cd buildout
sudo -u zope python2.6 bootstrap.py -c backend.cfg
sudo -u zope ./bin/buildout -v -c backend.cfg
echo "----- Start Zeo processes"
sudo -u zope ./bin/supervisord
sudo -u zope ./bin/supervisorctl status
echo "----- Make sure that Supervisor starts on reboot"
ln -s /home/zope/buildout/etc/supervisor /etc/init.d/supervisor
chmod +x /etc/init.d/supervisor
update-rc.d supervisor defaults

[buildout]
backend-dns-name = ec2-174-129-192-63.compute-1.amazonaws.com
frontend1-dns-name = ec2-174-129-116-33.compute-1.amazonaws.com
frontend2-dns-name = ec2-184-73-240-104.compute-1.amazonaws.com
Must deﬁne the FQDN
of the Zeo clients and
Zeo servers
When these names are resolved *on* the server, the DNS
server returns the private IP, which means you avoid paying
bandwidth charges for outgoing/incoming trafﬁc.

$ ./bin/aws start backend
$ ./bin/aws start frontend1
$ ./bin/aws start frontend2
$ ./bin/aws start proxy
Start up the servers
with one command

Fault tolerant
Eliminating single points of failure.

Distributed Architecture
Internet
Webserver
Cache Sys
Load Balancer
s1
s3
s4 s5
s2

Heartbeat
monitors Zeo
server and moves to
backup in case of
failure
DRDB
Distributed storage
system similar in
principle in RAID,
but which works
over the network.

document jazhar_diagrams.grafﬂe
modiﬁed 26 Oct 2009 20:51+0100 page 2
logical server
architecture
...
plone-fe-1.seas.harvard.eduplone-fe-0.seas.harvard.edu
plone-zope-0.web.private plone-zope-1.web.private
plone-zeo-0.web.private plone-zeo-1.web.private
Zope Client
ZEO Server ZEO Server
Zope Client Zope Client Zope Client
Apache
Varnish
Deliverance
Pound
browser clients
backend
frontend
internet
standard
connection
failover
connection
Apache
Varnish
Deliverance
Pound Supervisor
Supervisor
• HTTP:8221
• HTTP:8222 • HTTP:8223
• HTTP:8220
• HTTP:8889
• HTTP:9999• HTTP:80
• HTTPS:443
• HTTP:8000
• HTTP:9999
ldap
LDAPSupervisor
• HTTP:9999 • LDAP:389• HTTP:8222 • HTTP:8223
Supervisor
• HTTP:9999
host host
passive
resource
active
resource
Supervisor
• HTTP:9999
• HTTP:8220
• HTTP:8889
• HTTP:80
• HTTPS:443
• HTTP:8000

Improvements to the
deployment process
• time to launch a new site
was days
• brittle non-repeatable
installation process
• delay in procuring
hardware and installing
OS dependencies
• time to launch a new site
was minutes
• streamlined and
automated installation
process
• virtual machines are
launched instantly and
OS dependencies are
already installed on the disk
image.
Before After

Fabric
Scripts to deploy to remote hosts.
http://www.fabﬁle.org

Fabric
set(fab_user='plone',
fab_hosts=['localhost'],
local='/home/plone/prod/',
remote='/opt/zope/prod/',
site='budapest.buildout')
def deploy():
local('cd $(local)$(site)')
local('tar cvfz $(site).tgz buildout --exclude=.svn --exclude=*.pyc')
run('cd $(remote)$(site); ./bin/supervisorctl shutdown')
run('rm -rf $(remote)$(site)')
put('$(site).tar.gz', '$(remote)$(site).tar.gz')
run('cd $(root)$(site) && tar zxf $(site).tar.gz')
start()
def start():
run('cd $(remote)$(site)')
run('./bin/supervisord')

Deploy
$ fab deploy
• Provides a local packaging of the source into
a tarball
• Removes the old remotely deployed code
• Puts the source on the remote host
• Restarts the site
What does this one command do?

Beneﬁts of Fabric
• Chain local-command, remote-command,
and remote-copy commands
• Substitute variables, and the splitting of
distinct operations (like in the case of the
'restart' command) so you run them
separately, with a fab restart, for example.

Example Fabric file
• See Alex Clark’s Fabric file:
• http://github.com/aclark4life/Debian-Deploy-
Plone/blob/master/fabfile.py

collective.hostout
Buildout-based recipe for deploying to remote host
http://plone.org/products/collective.hostout

[buildout]
extends = buildout.cfg
parts += hostout prod
auto-checkout += collective.hostout
[sources]
collective.hostout = svn https://svn.plone.org/svn/collective/collective.hostout/trunk
[hostout]
recipe = collective.hostout
user = root
#password = blah
identity-file = /Users/nateaune/id_rsa-gsg-keypair
pre-commands =
# ${buildout:directory}/bin/supervisorctl shutdown || echo 'Unable to shutdown'
# post-commands =
# ${buildout:directory}/bin/supervisord
#effective-user = zope
[prod]
recipe = collective.hostout
extends = hostout
host = ec2-75-101-211-135.compute-1.amazonaws.com
buildout = buildout.cfg
path = /opt/zope/prod

Run the deploy
commands
$ ./bin/buildout -N
Installing prod.
Generated script '/bin/hostout'.
$ bin/hostout deploy
Invalid hostout hostouts are: prod
$ bin/hostout deploy prod
...

Go to Dylan’s talk
• Deploying with collective.hostout with
Dylan Jay.
• Today at 16:10-16:55 in Duchess 1+2.

Sprint idea
• Make a site to launch demo Plone sites using :
• t1.micro instances (free for 1st year)
• Enfold’s Ubuntu package
• xdv to show multitude of themes
• Could later include bundles of popular add-ons:
PloneEdu, multimedia, etc.

Questions?
• http://rhaptos.org
• http://pypi.python.org/pypi/mr.awsome
• http://fabﬁle.org
• http://pypi.python.org/pypi/collective.hostout
•

Scalable Plone hosting with Amazon EC2 for Rice University’s Rhaptos open learning platform

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Scalable Plone hosting with Amazon EC2 for Rice University’s Rhaptos open learning platform

Similar to Scalable Plone hosting with Amazon EC2 for Rice University’s Rhaptos open learning platform (20)

More from Jazkarta, Inc.

More from Jazkarta, Inc. (20)

Recently uploaded

Recently uploaded (20)

Scalable Plone hosting with Amazon EC2 for Rice University’s Rhaptos open learning platform