Castles in the Cloud:
Developing with Google App Engine
Kevin Noonan
Calbane Ltd.
IT@Cork, 20th April, 2009

Castles in the Cloud
Introduction

A working definition of Cloudware
• Outsourced Virtualization
o Virtual server running on outsourced server farm

The Cloud Computing Pyramid
•Michael Sheehan of GoGrid devised this classification

Main Players

Other Players
• Microsoft Windows Azure
o Including .NET Live Services
• Force.com from Salesforce
• SUN (recently announced)
• Mosso
• GoGrid
• FlexiScale
• Hosting365
• ...

What's Cloudware good for? (I)‫‏‬
• Guerilla development
o Pay-per-use model is ideal for startups
o Also good for guerillas inside the enterprise...

What's Cloudware good for? (II)‫‏‬
• Scaling up
o Clone the virtual server
o pay for extra resources (bandwidth, storage, RAM, etc.)‫‏‬

What's Cloudware good for? (III)‫‏‬
• In general, outsourcing IT infrastructure
o pay someone else to run your servers, network, database...

Castles in the Cloud
Who's using Cloudware & why not?

Startups & Large Firms on AWS
• Startups
o Twitter
o SmugMug
o Pixenate (Walter Higgin's site, from here in Ireland)‫‏‬
o Many Facebook apps, e.g. iLIke
• Large firms
o NASDAQ
o Activision
o Business Objects
o Hasbro
o New York Times
• See: http://aws.amazon.com/solutions/case-studies/

Startups on Google App Engine
• Startups
o BuddyPoke! (30M daily pageviews on App Engine)‫‏‬
o Pixverse (PixChat)‫‏‬
o Hyperic (Cloudstatus)‫‏‬
• One large company's experiment:
o GiftTag from BestBuy
• http://code.google.com/appengine/casestudies.html

Barriers to Enterprise Adoption
• Immaturity of the technology
o Although AWS is over three years old
• Concerns about reliability
• Concerns about portability
o no two clouds are the same
• Bureaucracy & inertia
• Lock-in to legacy systems
• Fear of loss-of-control & of job-losses
• ...

Black days in the Cloud
• Amazon S3 went down for two consecutive hours in Feb 2008.
• Amazon S3 (& Amazon SQS) went down for eight consecutive US daytime hours in
July 2008.
• App Engine had a five-hour partial-outage in June '08
• App Engine had a problem of increased latency (for web-serving) over a few days in
early Mar '09.
• Group for notification of App Engine downtime
o http://groups.google.com/group/google-appengine-downtime-notify

Castles in the Cloud
Google App Engine

Python: The Original of the Species

Java: the All New Flavour

Languages on App Engine's JVM
• Java
• JRuby
• Jython
• BeanShell
• JavaScript
• Scala
• Groovy
• …
• Will it Play in App Engine?
http://groups.google.com/group/google-appengine-java/web/will-it-play-in-app-engine

What is Google App Engine?
• Complete web stack
o Soon to be more general computing platform
• Runs on Google's infrastructure
• Bigtable Datastore
o Like a sorted hashtable
o Not a relational database
o Use GQL (with Python) instead of SQL
o Use JDO or JPA (with Java) for datastore work
• Memcache
• Image API
• Mail API
• Python library available for Force.com web services
• SDK for development on Windows/Mac/Linux
o Run & test locally before uploading to Cloud

A fabulous app: 'Gaebar'
Alternative: use AppRocket to replicate an App Engine datastore to an SQL database
http://code.google.com/p/approcket/

Limitations of Google App Engine
• Applications run in “The Sandbox”:
http://code.google.com/appengine/docs/java/runtime.html#The_Sandbox
• Native Python modules (written in C) do not work.
• Many Java libraries will not work (but many have been patched or will be).
See the JRE whitelist: http://code.google.com/appengine/docs/java/jrewhitelist.html
• No “long-lived” processing (requests must return in 30 seconds).
• Limited “offline processing” (this is coming)
o no forks, no spawned processes, no threads
o Just scheduled tasks, which work like “cron jobs” on UNIX systems
• HTTPS only on XYZ.appspot.com
o and only with Google's certificate.
o Due to limitation of SSL protocol, as applied to Google's Cloud.

Roadmap to June for App Engine
• Service for storing and serving large files
• Task queues for performing background processing
• Incoming email support
• XMPP (Jabber) API

'Batch processing' on App Engine today
• Use App Engine's task scheduling for “cron jobs” spaced a minute apart (or more seldom)
• A hack for more frequent jobs: run a cron-job on your PC or server to periodically hit a URL on App
Engine and kick off a task
• An outsourced hack for more frequent jobs: use webcron.org
o do the same as above (perhaps more reliably!)
o Cost: one hundredth of one Euro cent per 'hit'
• More sophisticated: hitch the clouds together
o 'App Engine Patch' has support for 'Boto'
o 'Boto' is a Python library for Amazon
o Use Boto (on App Engine) to queue work-packets on Amazon's Simple Queue Service (SQS)
o Kick off tasks on App Engine with a method above.
o Cost: $0.01 per 10,000 requests
o (+ $0.10 per GB transferred in...)

Web tools for App Engine
• “Xmas gifts” - announced in late December 2008
• Zoho Creator
o generates code and web-pages for deployment
o http://creator.zoho.com/
• App Engine Site Creator
o “Lightweight CMS” (with GUI interface)‫‏‬
o http://code.google.com/p/app-engine-site-creator/
o Apache 2.0 license

Billing on Google App Engine
• True pay-per-use
o free quotas & no bill to pay if quotas not exceeded.
o quite different model to AWS
• Developer can assign budget per resource.
• Prices
o $0.10 per CPU core-hour
o $0.15 per GB-month of storage
o $0.12 per GB outgoing bandwidth
o $0.10 per GB incoming bandwidth
o $0.0001 per email recipient for emails sent by the application

Free Quotas on App Engine
•Daily outgoing bandwidth: 10 GB
•Daily incoming bandwidth 10 GB
•Daily requests 1.3 million
• Daily CPU-hours: 46
• More quotas (mail, datastore, images, memcache)
http://code.google.com/appengine/docs/quotas.html

Other limitations on App Engine
• 1 GB of free storage per app.
•100 MB Memcache storage limit
• App Engine apps can now receive requests and send responses of up to 10MB.
• Caveat: 1 MB limit on datastore/memcache API call
• You can deploy up to 1,000 code and 1,000 static files with an app.
Workaround: use ZipImport (with Python) to store files in a Zip-archive:
http://code.google.com/appengine/articles/django10_zipimport.html
• 150MB cumulative limit on an app's code and static files
(as opposed to storage in the datastore).

Atypical Uses of App Engine
• Run Open Source software (e.g. blog / CMS):
http:/groups.google.com/group/google-appengine/web/google-app-engine-open-source-projects
• Store your static files on App Engine
o You can use App Engine as rudimentary 'Content Delivery Network'
• Expose a web-service on App Engine (while doing your main development elsewhere)
REST library for Django & App Engine: http://code.google.com/p/python-rest/
• Expose an interface to BigTable and exploit its scalable storage:
REST interface to datastore: http://code.google.com/p/app3/
• Use RESTlets for Java equivalent of the two above:
http://wiki.restlet.org/docs_1.2/13-restlet/252-restlet.html

Brief Case Study: LongURLPlease
• LongURLPlease expands “tiny URLs” to the original long versions.
• Original version written in a day by Darragh Curran, with barely any previous Python experience.
• Launched publicly on December 23rd, 2008.
• Rapidly became a platform in itself: at least nine other applications built on the API within 8 weeks
• Over four thousand downloads of the Firefox plugin
• 100K – 200K requests per day (each request can batch up to 4 URLs)
• Running costs: zero (since its usage is well below the free quotas)

Where to go from here (Python)
• App Engine Helper tutorial
o http://blog.pas.net.au/2009/02/17/django-and-google-app-engine-tutorial/
• App Engine Patch article (I recommend AEP over the “helper” above)
o http://code.google.com/appengine/articles/app-engine-patch.html
• App Engine cheatsheet: http://short.ie/cheatongae
• Code samples
o http://code.google.com/p/google-app-engine-samples/downloads/list

Where to go from here (Java)
• Sign up for the Java “early look”
http://appengine.google.com/promo/java_runtime
• Install the Eclipse plugin (& SDK)
http://code.google.com/appengine/docs/java/tools/eclipse.html
(The Java samples are in the SDK.)
• Read Google's docs and follow the tutorial:
http://code.google.com/appengine/docs/java/overview.html
• Or pick your own language and get up-and-running on the JVM:
http://groups.google.com/group/google-appengine-java/web/will-it-play-in-app-engine

Castles in the Cloud
Reference
2/27/2009

Articles & Papers
• 10 Reasons Enterprises Aren't Ready to Trust the Cloud
o http://gigaom.com/2008/07/01/10-reasons-enterprises-arent-ready-to-trust-the-cloud/
• The Information Factories, by George Gilder
o http://www.wired.com/wired/archive/14.10/cloudware_pr.html
o \"When the network becomes as fast as the processor, the computer hollows out and spreads across the network.\"
- in 1993, from Sun Microsystems' CTO, Eric Schmidt (now CEO of Google)
• Above the Clouds: A Berkeley View of Cloud Computing
o http://www.eecs.berkeley.edu/Pubs/TechRpts/2009/EECS-2009-28.html

Resources for Google App Engine
• Google App Engine: http://appengine.google.com
• Videos from conference Google held in 2008
o http://sites.google.com/site/io/
• Chat with developers from the App Engine team
o first Wednesday of the month: 3am next day (Dublin/London time).
o third Wednesday of the month: 5pm (Dublin/London time).
o IRC channel: #appengine on irc.freenode.net
o http://en.wikipedia.org/wiki/List_of_IRC_clients
• Google Group
o http://groups.google.com/group/google-appengine/
• Paper on Bigtable:
o http://labs.google.com/papers/bigtable.html

Book (I)‫‏‬
Developing with Google App Engine
by Eugene Ciurana
Published:
Feb 2009
http://apress.com/

Book (II)‫‏‬
Google App Engine in Action
by Noah Gift & Michael Orr
To be published:
May 2009
http://www.manning.com/gift/

Book (III)‫‏‬
Google App Engine
by Dan Sanderson
To be published:
Sept 2009
http://oreilly.com/

Castles in the Cloud
Wrapping up
2/27/2009

Credits
• Photos used under Creative Commons license:
• Photo Credits
o http://www.flickr.com/photos/notsogoodphotography/
o http://www.flickr.com/photos/barto/
o http://www.flickr.com/photos/kitoy/
o http://www.flickr.com/photos/wolfgangstaudt/
o http://www.flickr.com/photos/motumboe/
o http://www.flickr.com/photos/ragnar1984/
o http://www.flickr.com/photos/janekm/
o http://www.flickr.com/photos/raylopez/
o http://www.flickr.com/photos/pagedooley/
o http://www.flickr.com/photos/gustty/
o http://www.flickr.com/photos/mike9alive/
o http://www.flickr.com/photos/tambako/
o http://www.flickr.com/photos/nate
o http://www.flickr.com/photos/freewine/

Cloudware is the arsenal of freedom
• Go guerilla
o Experiment with cloudware
o Consider using cloudware for your new projects
o Test-drive your favourite language on the App Engine SDK
o Await availability for access to the production Java runtime

Contact Information
Kevin Noonan
kevin_noonan on Twitter
http://www.calbane.com