These are my slides from my talk at LA.rb, covering research at UCSB on the AppScale project. This is a condensed version of the talk I gave at SBonRails - see that talk for about twice as much material on these topics.

1. The AppScale Project
Presented by Chris Bunch
(on behalf of the AppScale team)
April 14, 2011 @ LA.rb

3. Overview
• Google App Engine
• AppScale - now with 50% Ruby!
• Neptune - A Ruby DSL for the cloud

4. Google App Engine
• A web framework introduced in 2008
• Python and Java supported
• Offers a Platform-as-a-Service: Use
Google’s APIs to achieve scale
• Upload your app to Google

5. Quotas

6. Data Model
• Not relational - semi-structured schema
• Compared to models in Rails
• Exposes a get / put / delete / query
interface

7. Storing Data
• Datastore API - Persistent storage
• Memcache API - Transient storage
• User can set expiration times
• Blobstore API - Store large files
• need to enable billing to use it

8. Be Social!
• Mail API - Send and receive e-mail
• XMPP API - Send and receive IMs
• Channel API - Creating persistent
connections via XMPP
• Use for chat rooms, games, etc.

9. Background Tasks
• Cron API - Access a URL periodically
• Descriptive language: “every 5 minutes”,
“every 1st Sun of Jan, Mar, Dec”, etc.
• Uses a separate cron.yaml file
• TaskQueue API - Within your app, fire off
tasks to be done later

10. Dealing with Users
• Users API: Uses Google Accounts
• Don’t write that ‘forgot password’ page
ever again!
• Authorization: via app.yaml:
• anyone, must login, or admin only

11. When Services Fail
• Originally: failures throw exceptions
• Just catch them all!
• Capabilities API: Check if a service is
available
• Datastore, Memcache, and so on

12. Deploying Your App
• Develop locally on SDK
• Stub implementations of most APIs
• Then deploy to Google

13. How to Scale
• Limitations on the programming model:
• No filesystem interaction
• 30 second limit per web request
• Language libraries must be on whitelist
• Sandboxed execution

14. Enter AppScale
• App Engine is easy to use
• but we really want to tinker with the
internals!
• Need an open platform to experiment on
• test API implementations
• add new APIs

15. Enter AppScale
• Lots of NoSQL DBs out there
• Hard to compare DBs
• Configuration and deployment can be
complex
• Need one-button deployment

16. Deploying Your App
• run-instances: Start AppScale
• describe-instances:View cloud metadata
• upload-app: Deploy an App Engine app
• remove-app: Un-deploy an App Engine app
• terminate-instances: Stop AppScale

17. Deployment Models
• Cloud deployment: Amazon EC2 or
Eucalyptus (the open source
implementation of the EC2 APIs)
• Just specify how many machines you need
• Non-cloud deployment via Xen or KVM

19. AppController
• The brains of the outfit
• Runs on every node
• Handles configuration and deployment of
all services (including other
AppControllers)
• Written in Ruby

20. Load balancer
• Routes users to their app via nginx
• haproxy makes sure app servers are live
• Can’t assume the user has DNS:
• Thus we wrote the AppLoadBalancer
• Rails app that routes users to apps
• Performs authentication as well

22. App Server
• We modified the App Engine SDK
• Easier for Python (source included)
• Harder for Java (had to decompile)
• Removed non-scalable API implementations
• Goal: Use open source whenever
possible

23. A Common Feature
Request

24. Database Options
• Open source / open APIs / proprietary
• Master / slave v. peer-to-peer
• Differences in query languages
• Data model (key/val, semi-structured)
• In-memory or persistent
• Data consistency model
• Interfaces - REST / Thrift / libraries

25. Neptune
• Need a simple way to run compute-intensive jobs
• We have the code from the ‘net
• We have the resources - the cloud
• But the average user does not have the know how
• Our solution: create a domain specific language
for configuring cloud apps
• Based on Ruby

26. Syntax
• It’s as easy as:
neptune :type => “mpi”,
:code => “MpiNQueens”,
:nodes_to_use => 8,
:output => “/mpi/output-1.txt”

27. Neptune Supports:
• Message Passing Interface (MPI)
• MapReduce
• Unified Parallel C (UPC)
• X10
• Erlang

28. Extensibility
• Experts can add support for other
computational jobs
• Biochemists can run simulations via DFSP
and dwSSA
• Embarassingly parallel Monte Carlo
simulations

29. Wrapping It Up
• Thanks to the AppScale team, especially:
• Co-lead Navraj Chohan and advisor
Chandra Krintz
• Check us out on the web:
• http://appscale.cs.ucsb.edu
• http://neptune-lang.org