AppScale @ LA.rb


Published on

These are my slides from my talk at LA.rb, covering research at UCSB on the AppScale project. This is a condensed version of the talk I gave at SBonRails - see that talk for about twice as much material on these topics.

Published in: Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

AppScale @ LA.rb

  1. 1. The AppScale Project Presented by Chris Bunch (on behalf of the AppScale team) April 14, 2011 @ LA.rb
  2. 2. Overview• Google App Engine• AppScale - now with 50% Ruby!• Neptune - A Ruby DSL for the cloud
  3. 3. Google App Engine• A web framework introduced in 2008• Python and Java supported• Offers a Platform-as-a-Service: Use Google’s APIs to achieve scale • Upload your app to Google
  4. 4. Quotas
  5. 5. Data Model• Not relational - semi-structured schema• Compared to models in Rails• Exposes a get / put / delete / query interface
  6. 6. Storing Data• Datastore API - Persistent storage• Memcache API - Transient storage • User can set expiration times• Blobstore API - Store large files • need to enable billing to use it
  7. 7. Be Social!• Mail API - Send and receive e-mail• XMPP API - Send and receive IMs• Channel API - Creating persistent connections via XMPP • Use for chat rooms, games, etc.
  8. 8. Background Tasks• Cron API - Access a URL periodically • Descriptive language: “every 5 minutes”, “every 1st Sun of Jan, Mar, Dec”, etc. • Uses a separate cron.yaml file• TaskQueue API - Within your app, fire off tasks to be done later
  9. 9. Dealing with Users• Users API: Uses Google Accounts • Don’t write that ‘forgot password’ page ever again!• Authorization: via app.yaml: • anyone, must login, or admin only
  10. 10. When Services Fail• Originally: failures throw exceptions • Just catch them all!• Capabilities API: Check if a service is available • Datastore, Memcache, and so on
  11. 11. Deploying Your App• Develop locally on SDK • Stub implementations of most APIs• Then deploy to Google
  12. 12. How to Scale• Limitations on the programming model: • No filesystem interaction • 30 second limit per web request • Language libraries must be on whitelist • Sandboxed execution
  13. 13. Enter AppScale• App Engine is easy to use • but we really want to tinker with the internals!• Need an open platform to experiment on • test API implementations • add new APIs
  14. 14. Enter AppScale• Lots of NoSQL DBs out there • Hard to compare DBs• Configuration and deployment can be complex• Need one-button deployment
  15. 15. Deploying Your App• run-instances: Start AppScale• describe-instances:View cloud metadata• upload-app: Deploy an App Engine app• remove-app: Un-deploy an App Engine app• terminate-instances: Stop AppScale
  16. 16. Deployment Models• Cloud deployment: Amazon EC2 or Eucalyptus (the open source implementation of the EC2 APIs) • Just specify how many machines you need• Non-cloud deployment via Xen or KVM
  17. 17. AppController• The brains of the outfit• Runs on every node• Handles configuration and deployment of all services (including other AppControllers)• Written in Ruby
  18. 18. Load balancer• Routes users to their app via nginx• haproxy makes sure app servers are live• Can’t assume the user has DNS: • Thus we wrote the AppLoadBalancer • Rails app that routes users to apps • Performs authentication as well
  19. 19. App Server• We modified the App Engine SDK• Easier for Python (source included)• Harder for Java (had to decompile)• Removed non-scalable API implementations • Goal: Use open source whenever possible
  20. 20. A Common Feature Request
  21. 21. Database Options• Open source / open APIs / proprietary• Master / slave v. peer-to-peer• Differences in query languages• Data model (key/val, semi-structured)• In-memory or persistent• Data consistency model• Interfaces - REST / Thrift / libraries
  22. 22. Neptune• Need a simple way to run compute-intensive jobs• We have the code from the ‘net• We have the resources - the cloud• But the average user does not have the know how• Our solution: create a domain specific language for configuring cloud apps• Based on Ruby
  23. 23. Syntax• It’s as easy as: neptune :type => “mpi”, :code => “MpiNQueens”, :nodes_to_use => 8, :output => “/mpi/output-1.txt”
  24. 24. Neptune Supports:• Message Passing Interface (MPI)• MapReduce• Unified Parallel C (UPC)• X10• Erlang
  25. 25. Extensibility• Experts can add support for other computational jobs• Biochemists can run simulations via DFSP and dwSSA • Embarassingly parallel Monte Carlo simulations
  26. 26. Wrapping It Up• Thanks to the AppScale team, especially: • Co-lead Navraj Chohan and advisor Chandra Krintz• Check us out on the web: • •