SlideShare a Scribd company logo
1 of 49
Download to read offline
The AppScale Project
                                 Presented by Chris Bunch
                              (on behalf of the AppScale team)
                             March 7, 2011 @ sbonrails meetup




Thursday, March 10, 2011
Thursday, March 10, 2011
Overview

                    • Google App Engine
                    • AppScale - now with 50% Ruby!
                    • Research Directions
                    • Neptune - A Ruby DSL for the cloud

Thursday, March 10, 2011
Google App Engine

                    • A web framework introduced in 2008
                    • Python and Java supported
                    • Offers a Platform-as-a-Service: Use
                           Google’s APIs to achieve scale
                           • Upload your app to Google

Thursday, March 10, 2011
Quotas




Thursday, March 10, 2011
Data Model

                    • Not relational - semi-structured schema
                    • Compare to models in Rails
                    • Exposes a get / put / delete / query
                           interface




Thursday, March 10, 2011
Storing Data

                    • Datastore API - Persistent storage
                    • Memcache API - Transient storage
                     • User can set expiration times
                    • Blobstore API - Store large files
                     • need to enable billing to use it

Thursday, March 10, 2011
Be Social!

                    • Mail API - Send and receive e-mail
                    • XMPP API - Send and receive IMs
                    • Channel API - Creating persistent
                           connections via XMPP
                           • Use for chat rooms, games, etc.

Thursday, March 10, 2011
Background Tasks
                    • Cron API - Access a URL periodically
                     • Descriptive language: “every 5 minutes”,
                             “every 1st Sun of Jan, Mar, Dec”, etc.
                           • Uses a separate cron.yaml file
                    • Taskqueue API - Within your app, fire off
                           tasks to be done later


Thursday, March 10, 2011
Dealing with Users

                    • Users API: Uses Google Accounts
                     • Don’t write that ‘forgot password’ page
                           ever again!
                    • Authorization: via app.yaml:
                     • anyone, must login, or admin only

Thursday, March 10, 2011
When Services Fail

                    • Originally: failures throw exceptions
                     • Just catch them all!
                    • Capabilities API: Check if a service is
                           available
                           • Datastore, Memcache, and so on

Thursday, March 10, 2011
Deploying Your App
                    • Develop locally on SDK
                     • Stub implementations of most APIs
                    • Then deploy to Google



Thursday, March 10, 2011
How to Scale

                    • Limitations on the programming model:
                     • No filesystem interaction
                     • 30 second limit per web request
                     • Language libraries must be on whitelist
                     • Sandboxed execution

Thursday, March 10, 2011
Enter AppScale
                    • App Engine is easy to use
                     • but we really want to tinker with the
                           internals!
                    • Need an open platform to experiment on
                     • test API implementations
                     • add new APIs
Thursday, March 10, 2011
Enter AppScale

                    • Lots of NoSQL DBs out there
                     • Hard to compare DBs
                    • Configuration and deployment can be
                           complex
                    • Need one-button deployment

Thursday, March 10, 2011
Storing Data

                    • Datastore API - AppServers use a database
                           agnostic layer - sends requests to PBServer
                           • Named for data format: Protocol Buffers
                    • Memcache API - memcached
                    • Blobstore API - Custom server

Thursday, March 10, 2011
Be Social!

                    • Mail API - sendmail (disabled by default)
                    • XMPP API - ejabberd
                    • Channel API - strophejs


Thursday, March 10, 2011
Background Tasks
                    • Cron API - Uses Vixie Cron
                    • Taskqueue - Separate thread fetches web
                           page
                           • Both make a single attempt
                           • Will replace with distributed, fault-
                             tolerant versions


Thursday, March 10, 2011
Dealing with Users

                    • Users API: Defers users to
                           AppLoadBalancer
                           • Password reset via command-line tools
                    • Authorization: no major changes here


Thursday, March 10, 2011
Deploying Your App

                    • Develop locally on SDK
                     • Stub implementations of most APIs
                    • Then deploy to AppScale!
                     • Use your own cluster or via Amazon
                     • Command-line tools mirror Amazon’s

Thursday, March 10, 2011
Deploying Your App
                           • run-instances: Start AppScale
                           • describe-instances:View cloud metadata
                           • upload-app: Deploy an App Engine app
                           • remove-app: Un-deploy an App Engine
                             app
                           • terminate-instances: Stop AppScale
Thursday, March 10, 2011
Deployment Models

                    • Cloud deployment: Amazon EC2 or
                           Eucalyptus (the open source
                           implementation of the EC2 APIs)
                           • Just specify how many machines you need
                    • Non-cloud deployment via Xen or KVM

Thursday, March 10, 2011
Thursday, March 10, 2011
AppController
                    • The brains of the outfit
                    • Runs on every node
                    • Handles configuration and deployment of
                           all services (including other
                           AppControllers)
                    • Written in Ruby

Thursday, March 10, 2011
Load balancer
                    • Routes users to their app via nginx
                    • haproxy makes sure app servers are live
                    • Can’t assume the user has DNS:
                     • Thus we wrote the AppLoadBalancer
                     • Rails app that routes users to apps
                     • Performs authentication as well
Thursday, March 10, 2011
AppLoadBalancer




Thursday, March 10, 2011
App Server
                    • We modified the App Engine SDK
                    • Easier for Python (source included)
                    • Harder for Java (had to decompile)
                    • Removed non-scalable API implementations
                     • Goal: Use open source whenever
                           possible


Thursday, March 10, 2011
A Common Feature
                               Request




Thursday, March 10, 2011
Database Options
                    •      Open source / open APIs / proprietary
                    •      Master / slave v. peer-to-peer
                    •      Differences in query languages
                    •      Data model (key/val, semi-structured)
                    •      In-memory or persistent
                    •      Data consistency model
                    •      Interfaces - REST / Thrift / libraries


Thursday, March 10, 2011
In AppScale:
                    • BigTable clones:
                    • Master / slave relationship
                     • Master stores metadata
                     • Slaves store data
                    • Fault-tolerant to slave failure
                     • Partially tolerant to master failure
Thursday, March 10, 2011
In AppScale:

                    • Variably consistent DBs
                     • Voldemort and
                    • Both are peer-to-peer: no SPOF
                    • Voldemort: Specify consistency per table
                    • Cassandra: Specify consistency per request

Thursday, March 10, 2011
In AppScale:

                    • Relational:
                     • Not NoSQL but used like NoSQL
                    • Document-oriented:
                     • Targets append-heavy workloads

Thursday, March 10, 2011
In AppScale:
                    • Key-value datastores:
                     • MemcacheDB: like memcached but
                             persistent and replicated
                           • Scalaris: in-memory, no persistence
                           • SimpleDB: semi-structured but used as
                             key-value (will update this in the future)


Thursday, March 10, 2011
Research Ideas
                    •      Placement support
                    •      Monitoring
                    •      Shared memory
                    •      Cost modeling
                    •      Hybrid cloud
                    •      Active Cloud DB
                    •      Disaster Recovery
                    •      Neptune


Thursday, March 10, 2011
Placement Support




Thursday, March 10, 2011
Monitr




Thursday, March 10, 2011
Shared memory
                    • Since AppServer + DB are co-located,
                           reduce message overhead
                           • no serialization
                    • Leverage CoLoRs to do so across
                           languages
                           • AS is in Python or Java, DBS is Python
                    • Can be orders-of-magnitude faster
Thursday, March 10, 2011
Cost modeling
                    • Can we reproduce Google’s cost model?
                     • We can reproduce memory, network
                             bandwidth in / out, size and types of data
                           • Can’t reproduce CPU - it’s based on
                             Google’s load, which we can’t capture
                            • varies based on placement and time of
                               day


Thursday, March 10, 2011
Hybrid Cloud




Thursday, March 10, 2011
Database Agnostic
                             Transactions
                    • Want to support disparate DBs with ACID
                     • Leverage ZooKeeper for versioning
                     • And PBServer as the DB agnostic layer
                     • Needs strong consistency from DB itself
                       • And row-level atomicity on updates

Thursday, March 10, 2011
Active Cloud DB

                    • Need a common interface to DBs
                    • But not just for Java / Python
                    • Named after Rails’ ActiveRecord
                     • Exposes REST interface for DB
                    • Included in AppScale 1.3

Thursday, March 10, 2011
Disaster Recovery
                    • People are using App Engine as a
                           production level environment
                    • Need a way to automatically back up data
                    • Can leverage this data for data analytics
                    • Need to also seamlessly switch to AppScale
                           version if App Engine version goes down


Thursday, March 10, 2011
Neptune
                    •      Need a simple way to run compute-intensive jobs
                    •      We have the code from the ‘net
                    •      We have the resources - the cloud
                    •      But the average user does not have the know how
                    •      Our solution: create a domain specific language
                           for configuring cloud apps
                    •      Based on Ruby



Thursday, March 10, 2011
Syntax

                • It’s as easy as:
                           neptune :type => “mpi”,
                            :code => “MpiNQueens”,
                            :nodes_to_use => 8,
                            :output => “/mpi/output-1.txt”



Thursday, March 10, 2011
Neptune Supports:

                    • Message Passing Interface (MPI)
                    • MapReduce
                    • Unified Parallel C (UPC)
                    • X10
                    • Erlang

Thursday, March 10, 2011
Extensibility

                    • Experts can add support for other
                           computational jobs
                    • Biochemists can run simulations via DFSP
                           and dwSSA
                           • Embarassingly parallel Monte Carlo
                             simulations



Thursday, March 10, 2011
Compiling Code

                • You may not have the binaries, so compile
                           from source!
                • Auto-generates makefiles for beginners
                           neptune :type => “compile”,
                            :code => “/home/appscale/mpi_nqueens”



Thursday, March 10, 2011
Installing Neptune
                    • Just use good old ‘gem’:
                    • gem install neptune
                    • Current version is 0.0.4, fully compatible
                           with AppScale 1.5
                    • More info at our web page:
                     • http://neptune-lang.org
Thursday, March 10, 2011
Wrapping It Up
                    • Thanks to the AppScale team, especially:
                     • Co-lead Navraj Chohan and advisor
                           Professor Chandra Krintz
                    • Check us out on the web:
                     • http://appscale.cs.ucsb.edu
                     • http://code.google.com/p/appscale
Thursday, March 10, 2011

More Related Content

What's hot

Tech Talk on Autoscaling in Apache Stratos
Tech Talk on Autoscaling in Apache StratosTech Talk on Autoscaling in Apache Stratos
Tech Talk on Autoscaling in Apache StratosVishanth Bala
 
StreamSQL Feature Store (Apache Pulsar Summit)
StreamSQL Feature Store (Apache Pulsar Summit)StreamSQL Feature Store (Apache Pulsar Summit)
StreamSQL Feature Store (Apache Pulsar Summit)Simba Khadder
 
(DEV302) Hosting ASP.Net 5 Apps in AWS with Docker & AWS CodeDeploy
(DEV302) Hosting ASP.Net 5 Apps in AWS with Docker & AWS CodeDeploy(DEV302) Hosting ASP.Net 5 Apps in AWS with Docker & AWS CodeDeploy
(DEV302) Hosting ASP.Net 5 Apps in AWS with Docker & AWS CodeDeployAmazon Web Services
 
Five Years of EC2 Distilled
Five Years of EC2 DistilledFive Years of EC2 Distilled
Five Years of EC2 DistilledGrig Gheorghiu
 
Bridging the Gap: Connecting AWS and Kafka
Bridging the Gap: Connecting AWS and KafkaBridging the Gap: Connecting AWS and Kafka
Bridging the Gap: Connecting AWS and KafkaPengfei (Jason) Li
 
Apache Zeppelin & Cluster
Apache Zeppelin & ClusterApache Zeppelin & Cluster
Apache Zeppelin & ClusterJongyoul Lee
 
Heterogeneous Workflows With Spark At Netflix
Heterogeneous Workflows With Spark At NetflixHeterogeneous Workflows With Spark At Netflix
Heterogeneous Workflows With Spark At NetflixJen Aman
 
Kafka for Microservices – You absolutely need Avro Schemas! | Gerardo Gutierr...
Kafka for Microservices – You absolutely need Avro Schemas! | Gerardo Gutierr...Kafka for Microservices – You absolutely need Avro Schemas! | Gerardo Gutierr...
Kafka for Microservices – You absolutely need Avro Schemas! | Gerardo Gutierr...HostedbyConfluent
 
(APP309) Running and Monitoring Docker Containers at Scale | AWS re:Invent 2014
(APP309) Running and Monitoring Docker Containers at Scale | AWS re:Invent 2014(APP309) Running and Monitoring Docker Containers at Scale | AWS re:Invent 2014
(APP309) Running and Monitoring Docker Containers at Scale | AWS re:Invent 2014Amazon Web Services
 
Introduction to Node (15th May 2017)
Introduction to Node (15th May 2017)Introduction to Node (15th May 2017)
Introduction to Node (15th May 2017)Lucas Jellema
 
Data Pipelines with Kafka Connect
Data Pipelines with Kafka ConnectData Pipelines with Kafka Connect
Data Pipelines with Kafka ConnectKaufman Ng
 
Building a PaaS with Docker and AWS
Building a PaaS with Docker and AWSBuilding a PaaS with Docker and AWS
Building a PaaS with Docker and AWSAmazon Web Services
 
Immutable Infrastructure: the new App Deployment
Immutable Infrastructure: the new App DeploymentImmutable Infrastructure: the new App Deployment
Immutable Infrastructure: the new App DeploymentAxel Fontaine
 
Event Driven Architecture with Apache Camel
Event Driven Architecture with Apache CamelEvent Driven Architecture with Apache Camel
Event Driven Architecture with Apache Camelprajods
 
Container orchestration overview
Container orchestration overviewContainer orchestration overview
Container orchestration overviewWyn B. Van Devanter
 

What's hot (20)

Tech Talk on Autoscaling in Apache Stratos
Tech Talk on Autoscaling in Apache StratosTech Talk on Autoscaling in Apache Stratos
Tech Talk on Autoscaling in Apache Stratos
 
DevOpsCon Cloud Workshop
DevOpsCon Cloud Workshop DevOpsCon Cloud Workshop
DevOpsCon Cloud Workshop
 
Sas 2015 event_driven
Sas 2015 event_drivenSas 2015 event_driven
Sas 2015 event_driven
 
Docker in the Cloud
Docker in the CloudDocker in the Cloud
Docker in the Cloud
 
StreamSQL Feature Store (Apache Pulsar Summit)
StreamSQL Feature Store (Apache Pulsar Summit)StreamSQL Feature Store (Apache Pulsar Summit)
StreamSQL Feature Store (Apache Pulsar Summit)
 
eCAP Developer Walkthru
eCAP Developer WalkthrueCAP Developer Walkthru
eCAP Developer Walkthru
 
(DEV302) Hosting ASP.Net 5 Apps in AWS with Docker & AWS CodeDeploy
(DEV302) Hosting ASP.Net 5 Apps in AWS with Docker & AWS CodeDeploy(DEV302) Hosting ASP.Net 5 Apps in AWS with Docker & AWS CodeDeploy
(DEV302) Hosting ASP.Net 5 Apps in AWS with Docker & AWS CodeDeploy
 
Five Years of EC2 Distilled
Five Years of EC2 DistilledFive Years of EC2 Distilled
Five Years of EC2 Distilled
 
Bridging the Gap: Connecting AWS and Kafka
Bridging the Gap: Connecting AWS and KafkaBridging the Gap: Connecting AWS and Kafka
Bridging the Gap: Connecting AWS and Kafka
 
Apache Zeppelin & Cluster
Apache Zeppelin & ClusterApache Zeppelin & Cluster
Apache Zeppelin & Cluster
 
Heterogeneous Workflows With Spark At Netflix
Heterogeneous Workflows With Spark At NetflixHeterogeneous Workflows With Spark At Netflix
Heterogeneous Workflows With Spark At Netflix
 
Kafka for Microservices – You absolutely need Avro Schemas! | Gerardo Gutierr...
Kafka for Microservices – You absolutely need Avro Schemas! | Gerardo Gutierr...Kafka for Microservices – You absolutely need Avro Schemas! | Gerardo Gutierr...
Kafka for Microservices – You absolutely need Avro Schemas! | Gerardo Gutierr...
 
(APP309) Running and Monitoring Docker Containers at Scale | AWS re:Invent 2014
(APP309) Running and Monitoring Docker Containers at Scale | AWS re:Invent 2014(APP309) Running and Monitoring Docker Containers at Scale | AWS re:Invent 2014
(APP309) Running and Monitoring Docker Containers at Scale | AWS re:Invent 2014
 
Introduction to Node (15th May 2017)
Introduction to Node (15th May 2017)Introduction to Node (15th May 2017)
Introduction to Node (15th May 2017)
 
Tech4Africa 2014
Tech4Africa 2014Tech4Africa 2014
Tech4Africa 2014
 
Data Pipelines with Kafka Connect
Data Pipelines with Kafka ConnectData Pipelines with Kafka Connect
Data Pipelines with Kafka Connect
 
Building a PaaS with Docker and AWS
Building a PaaS with Docker and AWSBuilding a PaaS with Docker and AWS
Building a PaaS with Docker and AWS
 
Immutable Infrastructure: the new App Deployment
Immutable Infrastructure: the new App DeploymentImmutable Infrastructure: the new App Deployment
Immutable Infrastructure: the new App Deployment
 
Event Driven Architecture with Apache Camel
Event Driven Architecture with Apache CamelEvent Driven Architecture with Apache Camel
Event Driven Architecture with Apache Camel
 
Container orchestration overview
Container orchestration overviewContainer orchestration overview
Container orchestration overview
 

Similar to AppScale Talk at SBonRails

Red Dirt Ruby Conference
Red Dirt Ruby ConferenceRed Dirt Ruby Conference
Red Dirt Ruby ConferenceJohn Woodell
 
2011 June - Singapore GTUG presentation. App Engine program update + intro to Go
2011 June - Singapore GTUG presentation. App Engine program update + intro to Go2011 June - Singapore GTUG presentation. App Engine program update + intro to Go
2011 June - Singapore GTUG presentation. App Engine program update + intro to Goikailan
 
Aloha on-rails-2009
Aloha on-rails-2009Aloha on-rails-2009
Aloha on-rails-2009John Woodell
 
Webinar Mobile ECM Apps with Nuxeo EP
Webinar Mobile ECM Apps with Nuxeo EPWebinar Mobile ECM Apps with Nuxeo EP
Webinar Mobile ECM Apps with Nuxeo EPNuxeo
 
What is Google App Engine?
What is Google App Engine?What is Google App Engine?
What is Google App Engine?weschwee
 
JavaOne 2011 - Going Mobile With Java Based Technologies Today
JavaOne 2011 - Going Mobile With Java Based Technologies TodayJavaOne 2011 - Going Mobile With Java Based Technologies Today
JavaOne 2011 - Going Mobile With Java Based Technologies TodayWesley Hales
 
Java EE | Apache TomEE - Java EE Web Profile on Tomcat | Jonathan Gallimore
Java EE | Apache TomEE - Java EE Web Profile on Tomcat | Jonathan GallimoreJava EE | Apache TomEE - Java EE Web Profile on Tomcat | Jonathan Gallimore
Java EE | Apache TomEE - Java EE Web Profile on Tomcat | Jonathan GallimoreJAX London
 
Mobile ECM with JavaScript - JSE 2011
Mobile ECM with JavaScript - JSE 2011Mobile ECM with JavaScript - JSE 2011
Mobile ECM with JavaScript - JSE 2011Nuxeo
 
Gaelyk - Guillaume Laforge - GR8Conf Europe 2011
Gaelyk - Guillaume Laforge - GR8Conf Europe 2011Gaelyk - Guillaume Laforge - GR8Conf Europe 2011
Gaelyk - Guillaume Laforge - GR8Conf Europe 2011Guillaume Laforge
 
Javascript Views, Client-side or Server-side with NodeJS
Javascript Views, Client-side or Server-side with NodeJSJavascript Views, Client-side or Server-side with NodeJS
Javascript Views, Client-side or Server-side with NodeJSSylvain Zimmer
 
Velocity2011 chef-workshop
Velocity2011 chef-workshopVelocity2011 chef-workshop
Velocity2011 chef-workshopjtimberman
 
Mobile application development; Complete steps
Mobile application development; Complete stepsMobile application development; Complete steps
Mobile application development; Complete steps Abin Baby
 
Apache Curator: Past, Present and Future
Apache Curator: Past, Present and FutureApache Curator: Past, Present and Future
Apache Curator: Past, Present and FutureJordan Zimmerman
 
Gaelyk - Groovy Grails eXchange 2010 - Guillaume Laforge
Gaelyk - Groovy Grails eXchange 2010 - Guillaume LaforgeGaelyk - Groovy Grails eXchange 2010 - Guillaume Laforge
Gaelyk - Groovy Grails eXchange 2010 - Guillaume LaforgeGuillaume Laforge
 
How to Build Single Page HTML5 Apps that Scale
How to Build Single Page HTML5 Apps that ScaleHow to Build Single Page HTML5 Apps that Scale
How to Build Single Page HTML5 Apps that ScalePhil Leggetter
 
2011 JavaOne Apache TomEE Java EE 6 Web Profile
2011 JavaOne Apache TomEE Java EE 6 Web Profile2011 JavaOne Apache TomEE Java EE 6 Web Profile
2011 JavaOne Apache TomEE Java EE 6 Web ProfileDavid Blevins
 
The Fast, The Slow and the Lazy
The Fast, The Slow and the LazyThe Fast, The Slow and the Lazy
The Fast, The Slow and the LazyMaurício Linhares
 
20110903 candycane
20110903 candycane20110903 candycane
20110903 candycaneYusuke Ando
 

Similar to AppScale Talk at SBonRails (20)

App Engine Meetup
App Engine MeetupApp Engine Meetup
App Engine Meetup
 
Red Dirt Ruby Conference
Red Dirt Ruby ConferenceRed Dirt Ruby Conference
Red Dirt Ruby Conference
 
2011 June - Singapore GTUG presentation. App Engine program update + intro to Go
2011 June - Singapore GTUG presentation. App Engine program update + intro to Go2011 June - Singapore GTUG presentation. App Engine program update + intro to Go
2011 June - Singapore GTUG presentation. App Engine program update + intro to Go
 
Aloha on-rails-2009
Aloha on-rails-2009Aloha on-rails-2009
Aloha on-rails-2009
 
Webinar Mobile ECM Apps with Nuxeo EP
Webinar Mobile ECM Apps with Nuxeo EPWebinar Mobile ECM Apps with Nuxeo EP
Webinar Mobile ECM Apps with Nuxeo EP
 
What is Google App Engine?
What is Google App Engine?What is Google App Engine?
What is Google App Engine?
 
Groke
GrokeGroke
Groke
 
JavaOne 2011 - Going Mobile With Java Based Technologies Today
JavaOne 2011 - Going Mobile With Java Based Technologies TodayJavaOne 2011 - Going Mobile With Java Based Technologies Today
JavaOne 2011 - Going Mobile With Java Based Technologies Today
 
Java EE | Apache TomEE - Java EE Web Profile on Tomcat | Jonathan Gallimore
Java EE | Apache TomEE - Java EE Web Profile on Tomcat | Jonathan GallimoreJava EE | Apache TomEE - Java EE Web Profile on Tomcat | Jonathan Gallimore
Java EE | Apache TomEE - Java EE Web Profile on Tomcat | Jonathan Gallimore
 
Mobile ECM with JavaScript - JSE 2011
Mobile ECM with JavaScript - JSE 2011Mobile ECM with JavaScript - JSE 2011
Mobile ECM with JavaScript - JSE 2011
 
Gaelyk - Guillaume Laforge - GR8Conf Europe 2011
Gaelyk - Guillaume Laforge - GR8Conf Europe 2011Gaelyk - Guillaume Laforge - GR8Conf Europe 2011
Gaelyk - Guillaume Laforge - GR8Conf Europe 2011
 
Javascript Views, Client-side or Server-side with NodeJS
Javascript Views, Client-side or Server-side with NodeJSJavascript Views, Client-side or Server-side with NodeJS
Javascript Views, Client-side or Server-side with NodeJS
 
Velocity2011 chef-workshop
Velocity2011 chef-workshopVelocity2011 chef-workshop
Velocity2011 chef-workshop
 
Mobile application development; Complete steps
Mobile application development; Complete stepsMobile application development; Complete steps
Mobile application development; Complete steps
 
Apache Curator: Past, Present and Future
Apache Curator: Past, Present and FutureApache Curator: Past, Present and Future
Apache Curator: Past, Present and Future
 
Gaelyk - Groovy Grails eXchange 2010 - Guillaume Laforge
Gaelyk - Groovy Grails eXchange 2010 - Guillaume LaforgeGaelyk - Groovy Grails eXchange 2010 - Guillaume Laforge
Gaelyk - Groovy Grails eXchange 2010 - Guillaume Laforge
 
How to Build Single Page HTML5 Apps that Scale
How to Build Single Page HTML5 Apps that ScaleHow to Build Single Page HTML5 Apps that Scale
How to Build Single Page HTML5 Apps that Scale
 
2011 JavaOne Apache TomEE Java EE 6 Web Profile
2011 JavaOne Apache TomEE Java EE 6 Web Profile2011 JavaOne Apache TomEE Java EE 6 Web Profile
2011 JavaOne Apache TomEE Java EE 6 Web Profile
 
The Fast, The Slow and the Lazy
The Fast, The Slow and the LazyThe Fast, The Slow and the Lazy
The Fast, The Slow and the Lazy
 
20110903 candycane
20110903 candycane20110903 candycane
20110903 candycane
 

Recently uploaded

Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxUdaiappa Ramachandran
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarPrecisely
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDELiveplex
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfJamie (Taka) Wang
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Brian Pichman
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 

Recently uploaded (20)

Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptx
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
201610817 - edge part1
201610817 - edge part1201610817 - edge part1
201610817 - edge part1
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity Webinar
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )
 
20230104 - machine vision
20230104 - machine vision20230104 - machine vision
20230104 - machine vision
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 

AppScale Talk at SBonRails

  • 1. The AppScale Project Presented by Chris Bunch (on behalf of the AppScale team) March 7, 2011 @ sbonrails meetup Thursday, March 10, 2011
  • 3. Overview • Google App Engine • AppScale - now with 50% Ruby! • Research Directions • Neptune - A Ruby DSL for the cloud Thursday, March 10, 2011
  • 4. Google App Engine • A web framework introduced in 2008 • Python and Java supported • Offers a Platform-as-a-Service: Use Google’s APIs to achieve scale • Upload your app to Google Thursday, March 10, 2011
  • 6. Data Model • Not relational - semi-structured schema • Compare to models in Rails • Exposes a get / put / delete / query interface Thursday, March 10, 2011
  • 7. Storing Data • Datastore API - Persistent storage • Memcache API - Transient storage • User can set expiration times • Blobstore API - Store large files • need to enable billing to use it Thursday, March 10, 2011
  • 8. Be Social! • Mail API - Send and receive e-mail • XMPP API - Send and receive IMs • Channel API - Creating persistent connections via XMPP • Use for chat rooms, games, etc. Thursday, March 10, 2011
  • 9. Background Tasks • Cron API - Access a URL periodically • Descriptive language: “every 5 minutes”, “every 1st Sun of Jan, Mar, Dec”, etc. • Uses a separate cron.yaml file • Taskqueue API - Within your app, fire off tasks to be done later Thursday, March 10, 2011
  • 10. Dealing with Users • Users API: Uses Google Accounts • Don’t write that ‘forgot password’ page ever again! • Authorization: via app.yaml: • anyone, must login, or admin only Thursday, March 10, 2011
  • 11. When Services Fail • Originally: failures throw exceptions • Just catch them all! • Capabilities API: Check if a service is available • Datastore, Memcache, and so on Thursday, March 10, 2011
  • 12. Deploying Your App • Develop locally on SDK • Stub implementations of most APIs • Then deploy to Google Thursday, March 10, 2011
  • 13. How to Scale • Limitations on the programming model: • No filesystem interaction • 30 second limit per web request • Language libraries must be on whitelist • Sandboxed execution Thursday, March 10, 2011
  • 14. Enter AppScale • App Engine is easy to use • but we really want to tinker with the internals! • Need an open platform to experiment on • test API implementations • add new APIs Thursday, March 10, 2011
  • 15. Enter AppScale • Lots of NoSQL DBs out there • Hard to compare DBs • Configuration and deployment can be complex • Need one-button deployment Thursday, March 10, 2011
  • 16. Storing Data • Datastore API - AppServers use a database agnostic layer - sends requests to PBServer • Named for data format: Protocol Buffers • Memcache API - memcached • Blobstore API - Custom server Thursday, March 10, 2011
  • 17. Be Social! • Mail API - sendmail (disabled by default) • XMPP API - ejabberd • Channel API - strophejs Thursday, March 10, 2011
  • 18. Background Tasks • Cron API - Uses Vixie Cron • Taskqueue - Separate thread fetches web page • Both make a single attempt • Will replace with distributed, fault- tolerant versions Thursday, March 10, 2011
  • 19. Dealing with Users • Users API: Defers users to AppLoadBalancer • Password reset via command-line tools • Authorization: no major changes here Thursday, March 10, 2011
  • 20. Deploying Your App • Develop locally on SDK • Stub implementations of most APIs • Then deploy to AppScale! • Use your own cluster or via Amazon • Command-line tools mirror Amazon’s Thursday, March 10, 2011
  • 21. Deploying Your App • run-instances: Start AppScale • describe-instances:View cloud metadata • upload-app: Deploy an App Engine app • remove-app: Un-deploy an App Engine app • terminate-instances: Stop AppScale Thursday, March 10, 2011
  • 22. Deployment Models • Cloud deployment: Amazon EC2 or Eucalyptus (the open source implementation of the EC2 APIs) • Just specify how many machines you need • Non-cloud deployment via Xen or KVM Thursday, March 10, 2011
  • 24. AppController • The brains of the outfit • Runs on every node • Handles configuration and deployment of all services (including other AppControllers) • Written in Ruby Thursday, March 10, 2011
  • 25. Load balancer • Routes users to their app via nginx • haproxy makes sure app servers are live • Can’t assume the user has DNS: • Thus we wrote the AppLoadBalancer • Rails app that routes users to apps • Performs authentication as well Thursday, March 10, 2011
  • 27. App Server • We modified the App Engine SDK • Easier for Python (source included) • Harder for Java (had to decompile) • Removed non-scalable API implementations • Goal: Use open source whenever possible Thursday, March 10, 2011
  • 28. A Common Feature Request Thursday, March 10, 2011
  • 29. Database Options • Open source / open APIs / proprietary • Master / slave v. peer-to-peer • Differences in query languages • Data model (key/val, semi-structured) • In-memory or persistent • Data consistency model • Interfaces - REST / Thrift / libraries Thursday, March 10, 2011
  • 30. In AppScale: • BigTable clones: • Master / slave relationship • Master stores metadata • Slaves store data • Fault-tolerant to slave failure • Partially tolerant to master failure Thursday, March 10, 2011
  • 31. In AppScale: • Variably consistent DBs • Voldemort and • Both are peer-to-peer: no SPOF • Voldemort: Specify consistency per table • Cassandra: Specify consistency per request Thursday, March 10, 2011
  • 32. In AppScale: • Relational: • Not NoSQL but used like NoSQL • Document-oriented: • Targets append-heavy workloads Thursday, March 10, 2011
  • 33. In AppScale: • Key-value datastores: • MemcacheDB: like memcached but persistent and replicated • Scalaris: in-memory, no persistence • SimpleDB: semi-structured but used as key-value (will update this in the future) Thursday, March 10, 2011
  • 34. Research Ideas • Placement support • Monitoring • Shared memory • Cost modeling • Hybrid cloud • Active Cloud DB • Disaster Recovery • Neptune Thursday, March 10, 2011
  • 37. Shared memory • Since AppServer + DB are co-located, reduce message overhead • no serialization • Leverage CoLoRs to do so across languages • AS is in Python or Java, DBS is Python • Can be orders-of-magnitude faster Thursday, March 10, 2011
  • 38. Cost modeling • Can we reproduce Google’s cost model? • We can reproduce memory, network bandwidth in / out, size and types of data • Can’t reproduce CPU - it’s based on Google’s load, which we can’t capture • varies based on placement and time of day Thursday, March 10, 2011
  • 40. Database Agnostic Transactions • Want to support disparate DBs with ACID • Leverage ZooKeeper for versioning • And PBServer as the DB agnostic layer • Needs strong consistency from DB itself • And row-level atomicity on updates Thursday, March 10, 2011
  • 41. Active Cloud DB • Need a common interface to DBs • But not just for Java / Python • Named after Rails’ ActiveRecord • Exposes REST interface for DB • Included in AppScale 1.3 Thursday, March 10, 2011
  • 42. Disaster Recovery • People are using App Engine as a production level environment • Need a way to automatically back up data • Can leverage this data for data analytics • Need to also seamlessly switch to AppScale version if App Engine version goes down Thursday, March 10, 2011
  • 43. Neptune • Need a simple way to run compute-intensive jobs • We have the code from the ‘net • We have the resources - the cloud • But the average user does not have the know how • Our solution: create a domain specific language for configuring cloud apps • Based on Ruby Thursday, March 10, 2011
  • 44. Syntax • It’s as easy as: neptune :type => “mpi”, :code => “MpiNQueens”, :nodes_to_use => 8, :output => “/mpi/output-1.txt” Thursday, March 10, 2011
  • 45. Neptune Supports: • Message Passing Interface (MPI) • MapReduce • Unified Parallel C (UPC) • X10 • Erlang Thursday, March 10, 2011
  • 46. Extensibility • Experts can add support for other computational jobs • Biochemists can run simulations via DFSP and dwSSA • Embarassingly parallel Monte Carlo simulations Thursday, March 10, 2011
  • 47. Compiling Code • You may not have the binaries, so compile from source! • Auto-generates makefiles for beginners neptune :type => “compile”, :code => “/home/appscale/mpi_nqueens” Thursday, March 10, 2011
  • 48. Installing Neptune • Just use good old ‘gem’: • gem install neptune • Current version is 0.0.4, fully compatible with AppScale 1.5 • More info at our web page: • http://neptune-lang.org Thursday, March 10, 2011
  • 49. Wrapping It Up • Thanks to the AppScale team, especially: • Co-lead Navraj Chohan and advisor Professor Chandra Krintz • Check us out on the web: • http://appscale.cs.ucsb.edu • http://code.google.com/p/appscale Thursday, March 10, 2011