What is App Engine?

Ikai Lan
OSCON
July 21st, 2010
Twitter: @ikai
Agenda
Topics we'll cover
 Why App Engine?
    Traditional web stack scalability
    App Engine to the rescue!
 Services and APIs
 Deployment steps
 App Engine and Open Source
 Questions and Next Steps
What is Google App Engine?
Google App Engine is...


                ... a way for you to run
                your web applications
                 on Google’s scalable
                      infrastructure.



                             Google’s Data Centers
Start with... the basic LAMP stack


LAMP:
   Linux
   Apache
   MySql
   Programming Language
        (PHP, Python, Perl, etc.)




NOT Scalable

Single Point of Failure (SPOF)


                                     Google Confidential and Proprietary
Database on a separate server



Still not Scalable!


TWO Single Points of Failure




                                Google Confidential and Proprietary
Multiple Web Servers




              Now you need Load Balancing

          Database is still Single Point of Failure

                                               Google Confidential and Proprietary
Round Robin Load Balancing




              Register list of IPs with DNS

     DNS record is cached with a Time to Live (TTL)

                                              Google Confidential and Proprietary
Round Robin Load Balancing




But the TTL takes time to propagate and might not be respected

So if a machine goes down... :-(

And the database is still SPOF
                                                     Google Confidential and Proprietary
Master Slave Database




:-) Better read throughput   :-( Master is SPOF for writes

                             :-( Master may die before replication


                                                   Google Confidential and Proprietary
Partitioned Database




:-) Better R/W throughput   :-( More machines, more management

                            :-( Re-architect data model

                            :-( Rewrite queries
                                                    Google Confidential and Proprietary
Why build it all yourself?




                             Google Confidential and Proprietary
Why not use Google App Engine?




            Simple application configuration

              No systems administration

                No performance tuning

               AUTOMATIC SCALING!
                                               Google Confidential and Proprietary
The Cloud Computing Landscape



                           SaaS

                           PaaS


                             IaaS


Source: Gartner AADI Summit Dec 2009
App Engine Developers/Apps




                             Google Confidential and Proprietary
Google Confidential and Proprietary
By the numbers

 Over 100,000 applications
 250,000 developers
 Over 250 million daily pageviews
Underneath the hood
App Engine Components



                                  Load balancing

                                  Routing




Hosts static content

Separate from programming files




                                              Google Confidential and Proprietary
App Engine Components

                        Hosts application code

                        Handles concurrent requests

                        Enforces isolation for app safety

                        Maintains statelessness




Multiple Runtimes


                                       Copyright © Sun Microsystems Inc. All rights reserved.

                                                  Google Confidential and Proprietary
App Engine Services/APIs
Bigtable - The App Engine datastore



                           Distributed, partitioned datastore




Arbitrary horizontal scaling - scales to “Internet scale”

Replicated and fault tolerant

Parallel processing

Predictable query performance

No deadlocks
                                                      Google Confidential and Proprietary
Memcache




                              Distributed, very fast,
                                in-memory cache




Optimistic caching

Very stable, robust and specialized



                                                    Google Confidential and Proprietary
URL Fetch




                          Simple, HTTP communication




HTTP GET/POST to external service

Allows integration with third-party REST APIs



                                                Google Confidential and Proprietary
Mail




                        Inbound and outbound mail




Outbound mail

Inbound mail handling

Attachment processing

                                              Google Confidential and Proprietary
XMPP




                       Instant messaging for your application




Incoming and outgoing XMPP

No need to worry about setting up servers



                                                    Google Confidential and Proprietary
Task Queue




                           Background and scheduled
                                 computation




Background processing infrastructure

Scheduled jobs

Automatic handling of queuing and job polling

                                                Google Confidential and Proprietary
Images




                     Image manipulation




Resize

Crop

Image compositions

                                          Google Confidential and Proprietary
Blobstore




                             Heavy lifting for large files




Upload and distribute large files

Programmatic access to file contents



                                                       Google Confidential and Proprietary
User Accounts




                      Federated login for your application




Google Accounts or OpenID

Administrator management

No need to create user management system

                                                  Google Confidential and Proprietary
Getting started
Getting started with App Engine

 Download the SDK
    http://code.google.com/appengine
 Register for an Appspot account
    https://appengine.google.com
 Write code - deploy!
Starting a project
  Linux, MacOS, etc. command-line:
   $ dev_appserver.py helloworld # run dev svr
   $ appcfg.py update helloworld # deploy live
 Windows GUI (also avail for Mac):
Project contents




       app.yaml – main configuration file

       index.yaml – automatically
       generated to index your data

       main.py – your main application
       "controller" code goes here
main.py
Local development server
  $ dev_appserver.py helloworld




        (Can also use the launcher for Windows and OS X)
Deploying the application
 Set application identifier
 Run deploy script
 You're live!
main.py: Adding a handler
from google.appengine.ext import webapp
from google.appengine.ext.webapp.util import run_wsgi_app

class MainHandler(webapp.RequestHandler):
  def get(self):
    self.response.out.write('<h1>Hello world!</h1>')
    self.response.out.write('''
         <form action="/sign" method=post>
         <input type=text name=content>
         <br><input type=submit value="Sign Guestbook">
         </form>
    ''')

class GuestBook(webapp.RequestHandler):
  def post(self):
    self.response.out.write(
       '<h2>You wrote:</h2> %s' % self.request.get('content')
    )

application = webapp.WSGIApplication([
   ('/', MainHandler),
   ('/sign', GuestBook),
], debug=True)

# start_wsgi_app etc ...
main.py: Persisting to the datastore

class GuestBook(webapp.RequestHandler):
  def post(self):
    greeting = Greeting()
    greeting.content = self.request.get('content')
    greeting.put()
    self.redirect('/')
main.py: Collecting values from the datastore

class MainHandler(webapp.RequestHandler):
  def get(self):
    self.response.out.write('Hello world!')
    self.response.out.write('<h1>My GuestBook</h1><ol>')
    greetings = Greeting.all()
    for greeting in greetings:
         self.response.out.write('<li> %s' % greeting.content)
    self.response.out.write('''
         </ol><hr>
         <form action="/sign" method=post>
         <textarea name=content rows=3 cols=60></textarea>
         <br><input type=submit value="Sign Guestbook">
         </form>
    ''')
Running the deploy script


$ appcfg.py update helloworld
Scanning files on local disk.
Initiating update.
Email: ...
You're live!
Google App Engine and Open Source
TyphoonAE: Open Source implementation
Open Source JVM language runtimes
 Clojure
 Practical Lisp on the JVM
 JRuby/Mirah
 Ruby on the JVM. Mirah = Ruby-like language that
 compiles to Java (speed!)
 Groovy
 Dynamically typed Java-like language
 Scala
 Strongly typed, functional imperative
 Jython
 Java implementation of Python
Open Source Persistence Frameworks
 Objectify
 http://code.google.com/p/objectify-appengine/
 Twig
 http://code.google.com/p/twig-persist/
 Slim3 (popular in Japan)
 http://sites.google.com/site/slim3appengine/
 SimpleDS
 http://code.google.com/p/simpleds/
 Django-nonrel
 http://www.allbuttonspressed.com/projects/django-
 nonrel
Official features
 Datanucleus JDO/JPA
 http://code.google.com/p/datanucleus-appengine/
 App Engine Map/Reduce
 http://code.google.com/p/appengine-mapreduce/
 Python SDK
 Java SDK - soon!
Questions and Next Steps
Code time!
main.py: Skeleton application

from google.appengine.ext import webapp
from google.appengine.ext.webapp import util


class MainHandler(webapp.RequestHandler):
 def get(self):
  self.response.out.write('Hello world!')


def main():
 application = webapp.WSGIApplication([('/', MainHandler)],
                     debug=True)
 util.run_wsgi_app(application)


if __name__ == '__main__':
  main()
Next steps

 Download the SDK
    http://code.google.com/appengine
 Do the codelab
    http://code.google.com/appengine
 Register for an Appspot account
    https://appengine.google.com
 Get this presentation
    http://slideshare.net/ikailan

What is App Engine? O

  • 1.
    What is AppEngine? Ikai Lan OSCON July 21st, 2010 Twitter: @ikai
  • 2.
    Agenda Topics we'll cover Why App Engine? Traditional web stack scalability App Engine to the rescue! Services and APIs Deployment steps App Engine and Open Source Questions and Next Steps
  • 3.
    What is GoogleApp Engine?
  • 4.
    Google App Engineis... ... a way for you to run your web applications on Google’s scalable infrastructure. Google’s Data Centers
  • 5.
    Start with... thebasic LAMP stack LAMP: Linux Apache MySql Programming Language (PHP, Python, Perl, etc.) NOT Scalable Single Point of Failure (SPOF) Google Confidential and Proprietary
  • 6.
    Database on aseparate server Still not Scalable! TWO Single Points of Failure Google Confidential and Proprietary
  • 7.
    Multiple Web Servers Now you need Load Balancing Database is still Single Point of Failure Google Confidential and Proprietary
  • 8.
    Round Robin LoadBalancing Register list of IPs with DNS DNS record is cached with a Time to Live (TTL) Google Confidential and Proprietary
  • 9.
    Round Robin LoadBalancing But the TTL takes time to propagate and might not be respected So if a machine goes down... :-( And the database is still SPOF Google Confidential and Proprietary
  • 10.
    Master Slave Database :-)Better read throughput :-( Master is SPOF for writes :-( Master may die before replication Google Confidential and Proprietary
  • 11.
    Partitioned Database :-) BetterR/W throughput :-( More machines, more management :-( Re-architect data model :-( Rewrite queries Google Confidential and Proprietary
  • 12.
    Why build itall yourself? Google Confidential and Proprietary
  • 13.
    Why not useGoogle App Engine? Simple application configuration No systems administration No performance tuning AUTOMATIC SCALING! Google Confidential and Proprietary
  • 14.
    The Cloud ComputingLandscape SaaS PaaS IaaS Source: Gartner AADI Summit Dec 2009
  • 15.
    App Engine Developers/Apps Google Confidential and Proprietary
  • 16.
  • 17.
    By the numbers Over 100,000 applications 250,000 developers Over 250 million daily pageviews
  • 18.
  • 19.
    App Engine Components Load balancing Routing Hosts static content Separate from programming files Google Confidential and Proprietary
  • 20.
    App Engine Components Hosts application code Handles concurrent requests Enforces isolation for app safety Maintains statelessness Multiple Runtimes Copyright © Sun Microsystems Inc. All rights reserved. Google Confidential and Proprietary
  • 21.
  • 22.
    Bigtable - TheApp Engine datastore Distributed, partitioned datastore Arbitrary horizontal scaling - scales to “Internet scale” Replicated and fault tolerant Parallel processing Predictable query performance No deadlocks Google Confidential and Proprietary
  • 23.
    Memcache Distributed, very fast, in-memory cache Optimistic caching Very stable, robust and specialized Google Confidential and Proprietary
  • 24.
    URL Fetch Simple, HTTP communication HTTP GET/POST to external service Allows integration with third-party REST APIs Google Confidential and Proprietary
  • 25.
    Mail Inbound and outbound mail Outbound mail Inbound mail handling Attachment processing Google Confidential and Proprietary
  • 26.
    XMPP Instant messaging for your application Incoming and outgoing XMPP No need to worry about setting up servers Google Confidential and Proprietary
  • 27.
    Task Queue Background and scheduled computation Background processing infrastructure Scheduled jobs Automatic handling of queuing and job polling Google Confidential and Proprietary
  • 28.
    Images Image manipulation Resize Crop Image compositions Google Confidential and Proprietary
  • 29.
    Blobstore Heavy lifting for large files Upload and distribute large files Programmatic access to file contents Google Confidential and Proprietary
  • 30.
    User Accounts Federated login for your application Google Accounts or OpenID Administrator management No need to create user management system Google Confidential and Proprietary
  • 31.
  • 32.
    Getting started withApp Engine Download the SDK http://code.google.com/appengine Register for an Appspot account https://appengine.google.com Write code - deploy!
  • 33.
    Starting a project Linux, MacOS, etc. command-line: $ dev_appserver.py helloworld # run dev svr $ appcfg.py update helloworld # deploy live Windows GUI (also avail for Mac):
  • 34.
    Project contents app.yaml – main configuration file index.yaml – automatically generated to index your data main.py – your main application "controller" code goes here
  • 35.
  • 36.
    Local development server $ dev_appserver.py helloworld (Can also use the launcher for Windows and OS X)
  • 37.
    Deploying the application Set application identifier Run deploy script You're live!
  • 38.
    main.py: Adding ahandler from google.appengine.ext import webapp from google.appengine.ext.webapp.util import run_wsgi_app class MainHandler(webapp.RequestHandler): def get(self): self.response.out.write('<h1>Hello world!</h1>') self.response.out.write(''' <form action="/sign" method=post> <input type=text name=content> <br><input type=submit value="Sign Guestbook"> </form> ''') class GuestBook(webapp.RequestHandler): def post(self): self.response.out.write( '<h2>You wrote:</h2> %s' % self.request.get('content') ) application = webapp.WSGIApplication([ ('/', MainHandler), ('/sign', GuestBook), ], debug=True) # start_wsgi_app etc ...
  • 39.
    main.py: Persisting tothe datastore class GuestBook(webapp.RequestHandler): def post(self): greeting = Greeting() greeting.content = self.request.get('content') greeting.put() self.redirect('/')
  • 40.
    main.py: Collecting valuesfrom the datastore class MainHandler(webapp.RequestHandler): def get(self): self.response.out.write('Hello world!') self.response.out.write('<h1>My GuestBook</h1><ol>') greetings = Greeting.all() for greeting in greetings: self.response.out.write('<li> %s' % greeting.content) self.response.out.write(''' </ol><hr> <form action="/sign" method=post> <textarea name=content rows=3 cols=60></textarea> <br><input type=submit value="Sign Guestbook"> </form> ''')
  • 41.
    Running the deployscript $ appcfg.py update helloworld Scanning files on local disk. Initiating update. Email: ...
  • 42.
  • 43.
    Google App Engineand Open Source
  • 44.
    TyphoonAE: Open Sourceimplementation
  • 45.
    Open Source JVMlanguage runtimes Clojure Practical Lisp on the JVM JRuby/Mirah Ruby on the JVM. Mirah = Ruby-like language that compiles to Java (speed!) Groovy Dynamically typed Java-like language Scala Strongly typed, functional imperative Jython Java implementation of Python
  • 46.
    Open Source PersistenceFrameworks Objectify http://code.google.com/p/objectify-appengine/ Twig http://code.google.com/p/twig-persist/ Slim3 (popular in Japan) http://sites.google.com/site/slim3appengine/ SimpleDS http://code.google.com/p/simpleds/ Django-nonrel http://www.allbuttonspressed.com/projects/django- nonrel
  • 47.
    Official features DatanucleusJDO/JPA http://code.google.com/p/datanucleus-appengine/ App Engine Map/Reduce http://code.google.com/p/appengine-mapreduce/ Python SDK Java SDK - soon!
  • 48.
  • 49.
  • 50.
    main.py: Skeleton application fromgoogle.appengine.ext import webapp from google.appengine.ext.webapp import util class MainHandler(webapp.RequestHandler): def get(self): self.response.out.write('Hello world!') def main(): application = webapp.WSGIApplication([('/', MainHandler)], debug=True) util.run_wsgi_app(application) if __name__ == '__main__': main()
  • 51.
    Next steps Downloadthe SDK http://code.google.com/appengine Do the codelab http://code.google.com/appengine Register for an Appspot account https://appengine.google.com Get this presentation http://slideshare.net/ikailan