Your SlideShare is downloading. ×
Develop and deploy scalable apps with                         Google App Engine                         David Chandler    ...
2Monday, March 14, 2011       2
App Engine is part of Google’s cloud platform                                      SaaS                                   ...
Googles Cloud Offerings   Your Apps                                            Google Apps                                ...
Easy to build                              Easy to manage                              Easy to scale          Google App E...
Hosting means hidden costs        Idle capacity        Software patches & upgrades        Maintenance        Traffic & uti...
Cloud development in a box        Hardware        Networking        Operating system        Application runtime        • J...
What is Google App Engine?        Business: Google’s cloud computing platform        Technical: giant Web farm in front of...
Let’s do it        Sign up for AppEngine beta        Download Google plugin for Eclipse        Create new Google Web appli...
What didn’t we do?        Install Tomcat        Configure Tomcat        Start Tomcat        Stop Tomcat        Install a d...
What didn’t we do?        But what would we do differently to SCALE        • our Web tier?        • our data tier?        ...
App Engine scaling checklist                                      Google Confidential and Proprietary   12Monday, March 14...
Google AppEngine at a glance                                      Google Confidential and Proprietary   13Monday, March 14...
What makes AppEngine special?                                                Google Confidential and Proprietary   14Monda...
Scalable serving infrastructure :-)                                             Google Confidential and Proprietary   15Mo...
BigTable: a Distributed Storage System       Fay Chang, Jeffrey Dean, Sanjay Ghemawat, Wilson C. Hsieh, Deborah A. Wallach...
No servers to configure                                 Google Confidential and Proprietary   17Monday, March 14, 2011    ...
Easy deployment                         OR                              Google Confidential and Proprietary   18Monday, Ma...
Powerful admin console                                Google Confidential and Proprietary   19Monday, March 14, 2011      ...
Distributed frontends             Edge Caching             Priority routing             Geographically close connection   ...
Free to get started        6.5 CPU hrs/day        1 GB storage        650K URL Fetch calls        2,000 recipients emailed...
Who’s using App Engine?                                 Google Confidential and Proprietary   22Monday, March 14, 2011    ...
By the numbers          >100K Developers / Month                          >150K Apps / Week                         >1B Pa...
whitehouse.gov/openforquestions                                         Google Confidential and Proprietary   24Monday, Ma...
Building with App Engine                         ∑                                                    Google Confidential ...
Language runtimes                                Duke, the Java mascot                           Copyright © Sun Microsyst...
Specialized services                         Memcache   Datastore   URL Fetch                           Mail      XMPP    ...
Ensuring portability                              Google Confidential and Proprietary   28Monday, March 14, 2011          ...
Building the App Engine way        BigTable is not a relational database        Can’t start your own threads        JRE wh...
Datastore APIs        Native        • get(Key key), put(Entity e), etc.        • But…entities are not POJOs, so you need.....
Objectify goodies        Light        • 36k jar, no external dependencies        Fast        • no bytecode enhancement on ...
How to do background processing?        Your application can’t start threads        • Causes problems for some frameworks ...
Queue a Task        Define queue in WEB-INF/queue.xml          <queue>                         <name>mail</name>          ...
Is there an easier way?                                 Google Confidential and Proprietary   34Monday, March 14, 2011    ...
Is there an easier way?        Vince Bonfanti’s Deferred task servlet        Task throttling can help with quotas         ...
What’s New In App Engine?                                                 Google Confidential and Proprietary   36Monday, ...
Namespaces API                                                                       SQL                                  ...
Namespace API - Memcache                           Memcache entry                                    entries     <app id> ...
Namespaces - Datastore                           Row data     <app id>            <namespace>                         <nam...
Channel API        Asynchronous Server ⟺ Client Communication        • Channel-based        • Bi-directional        Server...
Channel API demo        www.typrx.com     http://code.google.com/p/gwt-gae-channel/                                       ...
Channel API - Server Interface       /**        * ChannelService allows you to manage two-way connections        * with cl...
Channel API - Client (GWT)       http://code.google.com/p/gwt-gae-channel/                                                ...
MapReduce        MapReduce is a framework …        for processing huge datasets …        on certain kinds of distributable...
MapReduce                                              2010                         Google Developer Day 45               ...
Mapper API        Part 1 of the MapReduce story        Batch processing at scale        Sample Use Cases        • Report G...
Mapper API - OSS Project        Open source software - Apache 2 license        • http://mapreduce.appspot.com        • Pat...
Growing with App Engine                         * free monthly quota of ~5 million page views still in full effect        ...
Google App Engine for Business        Enterprise application management        • Centralized domain console        Enterpr...
Secure Data Connector (SDC)                                     Google Confidential and Proprietary   50Monday, March 14, ...
2½ years in review                     Apr   2008   Python launch                     May   2008   Memcache, Images API   ...
Monday, March 14, 2011   52
Thank you        Read more                   http://code.google.com/appengine/                   http://googleappengine.bl...
Upcoming SlideShare
Loading in...5
×

Develop and Deploy Scalable Apps with Google App Engine

4,325

Published on

Published in: Technology
0 Comments
6 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
4,325
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
0
Comments
0
Likes
6
Embeds 0
No embeds

No notes for slide

Transcript of "Develop and Deploy Scalable Apps with Google App Engine"

  1. 1. Develop and deploy scalable apps with Google App Engine David Chandler Developer Programs Engineer drfibonacci@google.com Google Confidential and Proprietary 1Monday, March 14, 2011 1
  2. 2. 2Monday, March 14, 2011 2
  3. 3. App Engine is part of Google’s cloud platform SaaS PaaS IaaS Source: Gartner AADI Summit Dec 2009 Google Confidential and Proprietary 3Monday, March 14, 2011 3
  4. 4. Googles Cloud Offerings Your Apps Google Apps Third party Apps: Google Apps Marketplace SaaS ________ Google App Engine PaaS Google Storage Prediction API IaaS BigQuery Google Confidential and Proprietary 4Monday, March 14, 2011 4
  5. 5. Easy to build Easy to manage Easy to scale Google App EngineMonday, March 14, 2011 5
  6. 6. Hosting means hidden costs Idle capacity Software patches & upgrades Maintenance Traffic & utilization forecasting Upgrades (install + config) Google Confidential and Proprietary 6Monday, March 14, 2011 6
  7. 7. Cloud development in a box Hardware Networking Operating system Application runtime • Java, Python Static file serving Services Fault tolerance Load balancing Google Confidential and Proprietary 7Monday, March 14, 2011 7
  8. 8. What is Google App Engine? Business: Google’s cloud computing platform Technical: giant Web farm in front of world’s largest HashMap Java developer: easiest and most scalable Java hosting environment Startup developer: FREE to get started App Engine expresses how Google thinks about infrastructure Google Confidential and Proprietary 8Monday, March 14, 2011 8
  9. 9. Let’s do it Sign up for AppEngine beta Download Google plugin for Eclipse Create new Google Web application Deploy it Google Confidential and Proprietary 9Monday, March 14, 2011 9
  10. 10. What didn’t we do? Install Tomcat Configure Tomcat Start Tomcat Stop Tomcat Install a database Configure a database Find a DBA to run scripts to populate a database… Google Confidential and Proprietary 10Monday, March 14, 2011 10
  11. 11. What didn’t we do? But what would we do differently to SCALE • our Web tier? • our data tier? • our asynchronous service tier? For example, • Azure: provision more SQL or IIS instances in admin console • AWS: same, using Rightscale or similar • AppEngine: see checklist… Google Confidential and Proprietary 11Monday, March 14, 2011 11
  12. 12. App Engine scaling checklist Google Confidential and Proprietary 12Monday, March 14, 2011 12
  13. 13. Google AppEngine at a glance Google Confidential and Proprietary 13Monday, March 14, 2011 13
  14. 14. What makes AppEngine special? Google Confidential and Proprietary 14Monday, March 14, 2011 14
  15. 15. Scalable serving infrastructure :-) Google Confidential and Proprietary 15Monday, March 14, 2011 15
  16. 16. BigTable: a Distributed Storage System Fay Chang, Jeffrey Dean, Sanjay Ghemawat, Wilson C. Hsieh, Deborah A. Wallach, Mike Burrows, Tushar Chandra, Andrew Fikes, and Robert E. Gruber Abstract Bigtable is a distributed storage system for managing structured data that is designed to scale to a very large size: petabytes of data across thousands of commodity servers. Many projects at Google store data in Bigtable, including web indexing, Google Earth, and Google Finance. These applications place very different demands on Bigtable, both in terms of data size (from URLs to web pages to satellite imagery) and latency requirements (from backend bulk processing to real-time data serving). Despite these varied demands, Bigtable has successfully provided a flexible, high- performance solution for all of these Google products. In this paper we describe the simple data model provided by Bigtable, which gives clients dynamic control over data layout and format, and we describe the design and implementation of Bigtable. Appeared in: OSDI06: Seventh Symposium on Operating System Design and Implementation, Seattle, WA, November, 2006. http://labs.google.com/papers/bigtable.html Google Confidential and Proprietary 16Monday, March 14, 2011 16
  17. 17. No servers to configure Google Confidential and Proprietary 17Monday, March 14, 2011 17
  18. 18. Easy deployment OR Google Confidential and Proprietary 18Monday, March 14, 2011 18
  19. 19. Powerful admin console Google Confidential and Proprietary 19Monday, March 14, 2011 19
  20. 20. Distributed frontends Edge Caching Priority routing Geographically close connection endpoint Google Confidential and Proprietary 20Monday, March 14, 2011 20
  21. 21. Free to get started 6.5 CPU hrs/day 1 GB storage 650K URL Fetch calls 2,000 recipients emailed 1 GB/day bandwidth 100K tasks ~5M pageviews / month Google Confidential and Proprietary 21Monday, March 14, 2011 21
  22. 22. Who’s using App Engine? Google Confidential and Proprietary 22Monday, March 14, 2011 22
  23. 23. By the numbers >100K Developers / Month >150K Apps / Week >1B Page Views / Day 23Monday, March 14, 2011 23
  24. 24. whitehouse.gov/openforquestions Google Confidential and Proprietary 24Monday, March 14, 2011 24
  25. 25. Building with App Engine ∑ Google Confidential and Proprietary 25Monday, March 14, 2011 25
  26. 26. Language runtimes Duke, the Java mascot Copyright © Sun Microsystems Inc., all rights reserved. Google Confidential and Proprietary 26Monday, March 14, 2011 26
  27. 27. Specialized services Memcache Datastore URL Fetch Mail XMPP Task Queue Images Blobstore User Service Google Confidential and Proprietary 27Monday, March 14, 2011 27
  28. 28. Ensuring portability Google Confidential and Proprietary 28Monday, March 14, 2011 28
  29. 29. Building the App Engine way BigTable is not a relational database Can’t start your own threads JRE whitelist Don’t forget quotas Google Confidential and Proprietary 29Monday, March 14, 2011 29
  30. 30. Datastore APIs Native • get(Key key), put(Entity e), etc. • But…entities are not POJOs, so you need... JDO or JPA Third-party • Objectify (http://code.google.com/p/objectify-appengine) • Twig, SimpleDS, others Google Confidential and Proprietary 30Monday, March 14, 2011 30
  31. 31. Objectify goodies Light • 36k jar, no external dependencies Fast • no bytecode enhancement on start Simple • No PersistenceManager lifecycle means no DI required, no attach / detach hassle @Cache (uses AppEngine Memcache) @Embedded (objects as list tuples) Google Confidential and Proprietary 31Monday, March 14, 2011 31
  32. 32. How to do background processing? Your application can’t start threads • Causes problems for some frameworks • See will-it-play-in-appengine wiki • No, you can’t run WebSphere or JBoss on AppEngine AppEngine provides Task Queue API • Servlet-based • Allows throttling Google Confidential and Proprietary 32Monday, March 14, 2011 32
  33. 33. Queue a Task Define queue in WEB-INF/queue.xml <queue> <name>mail</name> <rate>8/m</rate> </queue> Create worker servlet that handles task Queue the task queue.add(url("/mail").param("key", key)…) Google Confidential and Proprietary 33Monday, March 14, 2011 33
  34. 34. Is there an easier way? Google Confidential and Proprietary 34Monday, March 14, 2011 34
  35. 35. Is there an easier way? Vince Bonfanti’s Deferred task servlet Task throttling can help with quotas Google Confidential and Proprietary 35Monday, March 14, 2011 35
  36. 36. What’s New In App Engine? Google Confidential and Proprietary 36Monday, March 14, 2011 36
  37. 37. Namespaces API SQL App App Server Server public void doFilter(ServletRequest req, ServletResponse resp, FilterChain chain) throws IOException, ServletException {   NamespaceManager.set(...);   chain.doFilter(request, response); } 22 Google Confidential and ProprietaryMonday, March 14, 2011 37
  38. 38. Namespace API - Memcache Memcache entry entries <app id> <namespace> <app id> <key> <value> <key> <value> <key> <value> <key> <value> <key> <value> <key> <value> <key> <value> <key> <value> Google Confidential and Proprietary 38Monday, March 14, 2011 38
  39. 39. Namespaces - Datastore Row data <app id> <namespace> <namespace> <app id> <kind> <key> <properties> Single property index <app id> <namespace> <namespace> <app id> <kind> <prop> <value> <key> Composite Index <app id> <namespace> <namespace> <app id> <kind> <prop1> <prop2> <key> Google Confidential and Proprietary 39Monday, March 14, 2011 39
  40. 40. Channel API Asynchronous Server ⟺ Client Communication • Channel-based • Bi-directional Server • Send messages via ChannelService object • Receive messages in a web hook Client • JavaScript library • Receive server messages in a callback Built on Gmail chat client (Google Talk) Google Confidential and Proprietary 40Monday, March 14, 2011 40
  41. 41. Channel API demo www.typrx.com http://code.google.com/p/gwt-gae-channel/ Google Confidential and Proprietary 41Monday, March 14, 2011 41
  42. 42. Channel API - Server Interface /** * ChannelService allows you to manage two-way connections * with clients. */ public interface ChannelService { /** * Creates a channel associated with the provided applicationKey */ String createChannel(String applicationKey); /** * Sends a ChannelMessage to the client. */ void sendMessage(ChannelMessage message); /** * Parse the incoming message in request. This method * should only be called within a channel webhook. */ ChannelMessage parseMessage(HttpServletRequest request); } 42 Google Developer Day 2010Monday, March 14, 2011 42
  43. 43. Channel API - Client (GWT) http://code.google.com/p/gwt-gae-channel/ Google Confidential and Proprietary 43Monday, March 14, 2011 43
  44. 44. MapReduce MapReduce is a framework … for processing huge datasets … on certain kinds of distributable problems … using a large number of computers (nodes), collectively referred to as a cluster http://en.wikipedia.org/wiki/MapReduce Think search indexing... Google Confidential and Proprietary 44Monday, March 14, 2011 44
  45. 45. MapReduce 2010 Google Developer Day 45 Google Confidential and ProprietaryMonday, March 14, 2011 45
  46. 46. Mapper API Part 1 of the MapReduce story Batch processing at scale Sample Use Cases • Report Generation • Data export • Schema Migration Built on Task Queues Rate configurable Google Confidential and Proprietary 46Monday, March 14, 2011 46
  47. 47. Mapper API - OSS Project Open source software - Apache 2 license • http://mapreduce.appspot.com • Patches welcome :) Entirely user-space • Just include it in your project! Python & Java versions Features • Automatic sharding • Automatic rate limiting • /mapreduce/status • Counters • Batching datastore operations Google Confidential and Proprietary 47Monday, March 14, 2011 47
  48. 48. Growing with App Engine * free monthly quota of ~5 million page views still in full effect Google Confidential and Proprietary 48Monday, March 14, 2011 48
  49. 49. Google App Engine for Business Enterprise application management • Centralized domain console Enterprise reliability and support Same scalable platform. Designed for enterprise. • 99.9% Service Level Agreement • Premium Developer Support Hosted SQL • Managed relational SQL database in the cloud SSL on your domain • Including "naked" domain support Google App Engine for Business Secure by default • Integrated Single Sign On (SSO Google Confidential and Proprietary 49Monday, March 14, 2011 49
  50. 50. Secure Data Connector (SDC) Google Confidential and Proprietary 50Monday, March 14, 2011 50
  51. 51. 2½ years in review Apr 2008 Python launch May 2008 Memcache, Images API Jul 2008 Logs export Aug 2008 Batch write/delete Oct 2008 HTTPS support Dec 2008 Status dashboard, quota details Feb 2009 Billing, larger files Apr 2009 Java launch, DB import, cron support, SDC May 2009 Key-only queries Jun 2009 Task queues Aug 2009 Kindless queries Sep 2009 XMPP Oct 2009 Incoming email Dec 2009 Blobstore API, Java Pre-compilation Feb 2010 Datastore cursors, !1000, automatic datastore retries Mar 2010 Appstats, Read policies, IPv6 May 2010 App Engine for Business, bulkloader, OpenID, OAuth Jun 2010 Task queue increases, Python pre-compilation, Blobs++ Jul 2010 Mapper API Aug 2010 Multi-tenancy, High perf. dynamic image serving, Custom err pages Oct 2010 Instances Console, Delete Kind/App Data Nov 2010 Channel API, Warmup / Always On, 10min tasks 51Monday, March 14, 2011 51
  52. 52. Monday, March 14, 2011 52
  53. 53. Thank you Read more http://code.google.com/appengine/ http://googleappengine.blogspot.com Contact info David Chandler Developer Programs Engineer drfibonacci@google.com http://turbomanage.wordpress.com Google Confidential and Proprietary 53Monday, March 14, 2011 53

×