Develop and Deploy Scalable Apps with Google App Engine

  • 4,194 views
Uploaded on

 

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
4,194
On Slideshare
0
From Embeds
0
Number of Embeds
2

Actions

Shares
Downloads
0
Comments
0
Likes
5

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Develop and deploy scalable apps with Google App Engine David Chandler Developer Programs Engineer drfibonacci@google.com Google Confidential and Proprietary 1Monday, March 14, 2011 1
  • 2. 2Monday, March 14, 2011 2
  • 3. App Engine is part of Google’s cloud platform SaaS PaaS IaaS Source: Gartner AADI Summit Dec 2009 Google Confidential and Proprietary 3Monday, March 14, 2011 3
  • 4. Googles Cloud Offerings Your Apps Google Apps Third party Apps: Google Apps Marketplace SaaS ________ Google App Engine PaaS Google Storage Prediction API IaaS BigQuery Google Confidential and Proprietary 4Monday, March 14, 2011 4
  • 5. Easy to build Easy to manage Easy to scale Google App EngineMonday, March 14, 2011 5
  • 6. Hosting means hidden costs Idle capacity Software patches & upgrades Maintenance Traffic & utilization forecasting Upgrades (install + config) Google Confidential and Proprietary 6Monday, March 14, 2011 6
  • 7. Cloud development in a box Hardware Networking Operating system Application runtime • Java, Python Static file serving Services Fault tolerance Load balancing Google Confidential and Proprietary 7Monday, March 14, 2011 7
  • 8. What is Google App Engine? Business: Google’s cloud computing platform Technical: giant Web farm in front of world’s largest HashMap Java developer: easiest and most scalable Java hosting environment Startup developer: FREE to get started App Engine expresses how Google thinks about infrastructure Google Confidential and Proprietary 8Monday, March 14, 2011 8
  • 9. Let’s do it Sign up for AppEngine beta Download Google plugin for Eclipse Create new Google Web application Deploy it Google Confidential and Proprietary 9Monday, March 14, 2011 9
  • 10. What didn’t we do? Install Tomcat Configure Tomcat Start Tomcat Stop Tomcat Install a database Configure a database Find a DBA to run scripts to populate a database… Google Confidential and Proprietary 10Monday, March 14, 2011 10
  • 11. What didn’t we do? But what would we do differently to SCALE • our Web tier? • our data tier? • our asynchronous service tier? For example, • Azure: provision more SQL or IIS instances in admin console • AWS: same, using Rightscale or similar • AppEngine: see checklist… Google Confidential and Proprietary 11Monday, March 14, 2011 11
  • 12. App Engine scaling checklist Google Confidential and Proprietary 12Monday, March 14, 2011 12
  • 13. Google AppEngine at a glance Google Confidential and Proprietary 13Monday, March 14, 2011 13
  • 14. What makes AppEngine special? Google Confidential and Proprietary 14Monday, March 14, 2011 14
  • 15. Scalable serving infrastructure :-) Google Confidential and Proprietary 15Monday, March 14, 2011 15
  • 16. BigTable: a Distributed Storage System Fay Chang, Jeffrey Dean, Sanjay Ghemawat, Wilson C. Hsieh, Deborah A. Wallach, Mike Burrows, Tushar Chandra, Andrew Fikes, and Robert E. Gruber Abstract Bigtable is a distributed storage system for managing structured data that is designed to scale to a very large size: petabytes of data across thousands of commodity servers. Many projects at Google store data in Bigtable, including web indexing, Google Earth, and Google Finance. These applications place very different demands on Bigtable, both in terms of data size (from URLs to web pages to satellite imagery) and latency requirements (from backend bulk processing to real-time data serving). Despite these varied demands, Bigtable has successfully provided a flexible, high- performance solution for all of these Google products. In this paper we describe the simple data model provided by Bigtable, which gives clients dynamic control over data layout and format, and we describe the design and implementation of Bigtable. Appeared in: OSDI06: Seventh Symposium on Operating System Design and Implementation, Seattle, WA, November, 2006. http://labs.google.com/papers/bigtable.html Google Confidential and Proprietary 16Monday, March 14, 2011 16
  • 17. No servers to configure Google Confidential and Proprietary 17Monday, March 14, 2011 17
  • 18. Easy deployment OR Google Confidential and Proprietary 18Monday, March 14, 2011 18
  • 19. Powerful admin console Google Confidential and Proprietary 19Monday, March 14, 2011 19
  • 20. Distributed frontends Edge Caching Priority routing Geographically close connection endpoint Google Confidential and Proprietary 20Monday, March 14, 2011 20
  • 21. Free to get started 6.5 CPU hrs/day 1 GB storage 650K URL Fetch calls 2,000 recipients emailed 1 GB/day bandwidth 100K tasks ~5M pageviews / month Google Confidential and Proprietary 21Monday, March 14, 2011 21
  • 22. Who’s using App Engine? Google Confidential and Proprietary 22Monday, March 14, 2011 22
  • 23. By the numbers >100K Developers / Month >150K Apps / Week >1B Page Views / Day 23Monday, March 14, 2011 23
  • 24. whitehouse.gov/openforquestions Google Confidential and Proprietary 24Monday, March 14, 2011 24
  • 25. Building with App Engine ∑ Google Confidential and Proprietary 25Monday, March 14, 2011 25
  • 26. Language runtimes Duke, the Java mascot Copyright © Sun Microsystems Inc., all rights reserved. Google Confidential and Proprietary 26Monday, March 14, 2011 26
  • 27. Specialized services Memcache Datastore URL Fetch Mail XMPP Task Queue Images Blobstore User Service Google Confidential and Proprietary 27Monday, March 14, 2011 27
  • 28. Ensuring portability Google Confidential and Proprietary 28Monday, March 14, 2011 28
  • 29. Building the App Engine way BigTable is not a relational database Can’t start your own threads JRE whitelist Don’t forget quotas Google Confidential and Proprietary 29Monday, March 14, 2011 29
  • 30. Datastore APIs Native • get(Key key), put(Entity e), etc. • But…entities are not POJOs, so you need... JDO or JPA Third-party • Objectify (http://code.google.com/p/objectify-appengine) • Twig, SimpleDS, others Google Confidential and Proprietary 30Monday, March 14, 2011 30
  • 31. Objectify goodies Light • 36k jar, no external dependencies Fast • no bytecode enhancement on start Simple • No PersistenceManager lifecycle means no DI required, no attach / detach hassle @Cache (uses AppEngine Memcache) @Embedded (objects as list tuples) Google Confidential and Proprietary 31Monday, March 14, 2011 31
  • 32. How to do background processing? Your application can’t start threads • Causes problems for some frameworks • See will-it-play-in-appengine wiki • No, you can’t run WebSphere or JBoss on AppEngine AppEngine provides Task Queue API • Servlet-based • Allows throttling Google Confidential and Proprietary 32Monday, March 14, 2011 32
  • 33. Queue a Task Define queue in WEB-INF/queue.xml <queue> <name>mail</name> <rate>8/m</rate> </queue> Create worker servlet that handles task Queue the task queue.add(url("/mail").param("key", key)…) Google Confidential and Proprietary 33Monday, March 14, 2011 33
  • 34. Is there an easier way? Google Confidential and Proprietary 34Monday, March 14, 2011 34
  • 35. Is there an easier way? Vince Bonfanti’s Deferred task servlet Task throttling can help with quotas Google Confidential and Proprietary 35Monday, March 14, 2011 35
  • 36. What’s New In App Engine? Google Confidential and Proprietary 36Monday, March 14, 2011 36
  • 37. Namespaces API SQL App App Server Server public void doFilter(ServletRequest req, ServletResponse resp, FilterChain chain) throws IOException, ServletException {   NamespaceManager.set(...);   chain.doFilter(request, response); } 22 Google Confidential and ProprietaryMonday, March 14, 2011 37
  • 38. Namespace API - Memcache Memcache entry entries <app id> <namespace> <app id> <key> <value> <key> <value> <key> <value> <key> <value> <key> <value> <key> <value> <key> <value> <key> <value> Google Confidential and Proprietary 38Monday, March 14, 2011 38
  • 39. Namespaces - Datastore Row data <app id> <namespace> <namespace> <app id> <kind> <key> <properties> Single property index <app id> <namespace> <namespace> <app id> <kind> <prop> <value> <key> Composite Index <app id> <namespace> <namespace> <app id> <kind> <prop1> <prop2> <key> Google Confidential and Proprietary 39Monday, March 14, 2011 39
  • 40. Channel API Asynchronous Server ⟺ Client Communication • Channel-based • Bi-directional Server • Send messages via ChannelService object • Receive messages in a web hook Client • JavaScript library • Receive server messages in a callback Built on Gmail chat client (Google Talk) Google Confidential and Proprietary 40Monday, March 14, 2011 40
  • 41. Channel API demo www.typrx.com http://code.google.com/p/gwt-gae-channel/ Google Confidential and Proprietary 41Monday, March 14, 2011 41
  • 42. Channel API - Server Interface /** * ChannelService allows you to manage two-way connections * with clients. */ public interface ChannelService { /** * Creates a channel associated with the provided applicationKey */ String createChannel(String applicationKey); /** * Sends a ChannelMessage to the client. */ void sendMessage(ChannelMessage message); /** * Parse the incoming message in request. This method * should only be called within a channel webhook. */ ChannelMessage parseMessage(HttpServletRequest request); } 42 Google Developer Day 2010Monday, March 14, 2011 42
  • 43. Channel API - Client (GWT) http://code.google.com/p/gwt-gae-channel/ Google Confidential and Proprietary 43Monday, March 14, 2011 43
  • 44. MapReduce MapReduce is a framework … for processing huge datasets … on certain kinds of distributable problems … using a large number of computers (nodes), collectively referred to as a cluster http://en.wikipedia.org/wiki/MapReduce Think search indexing... Google Confidential and Proprietary 44Monday, March 14, 2011 44
  • 45. MapReduce 2010 Google Developer Day 45 Google Confidential and ProprietaryMonday, March 14, 2011 45
  • 46. Mapper API Part 1 of the MapReduce story Batch processing at scale Sample Use Cases • Report Generation • Data export • Schema Migration Built on Task Queues Rate configurable Google Confidential and Proprietary 46Monday, March 14, 2011 46
  • 47. Mapper API - OSS Project Open source software - Apache 2 license • http://mapreduce.appspot.com • Patches welcome :) Entirely user-space • Just include it in your project! Python & Java versions Features • Automatic sharding • Automatic rate limiting • /mapreduce/status • Counters • Batching datastore operations Google Confidential and Proprietary 47Monday, March 14, 2011 47
  • 48. Growing with App Engine * free monthly quota of ~5 million page views still in full effect Google Confidential and Proprietary 48Monday, March 14, 2011 48
  • 49. Google App Engine for Business Enterprise application management • Centralized domain console Enterprise reliability and support Same scalable platform. Designed for enterprise. • 99.9% Service Level Agreement • Premium Developer Support Hosted SQL • Managed relational SQL database in the cloud SSL on your domain • Including "naked" domain support Google App Engine for Business Secure by default • Integrated Single Sign On (SSO Google Confidential and Proprietary 49Monday, March 14, 2011 49
  • 50. Secure Data Connector (SDC) Google Confidential and Proprietary 50Monday, March 14, 2011 50
  • 51. 2½ years in review Apr 2008 Python launch May 2008 Memcache, Images API Jul 2008 Logs export Aug 2008 Batch write/delete Oct 2008 HTTPS support Dec 2008 Status dashboard, quota details Feb 2009 Billing, larger files Apr 2009 Java launch, DB import, cron support, SDC May 2009 Key-only queries Jun 2009 Task queues Aug 2009 Kindless queries Sep 2009 XMPP Oct 2009 Incoming email Dec 2009 Blobstore API, Java Pre-compilation Feb 2010 Datastore cursors, !1000, automatic datastore retries Mar 2010 Appstats, Read policies, IPv6 May 2010 App Engine for Business, bulkloader, OpenID, OAuth Jun 2010 Task queue increases, Python pre-compilation, Blobs++ Jul 2010 Mapper API Aug 2010 Multi-tenancy, High perf. dynamic image serving, Custom err pages Oct 2010 Instances Console, Delete Kind/App Data Nov 2010 Channel API, Warmup / Always On, 10min tasks 51Monday, March 14, 2011 51
  • 52. Monday, March 14, 2011 52
  • 53. Thank you Read more http://code.google.com/appengine/ http://googleappengine.blogspot.com Contact info David Chandler Developer Programs Engineer drfibonacci@google.com http://turbomanage.wordpress.com Google Confidential and Proprietary 53Monday, March 14, 2011 53