Scalable Apps with Google App Engine
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share

Scalable Apps with Google App Engine

  • 5,327 views
Uploaded on

Introduction to Google App Engine, new features, thoughts on the Datastore

Introduction to Google App Engine, new features, thoughts on the Datastore

More in: Technology , Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
5,327
On Slideshare
5,297
From Embeds
30
Number of Embeds
3

Actions

Shares
Downloads
65
Comments
0
Likes
1

Embeds 30

http://gae564-2011.wikispaces.asu.edu 24
http://dkrishnaprasad.wordpress.com 5
http://www.linkedin.com 1

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Building Scalable Apps with Google App Engine David Chandler Developer Programs Engineer drfibonacci@google.com CloudDevDay Detroit Jun 11, 2011 Google Confidential and Proprietary 1Monday, June 13, 2011 1
  • 2. Agenda A word about the cloud A Brief History of App Engine Building with App Engine What’s New in App Engine Q&A Google Confidential and Proprietary 2Monday, June 13, 2011 2
  • 3. 3Monday, June 13, 2011 3
  • 4. What is SaaS cloud computing? PaaS IaaS Software… Platform… Infrastructure… Postage Place Here … as a Service 4Monday, June 13, 2011 4
  • 5. App Engine is part of Google’s cloud platform SaaS PaaS IaaS Source: Gartner AADI Summit Dec 2009 Google Confidential and Proprietary 5Monday, June 13, 2011 5
  • 6. Googles Cloud Offerings Your Apps Google Apps Third party Apps: Google Apps Marketplace SaaS ________ Google App Engine PaaS Google Storage Prediction API IaaS BigQuery Google Confidential and Proprietary 6Monday, June 13, 2011 6
  • 7. Not all "Platforms" are alike Google Confidential and ProprietaryMonday, June 13, 2011 7
  • 8. Cloud development in a box Hardware Networking Operating system Application runtime • Java, Python Static file serving Services Fault tolerance Load balancing Google Confidential and Proprietary 8Monday, June 13, 2011 8
  • 9. What is Google App Engine? Business: Google’s cloud computing platform Technical: giant Web farm in front of world’s largest HashMap Java developer: easiest and most scalable Java hosting environment Startup developer: FREE to get started App Engine expresses how Google thinks about infrastructure Google Confidential and Proprietary 9Monday, June 13, 2011 9
  • 10. Let’s do it Sign up for AppEngine beta Install Google Plugin for Eclipse Create new Google Web application Deploy it Google Confidential and Proprietary 10Monday, June 13, 2011 10
  • 11. What did we actually do? Hot deployed an exploded WAR file NO server setup NO instance to provision NO networking NO J2EE config NO start/stop Google Confidential and Proprietary 11Monday, June 13, 2011 11
  • 12. What makes AppEngine special? Google Confidential and Proprietary 12Monday, June 13, 2011 12
  • 13. BigTable: a Distributed Storage System Fay Chang, Jeffrey Dean, Sanjay Ghemawat, Wilson C. Hsieh, Deborah A. Wallach, Mike Burrows, Tushar Chandra, Andrew Fikes, and Robert E. Gruber Abstract Bigtable is a distributed storage system for managing structured data that is designed to scale to a very large size: petabytes of data across thousands of commodity servers. Many projects at Google store data in Bigtable, including web indexing, Google Earth, and Google Finance. These applications place very different demands on Bigtable, both in terms of data size (from URLs to web pages to satellite imagery) and latency requirements (from backend bulk processing to real-time data serving). Despite these varied demands, Bigtable has successfully provided a flexible, high- performance solution for all of these Google products. In this paper we describe the simple data model provided by Bigtable, which gives clients dynamic control over data layout and format, and we describe the design and implementation of Bigtable. Appeared in: OSDI06: Seventh Symposium on Operating System Design and Implementation, Seattle, WA, November, 2006. http://labs.google.com/papers/bigtable.html Google Confidential and Proprietary 13Monday, June 13, 2011 13
  • 14. High Replication Datastore Default when you create a new app Actively replicated to multiple data centers Google Confidential and Proprietary 14Monday, June 13, 2011 14
  • 15. No servers to configure Google Confidential and Proprietary 15Monday, June 13, 2011 15
  • 16. Easy deployment OR Google Confidential and Proprietary 16Monday, June 13, 2011 16
  • 17. Powerful admin console Google Confidential and Proprietary 17Monday, June 13, 2011 17
  • 18. Distributed frontends Edge Caching Priority routing Geographically close connection endpoint Google Confidential and Proprietary 18Monday, June 13, 2011 18
  • 19. Free to get started 6.5 CPU hrs/day 1 GB storage 650K URL Fetch calls 2,000 recipients emailed 1 GB/day bandwidth 100K tasks $0.72 backend usage (9h on small instance) ~5M pageviews / month Google Confidential and Proprietary 19Monday, June 13, 2011 19
  • 20. A Brief History of App EngineMonday, June 13, 2011 20
  • 21. App Engine Growth 2008 2009 2010 2011 App Engine Launch Batch write/read Java Task Queues Blobstore Multitenancy Hi-Replication Python Https DB Import XMPP Appstats Instance Console Datastore Datastore Status- cron incoming email cursors Always On Backends! Memcache Dashboard Mapper hi-perf imag Channel API logs export 10 min tasks Files API Remote API Prosp SearchMonday, June 13, 2011 21
  • 22. By the Numbers 100,000+ Active Developers per MonthMonday, June 13, 2011 22
  • 23. By the Numbers 200,000+ Active apps per weekMonday, June 13, 2011 23
  • 24. By the Numbers 1.5B+ Pageviews per dayMonday, June 13, 2011 24
  • 25. Official Royal Wedding on App Engine On Wedding day, served: • Up to 32k requests per second • 37.7 million pageviews • 13.7 million visitors http://goo.gl/F1SGcMonday, June 13, 2011 25
  • 26. Google Confidential and ProprietaryMonday, June 13, 2011 26
  • 27. Building with App Engine ∑ Google Confidential and Proprietary 27Monday, June 13, 2011 27
  • 28. Language runtimes Duke, the Java mascot Copyright © Sun Microsystems Inc., all rights reserved. Google Confidential and Proprietary 28Monday, June 13, 2011 28
  • 29. Specialized services Memcache Datastore URL Fetch Mail XMPP Task Queue Images Blobstore User Service Google Confidential and Proprietary 29Monday, June 13, 2011 29
  • 30. Services are simple UserService svc = UserServiceFactory.getUserService(); User authenticatedUser = svc.getUser(); Google Confidential and Proprietary 30Monday, June 13, 2011 30
  • 31. Datastore APIs Native • DatastoreServiceFactory.getDatastoreService().get(Key key) • DatastoreServiceFactory.getDatastoreService().put(Entity entity) • But…entities are not POJOs JDO, JPA, or third party • Objectify (http://code.google.com/p/objectify-appengine) • Twig, SimpleDS, Slim3, others Objectify example (using DAOBase) • ofy().put(Person p); • ofy().get(Person.class, 238749); Google Confidential and Proprietary 31Monday, June 13, 2011 31
  • 32. Objectify goodies Light • 36k jar, no external dependencies Fast • no bytecode enhancement on start Simple • No PersistenceManager lifecycle means no DI required • No attach / detach hassle • No “find to delete” • No lazy fetching, proxy classes... @Cache (uses App Engine Memcache) @Embedded (objects as list tuples) Google Confidential and Proprietary 32Monday, June 13, 2011 32
  • 33. How to learn Datastore Understand consistency vs. concurrency tradeoff • Selective denormalization sometimes required • Use the power of the platform, don’t try to make it what it isn’t (relational)  Batch gets / puts  List properties for fast, lightweight indexes  Retrieve parent entity from child key Watch previous Google I/O presentations • Brett Slatkin 2009 on list properties and million-user fanout • Book: see GWT Books page at code.google.com Google Confidential and Proprietary 33Monday, June 13, 2011 33
  • 34. Secure Data Connector (SDC) Google Confidential and Proprietary 34Monday, June 13, 2011 34
  • 35. How to do background processing? Your application can’t start threads • Causes problems for some frameworks • See will-it-play-in-appengine wiki • No, you can’t run WebSphere or JBoss on AppEngine AppEngine provides Task Queue API • Servlet-based • Allows throttling Backends (new!) Google Confidential and Proprietary 35Monday, June 13, 2011 35
  • 36. Queue a Task Define queue in WEB-INF/queue.xml <queue> <name>mail</name> <rate>8/m</rate> </queue> Create worker servlet that handles task Queue the task queue.add(url("/mail").param("key", key)…) Google Confidential and Proprietary 36Monday, June 13, 2011 36
  • 37. Is there an easier way? Google Confidential and Proprietary 37Monday, June 13, 2011 37
  • 38. Is there an easier way? Vince Bonfanti’s Deferred task servlet Now part of App Engine API! • com.google.appengine.api.taskqueue.DeferredTask Task throttling can help with quotas Google Confidential and Proprietary 38Monday, June 13, 2011 38
  • 39. What’s New In App Engine? Google Confidential and Proprietary 39Monday, June 13, 2011 39
  • 40. Namespaces API What’s wrong with this query in a multi-tenant app? DELETE FROM Transactions WHERE AcctID = 2347272 DELETE FROM Transactions WHERE CustID = 2394872 AND AcctID = 2347272 Oops. Google Confidential and Proprietary 40Monday, June 13, 2011 40
  • 41. Namespaces API SQL App App Server Server public void doFilter(ServletRequest req, ServletResponse resp, FilterChain chain) throws IOException, ServletException {   NamespaceManager.set(...);   chain.doFilter(request, response); } 22 Google Confidential and ProprietaryMonday, June 13, 2011 41
  • 42. Namespace API - Memcache Memcache entry entries <app id> <namespace> <app id> <key> <value> <key> <value> <key> <value> <key> <value> <key> <value> <key> <value> <key> <value> <key> <value> Google Confidential and Proprietary 42Monday, June 13, 2011 42
  • 43. Namespaces - Datastore Row data <app id> <namespace> <namespace> <app id> <kind> <key> <properties> Single property index <app id> <namespace> <namespace> <app id> <kind> <prop> <value> <key> Composite Index <app id> <namespace> <namespace> <app id> <kind> <prop1> <prop2> <key> Google Confidential and Proprietary 43Monday, June 13, 2011 43
  • 44. Channel API Asynchronous Server ⟺ Client Communication • Channel-based • Bi-directional Server • Send messages via ChannelService object • Receive messages in a web hook Client • JavaScript library • Receive server messages in a callback Built on Gmail chat client (Google Talk) Google Confidential and Proprietary 44Monday, June 13, 2011 44
  • 45. Channel API demo www.typrx.com http://code.google.com/p/gwt-gae-channel/ Google Confidential and Proprietary 45Monday, June 13, 2011 45
  • 46. Channel API - Server Interface /** * ChannelService allows you to manage two-way connections * with clients. */ public interface ChannelService { /** * Creates a channel associated with the provided applicationKey */ String createChannel(String applicationKey); /** * Sends a ChannelMessage to the client. */ void sendMessage(ChannelMessage message); /** * Parse the incoming message in request. This method * should only be called within a channel webhook. */ ChannelMessage parseMessage(HttpServletRequest request); } Google Confidential and Proprietary 46Monday, June 13, 2011 46
  • 47. Channel API - Client (GWT) http://code.google.com/p/gwt-gae-channel/ Google Confidential and Proprietary 47Monday, June 13, 2011 47
  • 48. Android Tooling in GPE In GPE 2.4 beta • Debug As | Local App Engine Connected Android Application • Launches Android emulator • Allows apps in the emulator to call App Engine dev server on localhost See App Engine + Android session from this year’s Google I/O Google Confidential and Proprietary 48Monday, June 13, 2011 48
  • 49. App Engine is Leaving Preview (Beta)! Full Google Product - 3 year deprecation support 99.95% SLA for all paid users Operational and Developer Support available New business-friendly Terms of Service Billing via invoice Rounded out feature set including Frontend and Backend capabilities New sustainable pricing model Google Confidential and Proprietary 49Monday, June 13, 2011 49
  • 50. SLAs are more than just 9s SLA is about transparency HRD does not need planned downtime Uptime will be measured by 1000s of various probes from around the globe Customers will be able to contribute to our metrics Upcoming Monitoring API to automatically monitor your own applications Google Confidential and Proprietary 50Monday, June 13, 2011 50
  • 51. Frontends and Backends • Instances (running unit for both Frontends and Backends) o Have set amount of CPU and Memory o Run your code • Frontends o Scale dynamically based on your traffic  The App Engine Scheduler determines when to spin up or down Frontend instances  Controls will be added to allow customers to optimize the Scheduler for particular usage • Backends (Launched at I/O with 1.5.0!) o Run based on admin control o Choose CPU and Memory size o Long-running (no 30-sec HTTP request limit) Google Confidential and Proprietary 51Monday, June 13, 2011 51
  • 52. New Pricing Model Overview • Usage types: o Free - similar to today o Paid - $9/app/mo (+ usage) including SLA o Premier - $500/account/mo (+ usage) including operational support • No more CPU hours o Charging for size and number of running Frontends/Backends (incorporating CPU + memory) o APIs (including Datastore) charged based on operations • Add-ons o For example: SSL for custom domains will be available for monthly fee Google Confidential and Proprietary 52Monday, June 13, 2011 52
  • 53. Thank you! http://code.google.com Google Confidential and Proprietary 53Monday, June 13, 2011 53