Google App Engine - Devfest India 2010

5,135 views
5,023 views

Published on

Intro to Google App Engine, Google's cloud computing platform.
Presented at Devfest India 2010 Hyderabad andPune.

Published in: Technology, Business
1 Comment
9 Likes
Statistics
Notes
  • Thanks, Patrick for a very good presentation. With support for more languages coming on, surely it opens up the platform. Out of curiosity, though, what was the driver for moving away, if I may say so, from Python to Java or a JVM-based platform?

    - Saurabh
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total views
5,135
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
261
Comments
1
Likes
9
Embeds 0
No embeds

No notes for slide

Google App Engine - Devfest India 2010

  1. 1. Google App Engine Feb – March, 2010 Patrick Chanezon Developer Advocate Google Developer Relations chanezon@google.com Google Confidential and Proprietary 1 Monday, March 8, 2010
  2. 2. Agenda - App Engine introduction - Why build it? - App Engine tour - What's different? - Wrap up - Questions 2 Monday, March 8, 2010
  3. 3. 4. The spirit is unmoving, one, swifter than the mind. The senses do not reach It as It is ever ahead of them. Though Itself standing still It outstrips those who run. In It the all-pervading air supports the activities of beings Isha Upanishad 3 Monday, March 8, 2010
  4. 4. Vayu computing? Public domain image from http://commons.wikimedia.org/wiki/File:Vayu.jpg 4 Monday, March 8, 2010
  5. 5. Vayu Computing? 7−10 Then they said to Vayu (Air): "O Vayu! Find out who this great Spirit is." "Yes," he said, and hastened to It. Brahman asked him: "Who are you?" He replied "I am known as Vayu; I am also called Matarisva." Brahman said: "What power is in you, who are so well known?" Vayu replied: "I can carry off all−whatever there is on earth." Brahman put a straw before him and said: "Carry this." He rushed toward it with all his ardour but could not move it. Then he returned from the Spirit and said to the gods: "I could not find out who this Spirit is," Kena Upanishad 5 Monday, March 8, 2010
  6. 6. What is cloud computing? SaaS PaaS IaaS Postage Place Here 6 Monday, March 8, 2010
  7. 7. IaaS value proposition… 7 Monday, March 8, 2010
  8. 8. IaaS value proposition… 7 Monday, March 8, 2010
  9. 9. IaaS value proposition… 7 Monday, March 8, 2010
  10. 10. IaaS value proposition… 7 Monday, March 8, 2010
  11. 11. Google App Engine “We wear pagers so you don’t have to” pager 8 Monday, March 8, 2010
  12. 12. Google App Engine - Easy to build - Easy to maintain - Easy to scale 9 Monday, March 8, 2010
  13. 13. By the numbers - Built 100K apps - Maintained by 250K developers - Scaled to 250M pageviews daily semi-transparent collage of apps 10 Monday, March 8, 2010
  14. 14. gigya Socialize 11 Monday, March 8, 2010
  15. 15. gigya Socialize - traffic 12 Monday, March 8, 2010
  16. 16. gigya Socialize - traffic 12 Monday, March 8, 2010
  17. 17. Gigya Socialize - traffic 13 Monday, March 8, 2010
  18. 18. App Engine 14 Monday, March 8, 2010
  19. 19. App Engine 15 Monday, March 8, 2010
  20. 20. Time.com 16 Monday, March 8, 2010
  21. 21. Socialwok 17 Monday, March 8, 2010
  22. 22. Social networking at scale 18 Monday, March 8, 2010
  23. 23. Social networking at scale >40M Users 18 Monday, March 8, 2010
  24. 24. PubSubHubbub http://code.google.com/p/pubsubhubbub/ The future is distributed: There will be big hubs, many small hubs, and tons of publishers and subscribers. Publishers, subscribers, and hubs may play multiple roles. 19 Monday, March 8, 2010
  25. 25. Hell's Kitchen on facebook 20 Monday, March 8, 2010
  26. 26. Handling peak traffic: mental_floss 21 Monday, March 8, 2010
  27. 27. Elastic / redundant capacity 22 Monday, March 8, 2010
  28. 28. Short term events: earthhour.org 23 Monday, March 8, 2010
  29. 29. http://www.allforgood.org/ 24 Monday, March 8, 2010
  30. 30. http://www.snapabug.com/ 25 Monday, March 8, 2010
  31. 31. 26 Monday, March 8, 2010
  32. 32. Japan: Q&A for our Future 27 Monday, March 8, 2010
  33. 33. http://www.google.com/tipjar 28 Monday, March 8, 2010
  34. 34. Chillingo Crystal Gaming meets Social Zombie Dash Angry Birds LITE underground Meltdown Cogs Mission Deep Sea Speed Forge Guerilla Bob Ravensword: Angry Birds Extreme The Fallen King Monday, March 8, 2010
  35. 35. Why build it? Wrench 30 Monday, March 8, 2010
  36. 36. It's just too difficult 31 Monday, March 8, 2010
  37. 37. Hosting means hidden costs • Idle capacity • Software patches & upgrades • License fees • Lots of maintenance • Traffic & utilization forecasting • Upgrades 32 Monday, March 8, 2010
  38. 38. Cloud development in a box • SDK & “The Cloud” • Hardware • Networking • Operating system • Application runtime Java, Python o • Static file serving • Services • Fault tolerance • Load balancing 33 Monday, March 8, 2010
  39. 39. Distributed Cloud Computing Reliability and scalability Monday, March 8, 2010
  40. 40. Know your Distributed Memes - Divide & conquer - Sharding - Tolerate / expect failure - Idempotency - Just scale it up 35 Monday, March 8, 2010
  41. 41. Rocket science Monolithic Computing 36 Monday, March 8, 2010
  42. 42. Rocket science Monolithic Computing Hot spare “Just in case” 36 Monday, March 8, 2010
  43. 43. Distributed Meme: Expect, tolerate and design for failure MTFB: O(Years) 37 Monday, March 8, 2010
  44. 44. Distributed Meme: Expect, tolerate and design for failure MTFB: O(Years) 37 Monday, March 8, 2010
  45. 45. Distributed Meme: Expect, tolerate and design for failure MTFB: O(Years) MTFB: O(Months) 37 Monday, March 8, 2010
  46. 46. Distributed Meme: Expect, tolerate and design for failure MTFB: O(Years) MTFB: O(Months) 37 Monday, March 8, 2010
  47. 47. Distributed Meme: Expect, tolerate and design for failure MTFB: O(Years) MTFB: O(Months) MTFB: O(Minutes/Seconds) 37 Monday, March 8, 2010
  48. 48. Monolithic computing stop watch small stop watch large 38 Monday, March 8, 2010
  49. 49. Distributed web hosting platform 39 Monday, March 8, 2010
  50. 50. Distributed web hosting platform X 39 Monday, March 8, 2010
  51. 51. Distributed web hosting platform Retry X 39 Monday, March 8, 2010
  52. 52. Vertical / horizontal scalability 40 Monday, March 8, 2010
  53. 53. Vertical / horizontal scalability 40 Monday, March 8, 2010
  54. 54. Vertical / horizontal scalability 40 Monday, March 8, 2010
  55. 55. Vertical / horizontal scalability 40 Monday, March 8, 2010
  56. 56. Vertical / horizontal scalability 40 Monday, March 8, 2010
  57. 57. Vertical / horizontal scalability 40 Monday, March 8, 2010
  58. 58. Vertical scalability 41 Monday, March 8, 2010
  59. 59. Vertical scalability Horizontal scalability 41 Monday, March 8, 2010
  60. 60. Vertical scalability Horizontal scalability 41 Monday, March 8, 2010
  61. 61. Tolerate, plan for and expect small failures Monday, March 8, 2010
  62. 62. Tolerate, plan for and expect small failures Monday, March 8, 2010
  63. 63. Tolerate, plan for and expect small failures Monday, March 8, 2010
  64. 64. Tolerate, plan for and expect small failures Monday, March 8, 2010
  65. 65. Tolerate, plan for and expect small failures Monday, March 8, 2010
  66. 66. Tolerate, plan for and expect small failures Monday, March 8, 2010
  67. 67. Easy to deploy & scale 1 2 http://www.yourdomain.com/ 43 Monday, March 8, 2010
  68. 68. Thinking about scalability Just a few users.... ....the tools, platform and design don't matter too much 44 Monday, March 8, 2010
  69. 69. Thinking about scalability Lots and lots of users... ....you must design for scalability 45 Monday, March 8, 2010
  70. 70. But the problem is... When? http://digg.com/ http://slashdot.org/ 46 Monday, March 8, 2010
  71. 71. The real problem is... When it happens. Can you afford to put everything else on hold? 47 Monday, March 8, 2010
  72. 72. Google App Engine Leveraging Google's platform to better serve your customers 48 Monday, March 8, 2010
  73. 73. Distributed web hosting platform 49 Monday, March 8, 2010
  74. 74. Distributed web hosting platform - Great for web apps • Request based, data backed - Parallel processing - Scales automatically - Available globally - Configuration free - Built-in DoS protections 50 Monday, March 8, 2010
  75. 75. Distributed datastore 51 Monday, March 8, 2010
  76. 76. Distributed datastore 52 Monday, March 8, 2010
  77. 77. Distributed datastore - Arbitrary horizontal scaling 52 Monday, March 8, 2010
  78. 78. Distributed datastore - Arbitrary horizontal scaling - Parallel processing 52 Monday, March 8, 2010
  79. 79. Distributed datastore - Arbitrary horizontal scaling - Parallel processing - Scales to 'Internet scale' 52 Monday, March 8, 2010
  80. 80. Distributed datastore - Arbitrary horizontal scaling - Parallel processing - Scales to 'Internet scale' - Predictable query performance 52 Monday, March 8, 2010
  81. 81. Distributed datastore - Arbitrary horizontal scaling - Parallel processing - Scales to 'Internet scale' - Predictable query performance - No deadlocks 52 Monday, March 8, 2010
  82. 82. Distributed datastore - Arbitrary horizontal scaling - Parallel processing - Scales to 'Internet scale' - Predictable query performance - No deadlocks - No global schema 52 Monday, March 8, 2010
  83. 83. Distributed memcache Memcache 53 Monday, March 8, 2010
  84. 84. Distributed Meme: Divide & Conquer Specialized services Memcache Datastore URL Fetch Mail XMPP Task Queue Images Blobstore User Service 54 Monday, March 8, 2010
  85. 85. Distributed Meme: Sharding Single Sharded Counter Counter Counter Counter 55 Monday, March 8, 2010
  86. 86. Distributed Meme: Sharding Single Sharded Counter Counter Counter Counter 55 Monday, March 8, 2010
  87. 87. Distributed Meme: Sharding Single Sharded Counter Counter Counter Counter 55 Monday, March 8, 2010
  88. 88. Language runtimes Duke, the Java mascot Copyright © Sun Microsystems Inc., all rights reserved. 56 Monday, March 8, 2010
  89. 89. JVM languages - Scala - JRuby (Ruby) - Groovy - Quercus (PHP) - Rhino (JavaScript) - Jython (Python) 57 Monday, March 8, 2010
  90. 90. Ensuring portability 58 Monday, March 8, 2010
  91. 91. Ensuring portability 58 Monday, March 8, 2010
  92. 92. Complete Java development stack 59 Monday, March 8, 2010
  93. 93. Google Plugin for Eclipse 60 Monday, March 8, 2010
  94. 94. Google Plugin for Eclipse 61 Monday, March 8, 2010
  95. 95. Google Web Toolkit (GWT) 62 Monday, March 8, 2010
  96. 96. Google Web Toolkit (GWT) IE Firefo x i Safar e m Chro Opera 63 Monday, March 8, 2010
  97. 97. GWT Debugging in the browser 64 Monday, March 8, 2010
  98. 98. Quotes to remember “The fastest code is that which does not run.” Joel Webber GWT co-creator 65 Monday, March 8, 2010
  99. 99. Google Apps + your apps Your custom Our Google Apps applications Google's scalable serving architecture 66 Monday, March 8, 2010
  100. 100. Google Apps integration http://appid.appspot.com/ http://yourapp.yourdomain.com/ 67 Monday, March 8, 2010
  101. 101. 2. Log in to your Google Apps domain as an admin 68 Monday, March 8, 2010
  102. 102. 3. Accept terms and specify web address 69 Monday, March 8, 2010
  103. 103. 4. The new feature is available in your control panel 70 Monday, March 8, 2010
  104. 104. 5. Additional configuration is available for each feature 71 Monday, March 8, 2010
  105. 105. Google Apps + App Engine 72 Monday, March 8, 2010
  106. 106. Federate your on-premise data 73 Monday, March 8, 2010
  107. 107. Secure Data Connector (SDC) 74 Monday, March 8, 2010
  108. 108. Secure Data Connector 75 Monday, March 8, 2010
  109. 109. Secure Data Connector and 50+ more... 76 Monday, March 8, 2010
  110. 110. Using Secure Data Connector 77 Monday, March 8, 2010
  111. 111. Using Secure Data Connector Installation - Determine access rules - Configure and install SDC 77 Monday, March 8, 2010
  112. 112. Using Secure Data Connector Installation - Determine access rules - Configure and install SDC Getting ready to serve - SDC opens SSL tunnel 77 Monday, March 8, 2010
  113. 113. Using Secure Data Connector Installation - Determine access rules - Configure and install SDC Getting ready to serve - SDC opens SSL tunnel Serving - User request sent to App Engine - User authenticated - App makes request through tunnel - SDC performs access checks - Results returned 77 Monday, March 8, 2010
  114. 114. Database export / import 78 Monday, March 8, 2010
  115. 115. Cron support - clean-up DB - prepare r eports - batch ma il - perform calcs - call mom 79 Monday, March 8, 2010
  116. 116. Cron support <?xml version="1.0" encoding="UTF-8"?> <cronentries>   <cron>     <url>/recache</url>     <description>Repopulate the cache every 2 minutes</description>     <schedule>every 2 minutes</schedule>   </cron>   <cron>     <url>/weeklyreport</url>     <description>Mail out a weekly report</description>     <schedule>every monday of month 08:30</schedule>     <timezone>America/New_York</timezone>   </cron> </cronentries> ("every"|ordinal) (days) "of" (monthspec) (time) every 5 minutes every 12 hours 2nd,third mon,wed,thu of march 17:00 every monday of month 09:00 1st monday of sep,oct,nov 17:00 80 Monday, March 8, 2010
  117. 117. Distributed Meme: Divide & Conquer Task Queues Task 12 Task 67 Task 42 Task 2 Task 91 Task 16 Tas Offline Processing on App Engine: A Look Ahead Brett Slatkin May, 2009 81 Monday, March 8, 2010
  118. 118. Tasks Queues - Motivation • Why do background processing? o Do work continuously without user requests o Incrementally process data, compute results o Smooth out load patterns, lower user latency • Simple idea in general: 1.Describe the work you want to do now 2.Save the description somewhere 3.Have something else execute the work later • Work executed in the order received (best-effort FIFO) • If execution fails, work will be retried until successful Task 12 Task 67 Task 42 Task 2 Task 91 Task 16 Tas Monday, March 8, 2010
  119. 119. Task Queue Benefits • Benefits o Asynchronous Why do work now when we can do it later? o Low-latency (for users) Tasks are light-weight; ~3x faster than Datastore o Reliable Once written, a task will eventually complete o Scalable Storage of new tasks has no contention Parallelizable with multiple workers • Many features can extend this basic concept Task 12 Task 67 Task 42 Task 2 Task 91 Task 16 Tas Monday, March 8, 2010
  120. 120. How do traditional task queues work? Monday, March 8, 2010
  121. 121. How do traditional task queues work? (2) • Polling has problems: o Worker sits in a loop polling the front of the queue Not event driven; wasted work o Workers stay resident when there's no work to do Wastes machine resources o Fixed number of workers Admins must manually add more workers to keep up or queue will grow without bounds • Limited optimization possible o Many systems fake a polling interface with something event-driven under the hood o Long-lived, hanging connections Monday, March 8, 2010
  122. 122. How does our Task Queue API work? (2) Monday, March 8, 2010
  123. 123. How does our Task Queue API work? (3) Monday, March 8, 2010
  124. 124. Your application's health 88 Monday, March 8, 2010
  125. 125. App Engine's health history 89 Monday, March 8, 2010
  126. 126. Scalable serving architecture creative commons licensed photograph from cote 90 Monday, March 8, 2010
  127. 127. Distributed frontends 91 Monday, March 8, 2010
  128. 128. Distributed frontends - Edge Caching 91 Monday, March 8, 2010
  129. 129. Distributed frontends - Edge Caching - Priority routing 91 Monday, March 8, 2010
  130. 130. Distributed frontends - Edge Caching - Priority routing - Geographically close connection endpoint 91 Monday, March 8, 2010
  131. 131. Life of a request App App's User Engine Runtime Frontend 92 Monday, March 8, 2010
  132. 132. Life of a request App App's User Engine Runtime Frontend 93 Monday, March 8, 2010
  133. 133. Scales dynamically / automatically User User App App's User Engine Runtime Frontend User User 94 Monday, March 8, 2010
  134. 134. Scales dynamically / automatically User App's User Runtime App App's User Engine Runtime Frontend App's Runtime User User 95 Monday, March 8, 2010
  135. 135. Multiple versions http://fredsa.appspot.com/ Version 1.1 App User Datastore Engine Frontend Version 2.1 http://2.1.fredsa.appspot.com/ 96 Monday, March 8, 2010
  136. 136. Distributed datastore http://labs.google.com/papers/bigtable.html 97 Monday, March 8, 2010
  137. 137. Bigtable : A distributed, sharded, sorted array  Row key    Row data 98 Monday, March 8, 2010
  138. 138. Bigtable : A distributed, sharded, sorted array  Row key    Row data Shard 1 Shard 2 . . Shard n 99 Monday, March 8, 2010
  139. 139. Datastore design - Distributed - Bigtable + entity groups - ACID transactions - Optimistic concurrency - Entities + indexes - Protobuf encoded entities 100 Monday, March 8, 2010
  140. 140. Datastore properties - Core value types - List properties - Text & binary blobs - Reference 101 Monday, March 8, 2010
  141. 141. What's different? 102 Monday, March 8, 2010
  142. 142. Datastore - what's new - Distributed - Scales to 'internet scale' - No deadlocks - Predictable query performance 103 Monday, March 8, 2010
  143. 143. Datastore - what's different - No inner/outer/natural joins - Dense index scans - Per entity metadata - Soft schema - No more DDL 104 Monday, March 8, 2010
  144. 144. Datastore - what's familiar - Indexes - ACID transactions - GQL SELECT * FROM Story WHERE title = 'App Engine Launch' AND author = :current_user AND rating >= 10 ORDER BY rating, created DESC 105 Monday, March 8, 2010
  145. 145. The new ACID Atomic Associative Consistent Commutative Isolated Idempotent Durable Distributed 106 Monday, March 8, 2010
  146. 146. Entities 107 Monday, March 8, 2010
  147. 147. Entity groups 108 Monday, March 8, 2010
  148. 148. Amortizing costs Count(COUNT) Transaction SUM(Order Amount) Create Update Delete Customer Order 109 Monday, March 8, 2010
  149. 149. An evolving platform 110 Monday, March 8, 2010
  150. 150. 23 months in review Apr 2008 Python launch May 2008 Memcache, Images API Jul 2008 Logs export Aug 2008 Batch write/delete Oct 2008 HTTPS support Dec 2008 Status dashboard, quota details Feb 2009 Billing, larger files Apr 2009 Java launch, DB import, cron support, SDC May 2009 Key-only queries Jun 2009 Task queues Aug 2009 Kindless queries Sep 2009 XMPP Oct 2009 Incoming email Dec 2009 Blobstore Feb 2010 Datastore cursors, Async Urlfetch 111 Monday, March 8, 2010
  151. 151. App Engine Roadmap - Support for mapping operations across datasets - Alerting system for exceptions in your application - Datastore dump and restore facility 112 Monday, March 8, 2010
  152. 152. Wrap up 113 Monday, March 8, 2010
  153. 153. Always free to get started ~5M pageviews/month • 6.5 CPU hrs/day • 1 GB storage • 650K URL Fetch calls/day • 2,000 recipients emailed • 1 GB/day bandwidth • 100,000 tasks enqueued • 650K XMPP messages/day 114 Monday, March 8, 2010
  154. 154. Purchase additional resources * * free monthly quota of ~5 million page views still in full effect 115 Monday, March 8, 2010
  155. 155. Thank you Read more http://code.google.com/appengine/ Contact info Patrick Chanezon Developer Advocate chanezon@google.com http://twitter.com/chanezon Questions ? 116 Monday, March 8, 2010
  156. 156. Thanks To Alon Levi, Fred Sauer, Brett Slatkin and others for their slides Monday, March 8, 2010

×