Google App Engine -  Devfest India 2010
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Google App Engine - Devfest India 2010

on

  • 6,060 views

Intro to Google App Engine, Google's cloud computing platform.

Intro to Google App Engine, Google's cloud computing platform.
Presented at Devfest India 2010 Hyderabad andPune.

Statistics

Views

Total Views
6,060
Views on SlideShare
5,872
Embed Views
188

Actions

Likes
9
Downloads
234
Comments
1

7 Embeds 188

http://jujo00obo2o234ungd3t8qjfcjrs3o6k-a-sites-opensocial.googleusercontent.com 80
https://jujo00obo2o234ungd3t8qjfcjrs3o6k-a-sites-opensocial.googleusercontent.com 73
http://www.slideshare.net 21
http://www.techgig.com 6
http://thetechnicalweb.blogspot.com 4
http://www.mefeedia.com 3
http://thetechnicalweb.blogspot.in 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • Thanks, Patrick for a very good presentation. With support for more languages coming on, surely it opens up the platform. Out of curiosity, though, what was the driver for moving away, if I may say so, from Python to Java or a JVM-based platform?

    - Saurabh
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Google App Engine - Devfest India 2010 Presentation Transcript

  • 1. Google App Engine Feb – March, 2010 Patrick Chanezon Developer Advocate Google Developer Relations chanezon@google.com Google Confidential and Proprietary 1 Monday, March 8, 2010
  • 2. Agenda - App Engine introduction - Why build it? - App Engine tour - What's different? - Wrap up - Questions 2 Monday, March 8, 2010
  • 3. 4. The spirit is unmoving, one, swifter than the mind. The senses do not reach It as It is ever ahead of them. Though Itself standing still It outstrips those who run. In It the all-pervading air supports the activities of beings Isha Upanishad 3 Monday, March 8, 2010
  • 4. Vayu computing? Public domain image from http://commons.wikimedia.org/wiki/File:Vayu.jpg 4 Monday, March 8, 2010
  • 5. Vayu Computing? 7−10 Then they said to Vayu (Air): "O Vayu! Find out who this great Spirit is." "Yes," he said, and hastened to It. Brahman asked him: "Who are you?" He replied "I am known as Vayu; I am also called Matarisva." Brahman said: "What power is in you, who are so well known?" Vayu replied: "I can carry off all−whatever there is on earth." Brahman put a straw before him and said: "Carry this." He rushed toward it with all his ardour but could not move it. Then he returned from the Spirit and said to the gods: "I could not find out who this Spirit is," Kena Upanishad 5 Monday, March 8, 2010
  • 6. What is cloud computing? SaaS PaaS IaaS Postage Place Here 6 Monday, March 8, 2010
  • 7. IaaS value proposition… 7 Monday, March 8, 2010
  • 8. IaaS value proposition… 7 Monday, March 8, 2010
  • 9. IaaS value proposition… 7 Monday, March 8, 2010
  • 10. IaaS value proposition… 7 Monday, March 8, 2010
  • 11. Google App Engine “We wear pagers so you don’t have to” pager 8 Monday, March 8, 2010
  • 12. Google App Engine - Easy to build - Easy to maintain - Easy to scale 9 Monday, March 8, 2010
  • 13. By the numbers - Built 100K apps - Maintained by 250K developers - Scaled to 250M pageviews daily semi-transparent collage of apps 10 Monday, March 8, 2010
  • 14. gigya Socialize 11 Monday, March 8, 2010
  • 15. gigya Socialize - traffic 12 Monday, March 8, 2010
  • 16. gigya Socialize - traffic 12 Monday, March 8, 2010
  • 17. Gigya Socialize - traffic 13 Monday, March 8, 2010
  • 18. App Engine 14 Monday, March 8, 2010
  • 19. App Engine 15 Monday, March 8, 2010
  • 20. Time.com 16 Monday, March 8, 2010
  • 21. Socialwok 17 Monday, March 8, 2010
  • 22. Social networking at scale 18 Monday, March 8, 2010
  • 23. Social networking at scale >40M Users 18 Monday, March 8, 2010
  • 24. PubSubHubbub http://code.google.com/p/pubsubhubbub/ The future is distributed: There will be big hubs, many small hubs, and tons of publishers and subscribers. Publishers, subscribers, and hubs may play multiple roles. 19 Monday, March 8, 2010
  • 25. Hell's Kitchen on facebook 20 Monday, March 8, 2010
  • 26. Handling peak traffic: mental_floss 21 Monday, March 8, 2010
  • 27. Elastic / redundant capacity 22 Monday, March 8, 2010
  • 28. Short term events: earthhour.org 23 Monday, March 8, 2010
  • 29. http://www.allforgood.org/ 24 Monday, March 8, 2010
  • 30. http://www.snapabug.com/ 25 Monday, March 8, 2010
  • 31. 26 Monday, March 8, 2010
  • 32. Japan: Q&A for our Future 27 Monday, March 8, 2010
  • 33. http://www.google.com/tipjar 28 Monday, March 8, 2010
  • 34. Chillingo Crystal Gaming meets Social Zombie Dash Angry Birds LITE underground Meltdown Cogs Mission Deep Sea Speed Forge Guerilla Bob Ravensword: Angry Birds Extreme The Fallen King Monday, March 8, 2010
  • 35. Why build it? Wrench 30 Monday, March 8, 2010
  • 36. It's just too difficult 31 Monday, March 8, 2010
  • 37. Hosting means hidden costs • Idle capacity • Software patches & upgrades • License fees • Lots of maintenance • Traffic & utilization forecasting • Upgrades 32 Monday, March 8, 2010
  • 38. Cloud development in a box • SDK & “The Cloud” • Hardware • Networking • Operating system • Application runtime Java, Python o • Static file serving • Services • Fault tolerance • Load balancing 33 Monday, March 8, 2010
  • 39. Distributed Cloud Computing Reliability and scalability Monday, March 8, 2010
  • 40. Know your Distributed Memes - Divide & conquer - Sharding - Tolerate / expect failure - Idempotency - Just scale it up 35 Monday, March 8, 2010
  • 41. Rocket science Monolithic Computing 36 Monday, March 8, 2010
  • 42. Rocket science Monolithic Computing Hot spare “Just in case” 36 Monday, March 8, 2010
  • 43. Distributed Meme: Expect, tolerate and design for failure MTFB: O(Years) 37 Monday, March 8, 2010
  • 44. Distributed Meme: Expect, tolerate and design for failure MTFB: O(Years) 37 Monday, March 8, 2010
  • 45. Distributed Meme: Expect, tolerate and design for failure MTFB: O(Years) MTFB: O(Months) 37 Monday, March 8, 2010
  • 46. Distributed Meme: Expect, tolerate and design for failure MTFB: O(Years) MTFB: O(Months) 37 Monday, March 8, 2010
  • 47. Distributed Meme: Expect, tolerate and design for failure MTFB: O(Years) MTFB: O(Months) MTFB: O(Minutes/Seconds) 37 Monday, March 8, 2010
  • 48. Monolithic computing stop watch small stop watch large 38 Monday, March 8, 2010
  • 49. Distributed web hosting platform 39 Monday, March 8, 2010
  • 50. Distributed web hosting platform X 39 Monday, March 8, 2010
  • 51. Distributed web hosting platform Retry X 39 Monday, March 8, 2010
  • 52. Vertical / horizontal scalability 40 Monday, March 8, 2010
  • 53. Vertical / horizontal scalability 40 Monday, March 8, 2010
  • 54. Vertical / horizontal scalability 40 Monday, March 8, 2010
  • 55. Vertical / horizontal scalability 40 Monday, March 8, 2010
  • 56. Vertical / horizontal scalability 40 Monday, March 8, 2010
  • 57. Vertical / horizontal scalability 40 Monday, March 8, 2010
  • 58. Vertical scalability 41 Monday, March 8, 2010
  • 59. Vertical scalability Horizontal scalability 41 Monday, March 8, 2010
  • 60. Vertical scalability Horizontal scalability 41 Monday, March 8, 2010
  • 61. Tolerate, plan for and expect small failures Monday, March 8, 2010
  • 62. Tolerate, plan for and expect small failures Monday, March 8, 2010
  • 63. Tolerate, plan for and expect small failures Monday, March 8, 2010
  • 64. Tolerate, plan for and expect small failures Monday, March 8, 2010
  • 65. Tolerate, plan for and expect small failures Monday, March 8, 2010
  • 66. Tolerate, plan for and expect small failures Monday, March 8, 2010
  • 67. Easy to deploy & scale 1 2 http://www.yourdomain.com/ 43 Monday, March 8, 2010
  • 68. Thinking about scalability Just a few users.... ....the tools, platform and design don't matter too much 44 Monday, March 8, 2010
  • 69. Thinking about scalability Lots and lots of users... ....you must design for scalability 45 Monday, March 8, 2010
  • 70. But the problem is... When? http://digg.com/ http://slashdot.org/ 46 Monday, March 8, 2010
  • 71. The real problem is... When it happens. Can you afford to put everything else on hold? 47 Monday, March 8, 2010
  • 72. Google App Engine Leveraging Google's platform to better serve your customers 48 Monday, March 8, 2010
  • 73. Distributed web hosting platform 49 Monday, March 8, 2010
  • 74. Distributed web hosting platform - Great for web apps • Request based, data backed - Parallel processing - Scales automatically - Available globally - Configuration free - Built-in DoS protections 50 Monday, March 8, 2010
  • 75. Distributed datastore 51 Monday, March 8, 2010
  • 76. Distributed datastore 52 Monday, March 8, 2010
  • 77. Distributed datastore - Arbitrary horizontal scaling 52 Monday, March 8, 2010
  • 78. Distributed datastore - Arbitrary horizontal scaling - Parallel processing 52 Monday, March 8, 2010
  • 79. Distributed datastore - Arbitrary horizontal scaling - Parallel processing - Scales to 'Internet scale' 52 Monday, March 8, 2010
  • 80. Distributed datastore - Arbitrary horizontal scaling - Parallel processing - Scales to 'Internet scale' - Predictable query performance 52 Monday, March 8, 2010
  • 81. Distributed datastore - Arbitrary horizontal scaling - Parallel processing - Scales to 'Internet scale' - Predictable query performance - No deadlocks 52 Monday, March 8, 2010
  • 82. Distributed datastore - Arbitrary horizontal scaling - Parallel processing - Scales to 'Internet scale' - Predictable query performance - No deadlocks - No global schema 52 Monday, March 8, 2010
  • 83. Distributed memcache Memcache 53 Monday, March 8, 2010
  • 84. Distributed Meme: Divide & Conquer Specialized services Memcache Datastore URL Fetch Mail XMPP Task Queue Images Blobstore User Service 54 Monday, March 8, 2010
  • 85. Distributed Meme: Sharding Single Sharded Counter Counter Counter Counter 55 Monday, March 8, 2010
  • 86. Distributed Meme: Sharding Single Sharded Counter Counter Counter Counter 55 Monday, March 8, 2010
  • 87. Distributed Meme: Sharding Single Sharded Counter Counter Counter Counter 55 Monday, March 8, 2010
  • 88. Language runtimes Duke, the Java mascot Copyright © Sun Microsystems Inc., all rights reserved. 56 Monday, March 8, 2010
  • 89. JVM languages - Scala - JRuby (Ruby) - Groovy - Quercus (PHP) - Rhino (JavaScript) - Jython (Python) 57 Monday, March 8, 2010
  • 90. Ensuring portability 58 Monday, March 8, 2010
  • 91. Ensuring portability 58 Monday, March 8, 2010
  • 92. Complete Java development stack 59 Monday, March 8, 2010
  • 93. Google Plugin for Eclipse 60 Monday, March 8, 2010
  • 94. Google Plugin for Eclipse 61 Monday, March 8, 2010
  • 95. Google Web Toolkit (GWT) 62 Monday, March 8, 2010
  • 96. Google Web Toolkit (GWT) IE Firefo x i Safar e m Chro Opera 63 Monday, March 8, 2010
  • 97. GWT Debugging in the browser 64 Monday, March 8, 2010
  • 98. Quotes to remember “The fastest code is that which does not run.” Joel Webber GWT co-creator 65 Monday, March 8, 2010
  • 99. Google Apps + your apps Your custom Our Google Apps applications Google's scalable serving architecture 66 Monday, March 8, 2010
  • 100. Google Apps integration http://appid.appspot.com/ http://yourapp.yourdomain.com/ 67 Monday, March 8, 2010
  • 101. 2. Log in to your Google Apps domain as an admin 68 Monday, March 8, 2010
  • 102. 3. Accept terms and specify web address 69 Monday, March 8, 2010
  • 103. 4. The new feature is available in your control panel 70 Monday, March 8, 2010
  • 104. 5. Additional configuration is available for each feature 71 Monday, March 8, 2010
  • 105. Google Apps + App Engine 72 Monday, March 8, 2010
  • 106. Federate your on-premise data 73 Monday, March 8, 2010
  • 107. Secure Data Connector (SDC) 74 Monday, March 8, 2010
  • 108. Secure Data Connector 75 Monday, March 8, 2010
  • 109. Secure Data Connector and 50+ more... 76 Monday, March 8, 2010
  • 110. Using Secure Data Connector 77 Monday, March 8, 2010
  • 111. Using Secure Data Connector Installation - Determine access rules - Configure and install SDC 77 Monday, March 8, 2010
  • 112. Using Secure Data Connector Installation - Determine access rules - Configure and install SDC Getting ready to serve - SDC opens SSL tunnel 77 Monday, March 8, 2010
  • 113. Using Secure Data Connector Installation - Determine access rules - Configure and install SDC Getting ready to serve - SDC opens SSL tunnel Serving - User request sent to App Engine - User authenticated - App makes request through tunnel - SDC performs access checks - Results returned 77 Monday, March 8, 2010
  • 114. Database export / import 78 Monday, March 8, 2010
  • 115. Cron support - clean-up DB - prepare r eports - batch ma il - perform calcs - call mom 79 Monday, March 8, 2010
  • 116. Cron support <?xml version="1.0" encoding="UTF-8"?> <cronentries>   <cron>     <url>/recache</url>     <description>Repopulate the cache every 2 minutes</description>     <schedule>every 2 minutes</schedule>   </cron>   <cron>     <url>/weeklyreport</url>     <description>Mail out a weekly report</description>     <schedule>every monday of month 08:30</schedule>     <timezone>America/New_York</timezone>   </cron> </cronentries> ("every"|ordinal) (days) "of" (monthspec) (time) every 5 minutes every 12 hours 2nd,third mon,wed,thu of march 17:00 every monday of month 09:00 1st monday of sep,oct,nov 17:00 80 Monday, March 8, 2010
  • 117. Distributed Meme: Divide & Conquer Task Queues Task 12 Task 67 Task 42 Task 2 Task 91 Task 16 Tas Offline Processing on App Engine: A Look Ahead Brett Slatkin May, 2009 81 Monday, March 8, 2010
  • 118. Tasks Queues - Motivation • Why do background processing? o Do work continuously without user requests o Incrementally process data, compute results o Smooth out load patterns, lower user latency • Simple idea in general: 1.Describe the work you want to do now 2.Save the description somewhere 3.Have something else execute the work later • Work executed in the order received (best-effort FIFO) • If execution fails, work will be retried until successful Task 12 Task 67 Task 42 Task 2 Task 91 Task 16 Tas Monday, March 8, 2010
  • 119. Task Queue Benefits • Benefits o Asynchronous Why do work now when we can do it later? o Low-latency (for users) Tasks are light-weight; ~3x faster than Datastore o Reliable Once written, a task will eventually complete o Scalable Storage of new tasks has no contention Parallelizable with multiple workers • Many features can extend this basic concept Task 12 Task 67 Task 42 Task 2 Task 91 Task 16 Tas Monday, March 8, 2010
  • 120. How do traditional task queues work? Monday, March 8, 2010
  • 121. How do traditional task queues work? (2) • Polling has problems: o Worker sits in a loop polling the front of the queue Not event driven; wasted work o Workers stay resident when there's no work to do Wastes machine resources o Fixed number of workers Admins must manually add more workers to keep up or queue will grow without bounds • Limited optimization possible o Many systems fake a polling interface with something event-driven under the hood o Long-lived, hanging connections Monday, March 8, 2010
  • 122. How does our Task Queue API work? (2) Monday, March 8, 2010
  • 123. How does our Task Queue API work? (3) Monday, March 8, 2010
  • 124. Your application's health 88 Monday, March 8, 2010
  • 125. App Engine's health history 89 Monday, March 8, 2010
  • 126. Scalable serving architecture creative commons licensed photograph from cote 90 Monday, March 8, 2010
  • 127. Distributed frontends 91 Monday, March 8, 2010
  • 128. Distributed frontends - Edge Caching 91 Monday, March 8, 2010
  • 129. Distributed frontends - Edge Caching - Priority routing 91 Monday, March 8, 2010
  • 130. Distributed frontends - Edge Caching - Priority routing - Geographically close connection endpoint 91 Monday, March 8, 2010
  • 131. Life of a request App App's User Engine Runtime Frontend 92 Monday, March 8, 2010
  • 132. Life of a request App App's User Engine Runtime Frontend 93 Monday, March 8, 2010
  • 133. Scales dynamically / automatically User User App App's User Engine Runtime Frontend User User 94 Monday, March 8, 2010
  • 134. Scales dynamically / automatically User App's User Runtime App App's User Engine Runtime Frontend App's Runtime User User 95 Monday, March 8, 2010
  • 135. Multiple versions http://fredsa.appspot.com/ Version 1.1 App User Datastore Engine Frontend Version 2.1 http://2.1.fredsa.appspot.com/ 96 Monday, March 8, 2010
  • 136. Distributed datastore http://labs.google.com/papers/bigtable.html 97 Monday, March 8, 2010
  • 137. Bigtable : A distributed, sharded, sorted array  Row key    Row data 98 Monday, March 8, 2010
  • 138. Bigtable : A distributed, sharded, sorted array  Row key    Row data Shard 1 Shard 2 . . Shard n 99 Monday, March 8, 2010
  • 139. Datastore design - Distributed - Bigtable + entity groups - ACID transactions - Optimistic concurrency - Entities + indexes - Protobuf encoded entities 100 Monday, March 8, 2010
  • 140. Datastore properties - Core value types - List properties - Text & binary blobs - Reference 101 Monday, March 8, 2010
  • 141. What's different? 102 Monday, March 8, 2010
  • 142. Datastore - what's new - Distributed - Scales to 'internet scale' - No deadlocks - Predictable query performance 103 Monday, March 8, 2010
  • 143. Datastore - what's different - No inner/outer/natural joins - Dense index scans - Per entity metadata - Soft schema - No more DDL 104 Monday, March 8, 2010
  • 144. Datastore - what's familiar - Indexes - ACID transactions - GQL SELECT * FROM Story WHERE title = 'App Engine Launch' AND author = :current_user AND rating >= 10 ORDER BY rating, created DESC 105 Monday, March 8, 2010
  • 145. The new ACID Atomic Associative Consistent Commutative Isolated Idempotent Durable Distributed 106 Monday, March 8, 2010
  • 146. Entities 107 Monday, March 8, 2010
  • 147. Entity groups 108 Monday, March 8, 2010
  • 148. Amortizing costs Count(COUNT) Transaction SUM(Order Amount) Create Update Delete Customer Order 109 Monday, March 8, 2010
  • 149. An evolving platform 110 Monday, March 8, 2010
  • 150. 23 months in review Apr 2008 Python launch May 2008 Memcache, Images API Jul 2008 Logs export Aug 2008 Batch write/delete Oct 2008 HTTPS support Dec 2008 Status dashboard, quota details Feb 2009 Billing, larger files Apr 2009 Java launch, DB import, cron support, SDC May 2009 Key-only queries Jun 2009 Task queues Aug 2009 Kindless queries Sep 2009 XMPP Oct 2009 Incoming email Dec 2009 Blobstore Feb 2010 Datastore cursors, Async Urlfetch 111 Monday, March 8, 2010
  • 151. App Engine Roadmap - Support for mapping operations across datasets - Alerting system for exceptions in your application - Datastore dump and restore facility 112 Monday, March 8, 2010
  • 152. Wrap up 113 Monday, March 8, 2010
  • 153. Always free to get started ~5M pageviews/month • 6.5 CPU hrs/day • 1 GB storage • 650K URL Fetch calls/day • 2,000 recipients emailed • 1 GB/day bandwidth • 100,000 tasks enqueued • 650K XMPP messages/day 114 Monday, March 8, 2010
  • 154. Purchase additional resources * * free monthly quota of ~5 million page views still in full effect 115 Monday, March 8, 2010
  • 155. Thank you Read more http://code.google.com/appengine/ Contact info Patrick Chanezon Developer Advocate chanezon@google.com http://twitter.com/chanezon Questions ? 116 Monday, March 8, 2010
  • 156. Thanks To Alon Levi, Fred Sauer, Brett Slatkin and others for their slides Monday, March 8, 2010