Google App Engine - Devfest India 2010

Google App Engine Feb – March, 2010 Patrick Chanezon Developer Advocate Google Developer Relations chanezon@google.com Google Confidential and Proprietary 1 Monday, March 8, 2010

Agenda - App Engine introduction - Why build it? - App Engine tour - What's different? - Wrap up - Questions 2 Monday, March 8, 2010

4. The spirit is unmoving, one, swifter than the mind. The senses do not reach It as It is ever ahead of them. Though Itself standing still It outstrips those who run. In It the all-pervading air supports the activities of beings Isha Upanishad 3 Monday, March 8, 2010

Vayu computing? Public domain image from http://commons.wikimedia.org/wiki/File:Vayu.jpg 4 Monday, March 8, 2010

Vayu Computing? 7−10 Then they said to Vayu (Air): "O Vayu! Find out who this great Spirit is." "Yes," he said, and hastened to It. Brahman asked him: "Who are you?" He replied "I am known as Vayu; I am also called Matarisva." Brahman said: "What power is in you, who are so well known?" Vayu replied: "I can carry off all−whatever there is on earth." Brahman put a straw before him and said: "Carry this." He rushed toward it with all his ardour but could not move it. Then he returned from the Spirit and said to the gods: "I could not find out who this Spirit is," Kena Upanishad 5 Monday, March 8, 2010

What is cloud computing? SaaS PaaS IaaS Postage Place Here 6 Monday, March 8, 2010

IaaS value proposition… 7 Monday, March 8, 2010

Google App Engine “We wear pagers so you don’t have to” pager 8 Monday, March 8, 2010

Google App Engine - Easy to build - Easy to maintain - Easy to scale 9 Monday, March 8, 2010

By the numbers - Built 100K apps - Maintained by 250K developers - Scaled to 250M pageviews daily semi-transparent collage of apps 10 Monday, March 8, 2010

gigya Socialize 11 Monday, March 8, 2010

gigya Socialize - traffic 12 Monday, March 8, 2010

Gigya Socialize - traffic 13 Monday, March 8, 2010

App Engine 14 Monday, March 8, 2010

App Engine 15 Monday, March 8, 2010

Time.com 16 Monday, March 8, 2010

Socialwok 17 Monday, March 8, 2010

Social networking at scale 18 Monday, March 8, 2010

Social networking at scale >40M Users 18 Monday, March 8, 2010

PubSubHubbub http://code.google.com/p/pubsubhubbub/ The future is distributed: There will be big hubs, many small hubs, and tons of publishers and subscribers. Publishers, subscribers, and hubs may play multiple roles. 19 Monday, March 8, 2010

Hell's Kitchen on facebook 20 Monday, March 8, 2010

Handling peak traffic: mental_floss 21 Monday, March 8, 2010

Elastic / redundant capacity 22 Monday, March 8, 2010

Short term events: earthhour.org 23 Monday, March 8, 2010

http://www.allforgood.org/ 24 Monday, March 8, 2010

http://www.snapabug.com/ 25 Monday, March 8, 2010

26 Monday, March 8, 2010

Japan: Q&A for our Future 27 Monday, March 8, 2010

http://www.google.com/tipjar 28 Monday, March 8, 2010

Chillingo Crystal Gaming meets Social Zombie Dash Angry Birds LITE underground Meltdown Cogs Mission Deep Sea Speed Forge Guerilla Bob Ravensword: Angry Birds Extreme The Fallen King Monday, March 8, 2010

Why build it? Wrench 30 Monday, March 8, 2010

It's just too difficult 31 Monday, March 8, 2010

Hosting means hidden costs • Idle capacity • Software patches & upgrades • License fees • Lots of maintenance • Traffic & utilization forecasting • Upgrades 32 Monday, March 8, 2010

Cloud development in a box • SDK & “The Cloud” • Hardware • Networking • Operating system • Application runtime Java, Python o • Static file serving • Services • Fault tolerance • Load balancing 33 Monday, March 8, 2010

Distributed Cloud Computing Reliability and scalability Monday, March 8, 2010

Know your Distributed Memes - Divide & conquer - Sharding - Tolerate / expect failure - Idempotency - Just scale it up 35 Monday, March 8, 2010

Rocket science Monolithic Computing 36 Monday, March 8, 2010

Rocket science Monolithic Computing Hot spare “Just in case” 36 Monday, March 8, 2010

Distributed Meme: Expect, tolerate and design for failure MTFB: O(Years) 37 Monday, March 8, 2010

Distributed Meme: Expect, tolerate and design for failure MTFB: O(Years) MTFB: O(Months) 37 Monday, March 8, 2010

Distributed Meme: Expect, tolerate and design for failure MTFB: O(Years) MTFB: O(Months) MTFB: O(Minutes/Seconds) 37 Monday, March 8, 2010

Monolithic computing stop watch small stop watch large 38 Monday, March 8, 2010

Distributed web hosting platform 39 Monday, March 8, 2010

Distributed web hosting platform X 39 Monday, March 8, 2010

Distributed web hosting platform Retry X 39 Monday, March 8, 2010

Vertical / horizontal scalability 40 Monday, March 8, 2010

Vertical scalability 41 Monday, March 8, 2010

Vertical scalability Horizontal scalability 41 Monday, March 8, 2010

Tolerate, plan for and expect small failures Monday, March 8, 2010

Easy to deploy & scale 1 2 http://www.yourdomain.com/ 43 Monday, March 8, 2010

Thinking about scalability Just a few users.... ....the tools, platform and design don't matter too much 44 Monday, March 8, 2010

Thinking about scalability Lots and lots of users... ....you must design for scalability 45 Monday, March 8, 2010

But the problem is... When? http://digg.com/ http://slashdot.org/ 46 Monday, March 8, 2010

The real problem is... When it happens. Can you afford to put everything else on hold? 47 Monday, March 8, 2010

Google App Engine Leveraging Google's platform to better serve your customers 48 Monday, March 8, 2010

Distributed web hosting platform 49 Monday, March 8, 2010

Distributed web hosting platform - Great for web apps • Request based, data backed - Parallel processing - Scales automatically - Available globally - Configuration free - Built-in DoS protections 50 Monday, March 8, 2010

Distributed datastore 51 Monday, March 8, 2010

Distributed datastore 52 Monday, March 8, 2010

Distributed datastore - Arbitrary horizontal scaling 52 Monday, March 8, 2010

Distributed datastore - Arbitrary horizontal scaling - Parallel processing 52 Monday, March 8, 2010

Distributed datastore - Arbitrary horizontal scaling - Parallel processing - Scales to 'Internet scale' 52 Monday, March 8, 2010

Distributed datastore - Arbitrary horizontal scaling - Parallel processing - Scales to 'Internet scale' - Predictable query performance 52 Monday, March 8, 2010

Distributed datastore - Arbitrary horizontal scaling - Parallel processing - Scales to 'Internet scale' - Predictable query performance - No deadlocks 52 Monday, March 8, 2010

Distributed datastore - Arbitrary horizontal scaling - Parallel processing - Scales to 'Internet scale' - Predictable query performance - No deadlocks - No global schema 52 Monday, March 8, 2010

Distributed memcache Memcache 53 Monday, March 8, 2010

Distributed Meme: Divide & Conquer Specialized services Memcache Datastore URL Fetch Mail XMPP Task Queue Images Blobstore User Service 54 Monday, March 8, 2010

Distributed Meme: Sharding Single Sharded Counter Counter Counter Counter 55 Monday, March 8, 2010

JVM languages - Scala - JRuby (Ruby) - Groovy - Quercus (PHP) - Rhino (JavaScript) - Jython (Python) 57 Monday, March 8, 2010

Ensuring portability 58 Monday, March 8, 2010

Complete Java development stack 59 Monday, March 8, 2010

Google Plugin for Eclipse 60 Monday, March 8, 2010

Google Plugin for Eclipse 61 Monday, March 8, 2010

Google Web Toolkit (GWT) 62 Monday, March 8, 2010

Google Web Toolkit (GWT) IE Firefo x i Safar e m Chro Opera 63 Monday, March 8, 2010

GWT Debugging in the browser 64 Monday, March 8, 2010

Quotes to remember “The fastest code is that which does not run.” Joel Webber GWT co-creator 65 Monday, March 8, 2010

Google Apps + your apps Your custom Our Google Apps applications Google's scalable serving architecture 66 Monday, March 8, 2010

Google Apps integration http://appid.appspot.com/ http://yourapp.yourdomain.com/ 67 Monday, March 8, 2010

2. Log in to your Google Apps domain as an admin 68 Monday, March 8, 2010

3. Accept terms and specify web address 69 Monday, March 8, 2010

4. The new feature is available in your control panel 70 Monday, March 8, 2010

5. Additional configuration is available for each feature 71 Monday, March 8, 2010

Google Apps + App Engine 72 Monday, March 8, 2010

Federate your on-premise data 73 Monday, March 8, 2010

Secure Data Connector (SDC) 74 Monday, March 8, 2010

Secure Data Connector 75 Monday, March 8, 2010

Secure Data Connector and 50+ more... 76 Monday, March 8, 2010

Using Secure Data Connector 77 Monday, March 8, 2010

Using Secure Data Connector Installation - Determine access rules - Configure and install SDC 77 Monday, March 8, 2010

Using Secure Data Connector Installation - Determine access rules - Configure and install SDC Getting ready to serve - SDC opens SSL tunnel 77 Monday, March 8, 2010

Using Secure Data Connector Installation - Determine access rules - Configure and install SDC Getting ready to serve - SDC opens SSL tunnel Serving - User request sent to App Engine - User authenticated - App makes request through tunnel - SDC performs access checks - Results returned 77 Monday, March 8, 2010

Database export / import 78 Monday, March 8, 2010

Cron support - clean-up DB - prepare r eports - batch ma il - perform calcs - call mom 79 Monday, March 8, 2010

Cron support <?xml version="1.0" encoding="UTF-8"?> <cronentries> <cron> <url>/recache</url> <description>Repopulate the cache every 2 minutes</description> <schedule>every 2 minutes</schedule> </cron> <cron> <url>/weeklyreport</url> <description>Mail out a weekly report</description> <schedule>every monday of month 08:30</schedule> <timezone>America/New_York</timezone> </cron> </cronentries> ("every"|ordinal) (days) "of" (monthspec) (time) every 5 minutes every 12 hours 2nd,third mon,wed,thu of march 17:00 every monday of month 09:00 1st monday of sep,oct,nov 17:00 80 Monday, March 8, 2010

Distributed Meme: Divide & Conquer Task Queues Task 12 Task 67 Task 42 Task 2 Task 91 Task 16 Tas Offline Processing on App Engine: A Look Ahead Brett Slatkin May, 2009 81 Monday, March 8, 2010

Tasks Queues - Motivation • Why do background processing? o Do work continuously without user requests o Incrementally process data, compute results o Smooth out load patterns, lower user latency • Simple idea in general: 1.Describe the work you want to do now 2.Save the description somewhere 3.Have something else execute the work later • Work executed in the order received (best-effort FIFO) • If execution fails, work will be retried until successful Task 12 Task 67 Task 42 Task 2 Task 91 Task 16 Tas Monday, March 8, 2010

Task Queue Benefits • Benefits o Asynchronous Why do work now when we can do it later? o Low-latency (for users) Tasks are light-weight; ~3x faster than Datastore o Reliable Once written, a task will eventually complete o Scalable Storage of new tasks has no contention Parallelizable with multiple workers • Many features can extend this basic concept Task 12 Task 67 Task 42 Task 2 Task 91 Task 16 Tas Monday, March 8, 2010

How do traditional task queues work? Monday, March 8, 2010

How do traditional task queues work? (2) • Polling has problems: o Worker sits in a loop polling the front of the queue Not event driven; wasted work o Workers stay resident when there's no work to do Wastes machine resources o Fixed number of workers Admins must manually add more workers to keep up or queue will grow without bounds • Limited optimization possible o Many systems fake a polling interface with something event-driven under the hood o Long-lived, hanging connections Monday, March 8, 2010

How does our Task Queue API work? (2) Monday, March 8, 2010

How does our Task Queue API work? (3) Monday, March 8, 2010

Your application's health 88 Monday, March 8, 2010

App Engine's health history 89 Monday, March 8, 2010

Scalable serving architecture creative commons licensed photograph from cote 90 Monday, March 8, 2010

Distributed frontends 91 Monday, March 8, 2010

Distributed frontends - Edge Caching 91 Monday, March 8, 2010

Distributed frontends - Edge Caching - Priority routing 91 Monday, March 8, 2010

Distributed frontends - Edge Caching - Priority routing - Geographically close connection endpoint 91 Monday, March 8, 2010

Life of a request App App's User Engine Runtime Frontend 92 Monday, March 8, 2010

Life of a request App App's User Engine Runtime Frontend 93 Monday, March 8, 2010

Scales dynamically / automatically User User App App's User Engine Runtime Frontend User User 94 Monday, March 8, 2010

Scales dynamically / automatically User App's User Runtime App App's User Engine Runtime Frontend App's Runtime User User 95 Monday, March 8, 2010

Multiple versions http://fredsa.appspot.com/ Version 1.1 App User Datastore Engine Frontend Version 2.1 http://2.1.fredsa.appspot.com/ 96 Monday, March 8, 2010

Distributed datastore http://labs.google.com/papers/bigtable.html 97 Monday, March 8, 2010

Bigtable : A distributed, sharded, sorted array Row key Row data 98 Monday, March 8, 2010

Bigtable : A distributed, sharded, sorted array Row key Row data Shard 1 Shard 2 . . Shard n 99 Monday, March 8, 2010

Datastore design - Distributed - Bigtable + entity groups - ACID transactions - Optimistic concurrency - Entities + indexes - Protobuf encoded entities 100 Monday, March 8, 2010

Datastore properties - Core value types - List properties - Text & binary blobs - Reference 101 Monday, March 8, 2010

What's different? 102 Monday, March 8, 2010

Datastore - what's new - Distributed - Scales to 'internet scale' - No deadlocks - Predictable query performance 103 Monday, March 8, 2010

Datastore - what's different - No inner/outer/natural joins - Dense index scans - Per entity metadata - Soft schema - No more DDL 104 Monday, March 8, 2010

Datastore - what's familiar - Indexes - ACID transactions - GQL SELECT * FROM Story WHERE title = 'App Engine Launch' AND author = :current_user AND rating >= 10 ORDER BY rating, created DESC 105 Monday, March 8, 2010

The new ACID Atomic Associative Consistent Commutative Isolated Idempotent Durable Distributed 106 Monday, March 8, 2010

Entities 107 Monday, March 8, 2010

Entity groups 108 Monday, March 8, 2010

Amortizing costs Count(COUNT) Transaction SUM(Order Amount) Create Update Delete Customer Order 109 Monday, March 8, 2010

An evolving platform 110 Monday, March 8, 2010

23 months in review Apr 2008 Python launch May 2008 Memcache, Images API Jul 2008 Logs export Aug 2008 Batch write/delete Oct 2008 HTTPS support Dec 2008 Status dashboard, quota details Feb 2009 Billing, larger files Apr 2009 Java launch, DB import, cron support, SDC May 2009 Key-only queries Jun 2009 Task queues Aug 2009 Kindless queries Sep 2009 XMPP Oct 2009 Incoming email Dec 2009 Blobstore Feb 2010 Datastore cursors, Async Urlfetch 111 Monday, March 8, 2010

App Engine Roadmap - Support for mapping operations across datasets - Alerting system for exceptions in your application - Datastore dump and restore facility 112 Monday, March 8, 2010

Wrap up 113 Monday, March 8, 2010

Always free to get started ~5M pageviews/month • 6.5 CPU hrs/day • 1 GB storage • 650K URL Fetch calls/day • 2,000 recipients emailed • 1 GB/day bandwidth • 100,000 tasks enqueued • 650K XMPP messages/day 114 Monday, March 8, 2010

Purchase additional resources * * free monthly quota of ~5 million page views still in full effect 115 Monday, March 8, 2010

Thank you Read more http://code.google.com/appengine/ Contact info Patrick Chanezon Developer Advocate chanezon@google.com http://twitter.com/chanezon Questions ? 116 Monday, March 8, 2010

Thanks To Alon Levi, Fred Sauer, Brett Slatkin and others for their slides Monday, March 8, 2010

http://jujo00obo2o234ungd3t8qjfcjrs3o6k-a-sites-opensocial.googleusercontent.com	80
https://jujo00obo2o234ungd3t8qjfcjrs3o6k-a-sites-opensocial.googleusercontent.com	73
http://www.slideshare.net	21
http://thetechnicalweb.blogspot.com	4
http://www.mefeedia.com	3
http://www.techgig.com	1
http://thetechnicalweb.blogspot.in	1

Google App Engine - Devfest India 2010

by Patrick Chanezon

Accessibility

Categories

Upload Details

Usage Rights

7 Embeds 183

Statistics

Google App Engine - Devfest India 2010 Presentation Transcript