App Engine Dev Days DC 20091026

Google App Engine Jonathan Blocksom Stack Overﬂow DevDays DC, 10/26/09

Submit a Question: http://tinyurl.com/gae-dd09 Attention Conservation Notice • Jonathan Blocksom jblocksom@google.com @jblocksom Software Engineer Google Public Sector Projects Team

Submit a Question: http://tinyurl.com/gae-dd09 We’re lonely Open positions at Google in DC

Submit a Question: http://tinyurl.com/gae-dd09 App Engine! • Launched April 2008 • Paid Use Feb 2009 • Java support May 2009 • Latest Release (1.2.7) October 2009 • Incoming Email • Previous Release (1.2.5) September 2009 • XMPP Support

Submit a Question: http://tinyurl.com/gae-dd09 Your Scalable Website? • Brad Fitzpatrick describes LiveJournal in ’07: net. LiveJournal Backend: 2007 Today (Roughly.) BIG-IP perlbal (httpd/proxy) Global Database bigip1 mod_perl bigip2 proxy1 master_a master_b web1 proxy2 web2 proxy3 Memcached web3 slave1 slave2 ... slave5 djabberd proxy4 mc1 web4 djabberd proxy5 ... mc2 User DB Cluster 1 djabberd webN mc3 uc1a uc1b mc4 User DB Cluster 2 ... uc2a uc2b gearmand Mogile Storage Nodes gearmand1 mcN User DB Cluster 3 sto1 sto2 gearmandN uc3a uc3b Mogile Trackers ... sto8 tracker1 tracker3 User DB Cluster N ucNa ucNb MogileFS Database “workers” gearwrkN Job Queues (xN) mog_a mog_b theschwkN jqNa jqNb slave1 slaveN http://danga.com/words/ 3

Submit a Question: http://tinyurl.com/gae-dd09 App Engine Scalable Website Request App Engine Response

Submit a Question: http://tinyurl.com/gae-dd09 What’s Scalable? • Google Moderator on App Engineused in March 2009 for White House Town Hall meeting

Submit a Question: http://tinyurl.com/gae-dd09 Scalable

Submit a Question: http://tinyurl.com/gae-dd09 Google Moderator team didn’t... • Provision a server • Install and configure an OS • Install and configure a web server • Install and configure a web framework • Install and configure a database • Install and configure a load balancer • Create user accounts • Crash in the middle of any of the above

Submit a Question: http://tinyurl.com/gae-dd09 App Engine Architecture

Submit a Question: http://tinyurl.com/gae-dd09 Static Content • Served by Google Static Content Servers • Very Efﬁcient

Submit a Question: http://tinyurl.com/gae-dd09 Dynamic Content • Java xor Python • Sandboxed • No ﬁle writes • No sockets • No spawning threads/processes • Smart about caching • Fairly standard CGI Framework • Measured and Instrumented

Submit a Question: http://tinyurl.com/gae-dd09 Datastore 1 / 3 • Based on BigTable, transitioning to MegaStore • ACID Compliant • 2008 Talk on Implementation: http://sites.google.com/site/io/under-the- covers-of-the-google-app-engine-datastore

Submit a Question: http://tinyurl.com/gae-dd09 Datastore 2 / 3 • Typical types: Int, String, DateTime, GeoPt, PhoneNumber, PostalAddress, Email, Link, Blob...

Submit a Question: http://tinyurl.com/gae-dd09 Datastore 3 / 3 • GQL: Google Query Language • If it’s not SQL, it must be good! SELECT [* | __key__] FROM <kind> [WHERE <condition> [AND <condition> ...]] [ORDER BY <property> [ASC | DESC] [, <property> [ASC | DESC] ...]] [LIMIT [<offset>,]<count>] [OFFSET <offset>] <condition> := <property> {< | <= | > | >= | = | != } <value> <condition> := <property> IN <list> <condition> := ANCESTOR IS <entity or key>

Submit a Question: http://tinyurl.com/gae-dd09 Memcache • Arbitrary key/value cache • Similar API to memcached • get, set • get_multi, set_multi for speed • Namespace Support • 1MB limit per entry • Free Quota 8.6M/day, 48K/min (800/sec)

Submit a Question: http://tinyurl.com/gae-dd09 Users & Authentication • @gmail.com addresses • Apps for Domain • Admin Privileges

Submit a Question: http://tinyurl.com/gae-dd09 URL Fetch • Load URL • Asynchronous support • HTTP / HTTPS • Max 10 second response (default 5) • Max 1MB data

Submit a Question: http://tinyurl.com/gae-dd09 Even More... • Scheduled Tasks • Task Queue • Email • Image Manipulation cron jobs • XMPP email • ... XMPP

Submit a Question: http://tinyurl.com/gae-dd09 Web Hooks Pattern • Incoming data / operations to perform are URLs with POST data • Handled same as web request Scheduled Tasks Task Queue tasks Incoming Email XMPP Messages

Submit a Question: http://tinyurl.com/gae-dd09 Templates? • Treetop Wants to know

Submit a Question: http://tinyurl.com/gae-dd09 Templates! • Django Templates built in Variable substitution, function calls, loops, etc • Can add your own Python temlpate lib

Submit a Question: http://tinyurl.com/gae-dd09 Let’s Review!

Submit a Question: http://tinyurl.com/gae-dd09 Developing w/ App Engine • Local test server • Write your code, html, etc on local machine • Send code to App Engine • Monitor in App Engine Admin Console

Submit a Question: http://tinyurl.com/gae-dd09 Dashboard Demo • http://appengine.google.com/

Submit a Question: http://tinyurl.com/gae-dd09 What have we done? • Served static web pages • Served dynamic web pages • Stored data (cached & persistent) • Ran a cron Job • Authenticated users • We could have... • Scheduled a task, sent or received email, loaded a URL, modiﬁed an image, rocked out with XMPP

Submit a Question: http://tinyurl.com/gae-dd09 Quotas • 34 global quotas in Dashboard • Individual quotas: request size time amount of data • Exceptions thrown when hit • Free and Paid

Submit a Question: http://tinyurl.com/gae-dd09 Quota Rule of Thumb • Less than 1000 of everything • The Zen of Quotas: Meet the quotas and you will be scalable

Submit a Question: http://tinyurl.com/gae-dd09 All For Good • Volunteering Opportunity Search Engine • Started March 2009 in cooperation with White House, CNCS, Craigslist, Google... • Released June 2009 • Open source, released under Apache License http://code.google.com/p/allforgood/ • Front end at http://allforgood.org API powers search results on serve.gov

Submit a Question: http://tinyurl.com/gae-dd09 Tech Overview • Partners provide listings in XML or Google Spreadsheets • AARP, United Way, AmeriCorp, HandsOnNetwork, United Way • AFG indexes feeds, stores outside GAE • Initially Google Base • Now SOLR on private server • GAE front end • API for external apps, partners • Integration with Google Maps, FriendConnect, Facebook

Submit a Question: http://tinyurl.com/gae-dd09 Tech Overview User or API Search Queries Backend Volunteering Opportunities (Base or App Engine s SOLR) lt su Volunteering Opportunities Re memcache Listings ... Volunteering Opportunities Crawler Static Content Google Code SVN

Submit a Question: http://tinyurl.com/gae-dd09 Why A Different Back End? • No full text search in App Engine Datastore

Submit a Question: http://tinyurl.com/gae-dd09 On The Promiscuous Nature of GAE • GAE plays well with others • Do full text search with something else! • Serve large ﬁles with S3! • Serve videos with YouTube!

Submit a Question: http://tinyurl.com/gae-dd09 Challenges with AFG • Parsing questionable data • Integration w/ Social Networks • Google Base is an Odd Duck • I Was New And Had No Clue What Was Going • App Engine Outages

Submit a Question: http://tinyurl.com/gae-dd09 Q &A

Submit a Question: http://tinyurl.com/gae-dd09 More Information • App Engine Documentation • Google I/O 2008 & 2009 • 2008 for intro, 2009 for deep dives / Java • http://code.google.com/events/io/2009/ sessions.html#appengine “App Engine Nitty-Gritty” “Spark Plug to Drive Train” • Google Group • Samples http://code.google.com/p/google-app-engine-samples/

App Engine Dev Days DC 20091026

0 comments

Notes on slide 1

Favorites, Groups & Events