WRITING APPS THE GOOGLE-Y WAY Pamela Fox, YOW! Australia 2010 (Brisbane)
Who am I? twitter.com/pamelafox [email_address] pamelafox.org you get the idea...
Who am I? Google Maps API Google Wave API 2006 2010 2008 Google App Engine
Who am I? wave side projects 92 apps
Who am I? Java pYthon
What is App Engine? <ul><li>“ Google App Engine enables you to build and host web apps on the same systems that power Goog...
What is a “web app”?
Static vs. Dynamic
Anonymous vs. Users
Intranet vs. Internet ~2 billion Hundreds - Thousands
What is a “web app”?
Some Google web apps
Some Google App Engine web apps www.gifttag.com www.buddypoke.com
Google apps on App Engine panoramio.com pubsubhubbub.appspot.com
How does App Engine work? <ul><li>You upload application code & resources to Google. </li></ul><ul><li>Google serves your ...
Demo: Guestbook awesomest-app.appspot.com http://code.google.com/p/google-app-engine-samples/source/browse/trunk/guestbook...
App Engine architecture
App Engine architecture user task
App Engine architecture
App Engine architecture LIMIT CPU LIMIT Memory LIMIT Time
App Engine architecture hardware ports globals file system Groovy, JRuby, Mirah, Clojure, Scala
App Engine architecture 141,241,791 calls 1 GB data $0.15 GB/month 45,000,000 calls 657,000 -  46,000,000 calls *Always ch...
The tricky bits
Datastore Entity Properties Key Entity Entity Entity Entity Path Kind Name/ID
Example: Speaker Entities Key Path Kind ID First Name Last Name Speaker1 - Speaker 1 Rod Johnson Key Path Kind ID First Na...
Modeling Speaker Entities <ul><li>class Speaker(db.model): </li></ul><ul><li>firstname = db.StringProperty(required=True) ...
Saving Speaker Entities <ul><li>rod = Speaker(firstname=&quot;Rod&quot;, lastname=&quot;Johnson&quot;) </li></ul><ul><li>g...
Updating Speaker Entities <ul><li>rod = Speaker.get_by_id(1) </li></ul><ul><li>guy = Speaker.get_by_id(2) </li></ul><ul><l...
Queries & Indexes Query Index Index Index Index Query Query Query Query Query
Queries & Indexes SELECT * from Speaker ORDER BY lastname LIMIT! (# of results) key lastname Speaker3 Fox Speaker4 Hohpe S...
Queries & Indexes SELECT * from Speaker ORDER by middlename key middlename Speaker2 L
Queries & Indexes SELECT * from Speaker WHERE keynote = True key keynote Speaker1 True Speaker2 True Speaker3 False Speake...
Queries & Indexes SELECT * from Speaker WHERE keynote = False key keynote Speaker1 True Speaker2 True Speaker3 False Speak...
Queries <ul><li>allspeakers = Speaker.all().order('lastname) </li></ul><ul><li>for speaker in allspeakers: </li></ul><ul><...
Custom Indexes <ul><li>speakers = Speaker.all().order('lastname') </li></ul><ul><li>.order('keynote') </li></ul>SELECT * f...
Custom Indexes <ul><li>speakers = Speaker.all().order('lastname') </li></ul><ul><li>.filter('keynote =', True) </li></ul>S...
Impossible Indexes SELECT * from Speaker WHERE lastname < 'Steele'  and firstname > 'Gregory' ...not in subsequent rows! k...
Impossible Indexes SELECT * from Speaker WHERE lastname > 'Fox' ORDER BY firstname  ...not in the correct order! key lastn...
Queries with Offset <ul><li>speakers = Speaker.all().fetch(2, 2) </li></ul>SELECT * from Speaker LIMIT 2 OFFSET 2 1 2 ...s...
Queries with Cursors <ul><li>query = db.Query(Speaker) </li></ul><ul><li>speakers = q.fetch(1000) </li></ul><ul><li>cursor...
More Properties class Talk(db.Model): title = db.StringProperty(required=True) abstract = db.TextProperty(required=True) s...
Back-References pamela = Speaker.all().filter('firstname = ', 'Pamela').get() for talk in pamela.talk_set: print talk.titl...
Searching List Properties talks = Talk.all().filter('tags = ', 'python') .fetch(10) SELECT * from Talk WHERE tags = 'Pytho...
Update Transactions commit journal apply entities apply indexes A B
Entity Groups pamela = Speaker.all().filter('firstname = ', 'Pamela').get() talk1 = Talk('Writing Apps the Googley Way', '...
Common Features
Counters 1 2 3 4 5 people have done something.
RageTube: Global Stats ragetube.net http://github.com/pamelafox/ragetube
RageTube: Global Stats SongStat yaycount viewcount title artist Key Path Kind Name (song) naycount mehcount
RageTube: Global Stats viewcount viewcount viewcount datastore memcache
RageTube: Global Stats class Song(db.Model): viewcount = db.IntegerProperty(default=0) title = db.StringProperty() artist ...
Ratings Rated by 500 users.
App Gallery: Ratings google.com/analytics/apps/
App Gallery: Ratings Comment Application total_ratings sum_ratings avg_rating rated_index comment_count rating
App Gallery: Ratings def UpdateAppCommentData(self, rating, operation): def UpdateCommentData(self, rating, operation): se...
Geospatial Queries
City-Go-Round: Agencies citygoround.org https://github.com/walkscore/City-Go-Round
City-Go-Round: Geo Queries Agency GeoModel location (GeoPt) location_geocells (StringListProperty)
City-Go-Round: Geo Queries def fetch_agencies_near(lat, long, bbox_side_in_miles): query = Agency.all() bbox = bbox_center...
Full Text Search <ul><li>pizza </li></ul>Search Thingy It's like pizza, but in the cloud. Other Thingy This will make you ...
Disclosed.ca: Search https://github.com/nurey/disclosed disclosed.ca
Disclosed.ca: Search Contract agency_name vendor_name description comments uri
Disclosed.ca: Search from search.core import SearchIndexProperty, porter_stemmer class Contract(db.Model): uri = db.String...
Disclosed.ca: Search Contract agency_name vendor_name description comments uri search_index (StringListProperty) SearchIndex
Disclosed.ca: Search SELECT FROM ContractSearch WHERE search_index = &quot;sheep&quot; key search_index ContractSearch1 ch...
More Learning http://ae-book.appspot.com http://code.google.com/appengine http://blog.notdot.net/
AppEngine: Now & Later &quot;Run your web apps on Google's infrastructure. Easy to build, easy to maintain, easy to scale....
Thanks for coming!
Upcoming SlideShare
Loading in...5
×

Writing Apps the Google-y Way (Brisbane)

2,471

Published on

Talk from Pamela Fox (me) at YOW 2010 in Brisbane. Covers App Engine and the datastore, with Python examples.

Published in: Business, Technology
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,471
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
13
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide

Writing Apps the Google-y Way (Brisbane)

  1. 1. WRITING APPS THE GOOGLE-Y WAY Pamela Fox, YOW! Australia 2010 (Brisbane)
  2. 2. Who am I? twitter.com/pamelafox [email_address] pamelafox.org you get the idea...
  3. 3. Who am I? Google Maps API Google Wave API 2006 2010 2008 Google App Engine
  4. 4. Who am I? wave side projects 92 apps
  5. 5. Who am I? Java pYthon
  6. 6. What is App Engine? <ul><li>“ Google App Engine enables you to build and host web apps on the same systems that power Google applications.” </li></ul><ul><li>http://code.google.com/appengine </li></ul>
  7. 7. What is a “web app”?
  8. 8. Static vs. Dynamic
  9. 9. Anonymous vs. Users
  10. 10. Intranet vs. Internet ~2 billion Hundreds - Thousands
  11. 11. What is a “web app”?
  12. 12. Some Google web apps
  13. 13. Some Google App Engine web apps www.gifttag.com www.buddypoke.com
  14. 14. Google apps on App Engine panoramio.com pubsubhubbub.appspot.com
  15. 15. How does App Engine work? <ul><li>You upload application code & resources to Google. </li></ul><ul><li>Google serves your application from scalable infrastructure. </li></ul><ul><li>You pay for only the resources that Google used in serving the application. </li></ul>
  16. 16. Demo: Guestbook awesomest-app.appspot.com http://code.google.com/p/google-app-engine-samples/source/browse/trunk/guestbook appengine.google.com localhost build deploy monitor
  17. 17. App Engine architecture
  18. 18. App Engine architecture user task
  19. 19. App Engine architecture
  20. 20. App Engine architecture LIMIT CPU LIMIT Memory LIMIT Time
  21. 21. App Engine architecture hardware ports globals file system Groovy, JRuby, Mirah, Clojure, Scala
  22. 22. App Engine architecture 141,241,791 calls 1 GB data $0.15 GB/month 45,000,000 calls 657,000 - 46,000,000 calls *Always check docs for latest quotas. 192,672,000 calls 558 GB data $0.15 GB/month 7,000 - 1,700,000 calls $0.0001 per mail sent 46,000,000 calls 1,046 GB data sent 100,000 - 20,000,000 calls
  23. 23. The tricky bits
  24. 24. Datastore Entity Properties Key Entity Entity Entity Entity Path Kind Name/ID
  25. 25. Example: Speaker Entities Key Path Kind ID First Name Last Name Speaker1 - Speaker 1 Rod Johnson Key Path Kind ID First Name Last Name Middle Name Suffix Speaker2 - Speaker 2 Guy Steele L Jr.
  26. 26. Modeling Speaker Entities <ul><li>class Speaker(db.model): </li></ul><ul><li>firstname = db.StringProperty(required=True) </li></ul><ul><li>lastname = db.StringProperty(required=True) </li></ul><ul><li>middlename = db.StringProperty() </li></ul><ul><li>namesuffix = db.StringProperty() </li></ul><ul><li>website = db.StringProperty() </li></ul><ul><li>keynote = db.BooleanProperty(default=False) </li></ul>
  27. 27. Saving Speaker Entities <ul><li>rod = Speaker(firstname=&quot;Rod&quot;, lastname=&quot;Johnson&quot;) </li></ul><ul><li>guy = Speaker(firstname=&quot;Guy&quot;, lastname=&quot;Steele&quot;, </li></ul><ul><li>middlename=&quot;L&quot;, namesuffix=&quot;Jr.&quot;) </li></ul><ul><li>rod.put() </li></ul><ul><li>guy.put() </li></ul>
  28. 28. Updating Speaker Entities <ul><li>rod = Speaker.get_by_id(1) </li></ul><ul><li>guy = Speaker.get_by_id(2) </li></ul><ul><li>rod.website = &quot;http://www.sexyspring.com&quot; </li></ul><ul><li>rod.keynote = True </li></ul><ul><li>guy.website = &quot;http://www.lusciouslisp.com&quot; </li></ul><ul><li>guy.keynote = True </li></ul><ul><li>db.put(rod, guy) </li></ul>LIMIT! (size/# of batch ops)
  29. 29. Queries & Indexes Query Index Index Index Index Query Query Query Query Query
  30. 30. Queries & Indexes SELECT * from Speaker ORDER BY lastname LIMIT! (# of results) key lastname Speaker3 Fox Speaker4 Hohpe Speaker1 Johnson Speaker2 Steele
  31. 31. Queries & Indexes SELECT * from Speaker ORDER by middlename key middlename Speaker2 L
  32. 32. Queries & Indexes SELECT * from Speaker WHERE keynote = True key keynote Speaker1 True Speaker2 True Speaker3 False Speaker4 False
  33. 33. Queries & Indexes SELECT * from Speaker WHERE keynote = False key keynote Speaker1 True Speaker2 True Speaker3 False Speaker4 False
  34. 34. Queries <ul><li>allspeakers = Speaker.all().order('lastname) </li></ul><ul><li>for speaker in allspeakers: </li></ul><ul><li>print speaker.firstname + '' + speaker.lastname + '' + speaker.website </li></ul><ul><li>keynotespeakers = Speaker.all().filter('keynote = ', True) </li></ul><ul><li>notspecialspeakers = Speaker.all().filter('keynote = ', False) </li></ul>LIMIT! (size of results)
  35. 35. Custom Indexes <ul><li>speakers = Speaker.all().order('lastname') </li></ul><ul><li>.order('keynote') </li></ul>SELECT * from Speaker ORDER BY lastname, keynote key lastname keynote Speaker3 Fox false Speaker4 Hohpe false Speaker1 Johnson true Speaker2 Steele true
  36. 36. Custom Indexes <ul><li>speakers = Speaker.all().order('lastname') </li></ul><ul><li>.filter('keynote =', True) </li></ul>SELECT * from Speaker WHERE lastname > 'Johnson' and keynote = true key lastname keynote Speaker3 Fox false Speaker4 Hohpe false Speaker1 Johnson true Speaker2 Steele true
  37. 37. Impossible Indexes SELECT * from Speaker WHERE lastname < 'Steele' and firstname > 'Gregory' ...not in subsequent rows! key lastname firstname Speaker3 Fox Pamela Speaker4 Hohpe Gregory Speaker1 Johnson Rod Speaker2 Steele Guy
  38. 38. Impossible Indexes SELECT * from Speaker WHERE lastname > 'Fox' ORDER BY firstname ...not in the correct order! key lastname firstname Speaker3 Fox Pamela Speaker4 Hohpe Gregory Speaker1 Johnson Rod Speaker2 Steele Guy
  39. 39. Queries with Offset <ul><li>speakers = Speaker.all().fetch(2, 2) </li></ul>SELECT * from Speaker LIMIT 2 OFFSET 2 1 2 ...slow! LIMIT! (# of offset) key lastname Speaker3 Fox Speaker4 Hohpe Speaker1 Johnson Speaker2 Steele
  40. 40. Queries with Cursors <ul><li>query = db.Query(Speaker) </li></ul><ul><li>speakers = q.fetch(1000) </li></ul><ul><li>cursor = q.cursor() </li></ul><ul><li>memcache.set('speaker_cursor', cursor) </li></ul><ul><li>... </li></ul><ul><li>last_cursor = memcache.get('speaker_cursor') </li></ul><ul><li>q.with_cursor(last_cursor) </li></ul><ul><li>speakers = q.fetch(1000) </li></ul>
  41. 41. More Properties class Talk(db.Model): title = db.StringProperty(required=True) abstract = db.TextProperty(required=True) speaker = db.ReferenceProperty(Speaker) tags = db.StringListProperty() pamela = Speaker.all().filter('firstname = ', 'Pamela').get() talk = Talk('Writing Apps the Googley Way', 'Bla bla bla', pamela, ['App Engine', 'Python']) talk.put() talk = Talk('Wonders of the Onesie', 'Bluh bluh bluh', pamela, ['Pajamas', 'Onesies']) talk.put()
  42. 42. Back-References pamela = Speaker.all().filter('firstname = ', 'Pamela').get() for talk in pamela.talk_set: print talk.title SELECT * from Talk WHERE speaker = Speaker3 key speaker Talk6 Speaker2 Talk1 Speaker3 Talk2 Speaker3 Talk5 Speaker4
  43. 43. Searching List Properties talks = Talk.all().filter('tags = ', 'python') .fetch(10) SELECT * from Talk WHERE tags = 'Python' LIMIT! (# of index rows) key lastname Talk1 App Engine Talk2 Pajamas Talk1 Python Talk2 Onesies
  44. 44. Update Transactions commit journal apply entities apply indexes A B
  45. 45. Entity Groups pamela = Speaker.all().filter('firstname = ', 'Pamela').get() talk1 = Talk('Writing Apps the Googley Way', 'Bla bla bla', pamela, ['App Engine', 'Python'], parent=pamela) talk2 = Talk('Wonders of the Onesie', 'Bluh bluh bluh', pamela, ['Pajamas', 'Onesies'], parent=pamela) db.put(talk1, talk2) def update_talks(): talk1.title = 'Writing Apps the Microsoft Way' talk2.title = 'Wonders of the Windows' db.put(talk1, talk2) db.run_in_transaction(update_talks)
  46. 46. Common Features
  47. 47. Counters 1 2 3 4 5 people have done something.
  48. 48. RageTube: Global Stats ragetube.net http://github.com/pamelafox/ragetube
  49. 49. RageTube: Global Stats SongStat yaycount viewcount title artist Key Path Kind Name (song) naycount mehcount
  50. 50. RageTube: Global Stats viewcount viewcount viewcount datastore memcache
  51. 51. RageTube: Global Stats class Song(db.Model): viewcount = db.IntegerProperty(default=0) title = db.StringProperty() artist = db.StringProperty() def get_viewcount(self): viewcount = self.viewcount cached_viewcount = memcache.get('viewcount-' + self.key().name(), self.key().kind()) if cached_viewcount: viewcount += cached_viewcount return viewcount @classmethod def flush_viewcount(cls, name): song = cls.get_by_key_name(name) value = memcache.get('viewcount-' + name, cls.kind()) memcache.decr('viewcount-' + name, value, cls.kind()) song.viewcount += value song.put() @classmethod def incr_viewcount(cls, name, interval=5, value=1): memcache.incr('viewcount-' + name, value, cls.kind()) interval_num = get_interval_number(datetime.now(), interval) task_name = '-'.join([cls.kind(), name.replace(' ', '-'), 'viewcount', str(interval), str(interval_num)]) deferred.defer(cls.flush_viewcount, name, _name=task_name) LIMIT! (# of tasks)
  52. 52. Ratings Rated by 500 users.
  53. 53. App Gallery: Ratings google.com/analytics/apps/
  54. 54. App Gallery: Ratings Comment Application total_ratings sum_ratings avg_rating rated_index comment_count rating
  55. 55. App Gallery: Ratings def UpdateAppCommentData(self, rating, operation): def UpdateCommentData(self, rating, operation): self.comment_count += 1 * operation self.sum_ratings += rating * operation self.total_ratings += 1 * operation self.avg_rating = int(round(self.sum_ratings / self.total_ratings)) self.rated_index = '%d:%d:%d' % (self.avg_rating, self.total_ratings, self.index) self.put() db.run_in_transaction(UpdateCommentData, self, rating, operation) app.UpdateAppCommentData(rating, db_models.Comment.ADD) comment = db_models.Comment() comment.application = app comment.rating = rating comment.put() query.order('-avg_rating').order('-rated_index')
  56. 56. Geospatial Queries
  57. 57. City-Go-Round: Agencies citygoround.org https://github.com/walkscore/City-Go-Round
  58. 58. City-Go-Round: Geo Queries Agency GeoModel location (GeoPt) location_geocells (StringListProperty)
  59. 59. City-Go-Round: Geo Queries def fetch_agencies_near(lat, long, bbox_side_in_miles): query = Agency.all() bbox = bbox_centered_at(lat, long, bbox_side_in_miles) return Agency.bounding_box_fetch(query, bbox, max_results = 50) def bounding_box_fetch(query, bbox, max_results=1000,): results = [] query_geocells = geocell.best_bbox_search_cells(bbox) for entity in query.filter('location_geocells IN', query_geocells): if len(results) == max_results: break if (entity.location.lat >= bbox.south and entity.location.lat <= bbox.north and entity.location.lon >= bbox.west and entity.location.lon <= bbox.east): results.append(entity) return results
  60. 60. Full Text Search <ul><li>pizza </li></ul>Search Thingy It's like pizza, but in the cloud. Other Thingy This will make you smell as delicious as pizza.
  61. 61. Disclosed.ca: Search https://github.com/nurey/disclosed disclosed.ca
  62. 62. Disclosed.ca: Search Contract agency_name vendor_name description comments uri
  63. 63. Disclosed.ca: Search from search.core import SearchIndexProperty, porter_stemmer class Contract(db.Model): uri = db.StringProperty(required=True) agency_name = db.StringProperty(required=True) vendor_name = db.StringProperty(required=True) description = db.StringProperty() comments = db.TextProperty() search_index = SearchIndexProperty(('agency_name', 'vendor_name', 'description', 'comments'), indexer=porter_stemmer) results = Contract.search_index.search(sheep').fetch(20)
  64. 64. Disclosed.ca: Search Contract agency_name vendor_name description comments uri search_index (StringListProperty) SearchIndex
  65. 65. Disclosed.ca: Search SELECT FROM ContractSearch WHERE search_index = &quot;sheep&quot; key search_index ContractSearch1 charter ContractSearch1 june ContractSearch1 sheep ContractSearch2 sheep ContractSearch1 wood
  66. 66. More Learning http://ae-book.appspot.com http://code.google.com/appengine http://blog.notdot.net/
  67. 67. AppEngine: Now & Later &quot;Run your web apps on Google's infrastructure. Easy to build, easy to maintain, easy to scale.&quot; <ul><li>Roadmap: </li></ul><ul><li>App Engine (Standard): </li></ul><ul><li>MapReduce </li></ul><ul><li>Bulk Import/Export </li></ul><ul><li>Channel API </li></ul><ul><li>Datastore Options </li></ul><ul><li>App Engine for Business: </li></ul><ul><li>SQL </li></ul><ul><li>SLA + Support </li></ul>
  68. 68. Thanks for coming!
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×