2011 aug-gdd-mexico-city-high-replication-datastore

2,501 views

Published on

Slides for my talk about the

Published in: Technology, News & Politics
0 Comments
7 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,501
On SlideShare
0
From Embeds
0
Number of Embeds
129
Actions
Shares
0
Downloads
57
Comments
0
Likes
7
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Does CLOUD COMPUTING just means your servers are SOMEWHERE ELSE? Or is it SOMETHING MORE?\nWHY put your servers in the cloud?\n- Don’t want to MANAGE servers?\n- Or is it the ELASTICITY and SCALABILITY of the cloud?\n- If so, you NEED: DISTRIBUTED cloud computing\n * TODAY we’ll talk about why\n
  • Does CLOUD COMPUTING just means your servers are SOMEWHERE ELSE? Or is it SOMETHING MORE?\nWHY put your servers in the cloud?\n- Don’t want to MANAGE servers?\n- Or is it the ELASTICITY and SCALABILITY of the cloud?\n- If so, you NEED: DISTRIBUTED cloud computing\n * TODAY we’ll talk about why\n
  • Does CLOUD COMPUTING just means your servers are SOMEWHERE ELSE? Or is it SOMETHING MORE?\nWHY put your servers in the cloud?\n- Don’t want to MANAGE servers?\n- Or is it the ELASTICITY and SCALABILITY of the cloud?\n- If so, you NEED: DISTRIBUTED cloud computing\n * TODAY we’ll talk about why\n
  • \n
  • \n
  • SPECIALIZATION => SCALABILITY\n
  • SPECIALIZATION => SCALABILITY\n
  • SPECIALIZATION => SCALABILITY\n
  • SPECIALIZATION => SCALABILITY\n
  • SPECIALIZATION => SCALABILITY\n
  • SPECIALIZATION => SCALABILITY\n
  • \n
  • \n
  • \n
  • SPECIALIZATION => SCALABILITY\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • 2011 aug-gdd-mexico-city-high-replication-datastore

    1. 1. High Replication Datastore Ikai Lan - @ikai Esto es Google August 9th, 2011
    2. 2. About the speaker• Developer Relations at Google based out of San Francisco, CA• Google+: http://plus.ikailan.com• Twitter: @ikai
    3. 3. About the speakerBIOGRAFÍA: Ikai es ingeniero de Desarrollo deProgramas en el motor de Google App. Antes deGoogle, trabajó como ingeniero programadorconstruyendo aplicaciones para móviles y redessociales en LinkedIn. Ikai es un ávido de latecnología, consumiendo cantidades de materialacerca de nuevos lenguajes de programación,estructuras o servicios. En sus ratos libres disfrutade California, ganando concursos de karaokechino y jugando futbol de bandera. Actualmentevive en el área de la Bahía de San Francisco,donde agoniza viendo como su equipo favoritoexplota temporada tras temporada. English original: http://code.google.com/team/
    4. 4. About the speakerBIOGRAFÍA: Ikai es ingeniero de Desarrollo deProgramas en el motor de Google App. Antes deGoogle, trabajó como ingeniero programadorconstruyendo aplicaciones para móviles y redessociales en LinkedIn. Ikai es un ávido de latecnología, consumiendo cantidades de materialacerca de nuevos lenguajes de programación,estructuras o servicios. En sus ratos libres disfrutade California, ganando concursos de karaokechino y jugando futbol de bandera. Actualmentevive en el área de la Bahía de San Francisco,donde agoniza viendo como su equipo favoritoexplota temporada tras temporada. !!! English original: http://code.google.com/team/
    5. 5. Agenda• What is Google App Engine?• Intro to High Replication Datastore• How does High Replication work under the hood?
    6. 6. If you’re not an App Engine developer ..• First off, shame on you• The code in these examples might not make sense• The concepts in these slides are always good to understand
    7. 7. SaaSPaaSIaaS Source: Gartner AADI Summit Dec 2009
    8. 8. SaaSPaaSIaaS Source: Gartner AADI Summit Dec 2009
    9. 9. SaaSPaaSIaaS Source: Gartner AADI Summit Dec 2009
    10. 10. SaaSPaaSIaaS Source: Gartner AADI Summit Dec 2009
    11. 11. SDK & “The Cloud”HardwareNetworkingOperating systemApplication runtime Java, PythonStatic file serving 20
    12. 12. Duke, the Java mascotGo Gopher Copyright © Sun Microsystems Inc., all rights reserved.
    13. 13. Dynamic Scaling: Low traffic App Server22
    14. 14. Dynamic Scaling: Low traffic App Server22
    15. 15. Dynamic Scaling: Grows with demand App App Server App Server Server22
    16. 16. Dynamic Scaling: Grows with demand App App Server App Server Server22
    17. 17. Dynamic Scaling: Grows with demand App App Server App Server Server22
    18. 18. Dynamic Scaling: Grows with demand App App Server App Server Server22
    19. 19. 150,000+ Apps100,000+ Developers2B Page views / day
    20. 20. Customer: WebFilingsDisruptive multi-tenant App Engine application adopted byFortune 500 companies.
    21. 21. Customer: The Royal WeddingPeak: 32,000 requests a second with no disruption!
    22. 22. Core APIsMemcache Datastore URL Fetch Mail XMPP Task Queue Images Blobstore User Service
    23. 23. App Engine Datastore Schemaless, non-relational datastore built on top of Google’s Bigtable technologyEnables rapid development and scalability
    24. 24. High Replication• Strongly consistent• Multi-data center• Consistent performance• High Reliability• No data loss
    25. 25. How do I use it?• Create a new application! Just remember the rules• Fetch by key and ancestor queries exhibit strongly consistent behavior• Queries without an ancestor exhibit eventually consistent behavior
    26. 26. Strong vs. Eventual• Strong consistency means immediately after the datastore tells us a write has been committed, the effects of that write are immediately visible• Eventual consistency means that after the datastore tells us a write has been committed, the effects of that write are visible after some time
    27. 27. Strong Consistency
    28. 28. Eventual Consistency
    29. 29. This is strongly consistentDatastoreService datastore = DatastoreServiceFactory .getDatastoreService(); Entity item = new Entity("Item");item.setProperty("data", 123);Key key = datastore.put(item);// This exhibits strong consistency.// It should return the item we just saved.Entity result = datastore.get(key); Get by key
    30. 30. This is strongly consistent// Save the entity rootEntity root = new Entity("Root");Key rootKey = datastore.put(root);// Save the childEntity childItem = new Entity("Item", rootKey); AncestorchildItem.setProperty("data", 123);datastore.put(childItem); queryQuery strongConsistencyQuery = new Query("Item");strongConsistencyQuery.setAncestor(rootKey);strongConsistencyQuery.addFilter("data", FilterOperator.EQUAL, 123);FetchOptions opts = FetchOptions.Builder.withDefaults();// This query exhibits strong consistency.// It will return the item we just saved.List<Entity> results = datastore.prepare(strongConsistencyQuery) .asList(opts);
    31. 31. This is eventually consistentEntity item = new Entity("Item");item.setProperty("data", 123);datastore.put(item);// Not an ancestor queryQuery eventuallyConsistentQuery = new Query("Item");eventuallyConsistentQuery.addFilter("data", FilterOperator.EQUAL, 123);FetchOptions opts = FetchOptions.Builder.withDefaults();// This query exhibits eventual consistency.// It will likely return an empty list.List<Entity> results = datastore.prepare(eventuallyConsistentQuery) .asList(opts);
    32. 32. Why?• Reads are transactional• On a read, we try to determine if we have the latest version of data• If not, we catch up the local node to the latest version
    33. 33. To understand this ..• We need some understanding of our implementation of Paxos• ...which necessitates some understanding of transactions• ... which necessitates some some understanding of entity groups
    34. 34. Under the hood
    35. 35. Entity GroupsEntity Usergroup root Blog Blog Entry Entry Entry Comment Comment Comment
    36. 36. Entity group root// Save the entity rootEntity root = new Entity("Root");Key rootKey = datastore.put(root);// Save the childEntity childItem = new Entity("Item", rootKey);childItem.setProperty("data", 123);datastore.put(childItem);Query strongConsistencyQuery = new Query("Item");strongConsistencyQuery.setAncestor(rootKey);strongConsistencyQuery.addFilter("data", FilterOperator.EQUAL, 123);FetchOptions opts = FetchOptions.Builder.withDefaults();// This query exhibits strong consistency.// It will return the item we just saved.List<Entity> results = datastore.prepare(strongConsistencyQuery) .asList(opts);
    37. 37. Entity group rootEntity Rootgroup root
    38. 38. Adding an entity child// Save the entity rootEntity root = new Entity("Root");Key rootKey = datastore.put(root);// Save the childEntity childItem = new Entity("Item", rootKey);childItem.setProperty("data", 123);datastore.put(childItem);Query strongConsistencyQuery = new Query("Item");strongConsistencyQuery.setAncestor(rootKey);strongConsistencyQuery.addFilter("data", FilterOperator.EQUAL, 123);FetchOptions opts = FetchOptions.Builder.withDefaults();// This query exhibits strong consistency.// It will return the item we just saved.List<Entity> results = datastore.prepare(strongConsistencyQuery) .asList(opts);
    39. 39. Entity Groups RootWe added Itema child
    40. 40. Entity Groups RootWe added Itema child Transactions “lock” on this entity
    41. 41. Transactional reads Datastore Version 11
    42. 42. Transactional reads AppServer Data read begins Datastore Version 11
    43. 43. Transactional reads Version 12 of data Still being committed App AppServer Data read begins Server Datastore Version 11
    44. 44. Transactional reads Version 12 of data Still being committed App AppServer Data read begins Server Datastore Version 11 Version 11 of data returned - this is fully committed version
    45. 45. Optimistic locking Optimistic because you assumemost of the time no modifications willoccur while you haveobject out - you only do work when this isn’t true
    46. 46. Optimistic Locking Datastore Version 11
    47. 47. Optimistic Locking AppServer Data read begins Datastore Version 11
    48. 48. Optimistic Locking App Server Data read begins DatastoreVersion 11 of data Version 11
    49. 49. Optimistic Locking Version 12 of data App Finished committed. App Server Data read begins Datastore has version 12. Server DatastoreVersion 11 of data Version 11
    50. 50. Optimistic Locking Version 12 of data App Finished committed. App Server Data read begins Datastore has version 12. Server DatastoreVersion 11 of data Version 12
    51. 51. Optimistic Locking Version 12 of data App Finished committed. App Server Data read begins Datastore has version 12. Server DatastoreVersion 11 of data Write data back Version 12
    52. 52. Optimistic Locking Version 12 of data App Finished committed. App Server Data read begins Datastore has version 12. Server DatastoreVersion 11 of data Write data back Version 12 Trying to write back to datastore: exception! Another client has modified data
    53. 53. Life of a distributed write 1. Writes to thejournal of multiple datastores App Server 2. Returns once datastores acknowledge receiving write
    54. 54. Life of a distributed write (part 2) The journal tracks writes that need to be applied Write being Item 1 <data> applied Version 25 Item 5 <data> Version 12 Write to be Item 1 <data>applied in future Version 26
    55. 55. Transactional reads Local datastore up to date AppServer Is the item caught up to the latest journal Tries to read from write? local datastore Journal Applied. Item 1 <data> Version 25 Applied. Item 5 <data> Version 12 Applied. Item 1 <data> Version 26
    56. 56. Transactional reads Local datastore up to date AppServer Is the item caught up to the latest journal Tries to read from write? local datastore Yes! Journal Applied. Item 1 <data> Return the Version 25 data in the Item 5 <data> Applied. Version 12 datastore Applied. Item 1 <data> Version 26
    57. 57. Transactional reads Local datastore not up to date - Step 1 App Is the item caught upServer to the latest journal write? Tries to read from local datastore Journal Applied. Item 1 <data> Version 25 Applied. Item 5 <data> Version 12 Unapplied. Item 1 <data> Version 26
    58. 58. Transactional reads Local datastore not up to date - Step 1 App Is the item caught upServer to the latest journal write? Tries to read from local datastore No. Journal Applied. Item 1 <data> Catch the Version 25 local data Item 5 <data> Applied. Version 12 up Unapplied. Item 1 <data> Version 26
    59. 59. Transactional reads Local datastore not up to date - Step 2 All datastores either: Waits for local return up-to-date data App datastore to return or force catch upServer Request data from remote datastore App uses data from first datastore that Request data from responds remote datastore
    60. 60. More reading• My example was grossly oversimplified• More details can be found here: http://www.cidrdb.org/cidr2011/Papers/ CIDR11_Paper32.pdf
    61. 61. Contradictory advice• Entity groups must be as big as possible to cover as much related data as you can• Entity groups must be small enough such that your write rate per entity group never goes above one write/second
    62. 62. Summary• Remember the rules of strong consistency vs. eventual consistency• Group your data into entity groups and use ancestor queries when possible• App Engine’s datastore gives you the best of all worlds: high reliability and strong data consistency
    63. 63. Questions?• Google+: http://plus.ikailan.com• Twitter: @ikai

    ×