Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Charlotte Software Engineering Meetup - Jan.2015

449 views

Published on

The slides for the talk at Charlotte Software Engineering meetup.

http://www.meetup.com/Charlotte-Software-Engineering-Meetup/events/227454809/

Published in: Technology
  • Be the first to comment

Charlotte Software Engineering Meetup - Jan.2015

  1. 1. Polyglot Persistence
  2. 2. Majid Fatemian R E D V E N T U R E S @majidfn
  3. 3. Relational Non-Relational Key-Value Big Data
  4. 4. Application Relational
  5. 5. Application Relational Relational Non-Relational
  6. 6. Relational
  7. 7. DATA
  8. 8. BIG DATA
  9. 9. Capture Search Storage Q u e r y Privacy Visualize Analysis
  10. 10. Non-Relational
  11. 11. 1 2 3 Non-Relational Cluster Schemaless
  12. 12. I. Document II. Key-Value III. Graph IV. Column MongoDB, CouchBase Memcache, Redis Neo4J, OrientDB Cassandra, HBase
  13. 13. P C A AvailabilityConsistency Partition Tolerance
  14. 14. P A AvailabilityConsistency Partition Tolerance C
  15. 15. CP CA AP MySQL Postgres MongoDBCouchBase Cassandra Riak
  16. 16. 2 x 1 x 1 x
  17. 17. 2011 2012 2013 2014
  18. 18. 2011 2012 2013 201
  19. 19. Application
  20. 20. Normalized ACID {Consistency Relational
  21. 21. Replication Fault Tolerance Availability {Relational
  22. 22. OPs / DBAs Tools Devs{DevOps Relational
  23. 23. 2011 2012 2013 201
  24. 24. Scalability Relational
  25. 25. Schema Relational {
 player_id: 1234,
 * level: 40,
 * duration: 75, * activity: activity,
 game_id: 4001,
 platform: WiiU, difficulty_level: medium,
 repetition: 178,
 score: 450,
 endurance:3
 } {
 player_id: 1234,
 level: 12,
 duration: 120, activity: activity
 }
  26. 26. Aggregated Detailed stats stats Players Profiles Individual Activities
  27. 27. Distribution Relational Data C Data B Data A Data Access Layer
  28. 28. ACID Transaction Partitioning Replication Tools OPs Data Volume Fixed Schema Scale Up Manual Sharding + -
  29. 29. NoSQL ...
  30. 30. mongoDB 1 2 3 OPs Devs Community
  31. 31. Schema-less Non-Relational player_id: 1234,
 * level: 40,
 * duration: 75, * activity: activity, game_id: 4001,
 platform: WiiU, difficulty_level: 2,
 repetition: 178,
 score: 450,
 endurance:3 player_id: 1234,
 level: 75,
 duration: 120, activity: activity

  32. 32. Scalability Non-Relational
  33. 33. Scalability Non-Relational +
  34. 34. Sharding Non-Relational
  35. 35. Sharding Non-Relational Data A {key}
  36. 36. Sharding Non-Relational Data A A Data B B Data C C {key}
  37. 37. Map/Reduce Non-Relational K, V K,V1 K,V2 K,V3
  38. 38. Map/Reduce Non-Relational Query K, V K,V1 K,V2 K,V3
  39. 39. Map/Reduce Non-Relational Query Map K, V K,V1 K,V2 K,V3
  40. 40. Map/Reduce Non-Relational Map K,V1 K,V2 K,V3
  41. 41. Map/Reduce Non-Relational Result ReduceMap K,V1 K,V2 K,V3
  42. 42. Denormalized Non-Relationalread / }write Disk Space
  43. 43. Data Model Non-RelationalQuery Aggregation }
  44. 44. Eventual- Consistency Non-Relational X1 X1 X1
  45. 45. Eventual- Consistency Non-Relational X2 W X1 X1 X2
  46. 46. Eventual- Consistency Non-Relational X2 W X1 X1 R X1 X2
  47. 47. Eventual- Consistency Non-Relational X2 W X2 X2 R X2 X2
  48. 48. Balancer Non-Relational
  49. 49. Balancer Non-Relational +
  50. 50. Balancer Non-Relational
  51. 51. Non-Relational Resilience A B C
  52. 52. Non-Relational Resilience A C
  53. 53. Locking Non-RelationalDB Level (2.2) }Document Level (3.x)
  54. 54. Backup Non-Relational Point-in-time
  55. 55. Denormalized data Disk Space Expertise Complex Querying
 Eventual Consistency Resource Usage DB-Level Locking - Schema-Less Aggregated Data Large Scale Data Sharding Map/Reduce Memory Storage Journaling +
  56. 56. Random Profile
 Search
 (1M) Full History Retrieval
 (6M) Insert
 (6M) Aggregation
 Map/Reduce
 (6M) 0 0.003 0.005 0.008 0.01 MySQL MongoDB
  57. 57. Persistence Polyglot
  58. 58. Players Profile Leaderboards Activity Details
  59. 59. Total Sum Activity 1 Activity n Activity m ……
  60. 60. Shard1 Shard2 Config
  61. 61. Data Access Layer Application
  62. 62. The Challenge
  63. 63. Inconsistency1
  64. 64. Data Access Layer Commit 2-Phase
  65. 65. Data Access Layer W Pending Commit 2-Phase
  66. 66. Data Access Layer Pending W Commit 2-Phase
  67. 67. Data Access Layer Done Commit 2-Phase
  68. 68. 2 Latency
  69. 69. 2 Latency vs. Consistency
  70. 70. Operations3
  71. 71. Operations3 Exponential complexity
  72. 72. Availability4
  73. 73. Availability4 Plan for failure
  74. 74. Data Access Layer
  75. 75. Data Access Layer
  76. 76. Data Access Layer
  77. 77. Data Integrity5
  78. 78. Data Integrity5 Backward compatibility
  79. 79. Unit {TEST Integration Functional
  80. 80. Data Migration6
  81. 81. Profile History } { profile: { GUID:…, platform: { name:’xbox’, xuid:… } }, activity:…, duration:…, datetime:… } 3rd Party
  82. 82. 1 One-Shot Migration
  83. 83. 1 One-Shot Migration 4 hours !!
  84. 84. 1 One-Shot Migration 4 hours !! Live !! 2 Profile versioning
  85. 85. +1
  86. 86. V +1 1
  87. 87. V +1 1
  88. 88. V +1 2
  89. 89. Final Results
  90. 90. Scalability 2011 2012 2013 2014
  91. 91. 9x response time
  92. 92. 3 years uptime
  93. 93. Go Hybrid!>
  94. 94. Go Hybrid! Thoughtfully >
  95. 95. Thank YOU! @majidfn

×