Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Polyglot
Databases
php[tek] 2015
@majidfn
Majid Fatemian
Ubisoft Montreal
Relational
DATA
BIG
DATA
NoSQL
1 2 3
Non-Relational Cluster Schemaless
1. Document
2. Key-Value
3. Graph
4. Column
MongoDB, CouchBase
Memcache, Redis
Neo4J, OrientDB
Cassandra, HBase
A
C P
Availability
Consistency Partition Tolerance
Consistency vailability
A
artition
PConsistency
vailability
A artition
P
MongoDB
MySQL
Postgres
CouchBase
Cassandra
Riak
Fitness Evolved
Fitness Evolved
Fitness Evolved
Fitness Evolved
3M 15K 160K
Players CCU Daily Ses.
STATS
2011
2012
2013
2014
2011
2012
2013
201
Application
Normalized
ACID
Transaction{Consistency
Relational
Replication
Table Partitioning
Availability
{Relational
OPs / DBAs
Tools
Devs{DevOps
Relational
2011
2012
2013
201
Aggregated
…
Detailed
Profile
Community
History
Leaderboards
Single Point Failure Availability
} Relational
Scalability
Relational
Schema
Relational
{

player_id: 1234,

* calories: 40,

* duration: 75,
* activity: jumping_ropes,

game_id: 4001,

platfo...
Player Game Cals Duration Activity Difficulty Score Reps.
1000 1 100 80 Box - - -
2000 2 90 100 Jump H 4.5 180
1000 2 80 120...
Player Game Cals Duration Activity Difficulty Score Reps.
1000 1 100 80 Box - - -
2000 2 90 100 Jump H 4.5 180
1000 2 80 120...
Distribution
Relational
Data C
Data B
Data A
Data Access Layer
ACID
Transaction
Partitioning
Replication
Tools
OPs
Data Volume
Fixed Schema
Scale Up
Manual Sharding
+ -
NoSQL
...
mongoDB
1 2 3
OPs DEVs Community
Schemaless
Non-Relational
player_id: 1234,

* calories: 40,

* duration: 75,
* activity: jumps,
game_id: 4001,

platform: ...
Scalability
Non-Relational
Scalability
Non-Relational
+
Sharding
Non-Relational
Sharding
Non-Relational
Data A
A
Data B
B Data CC
•DB Managed
•Balanced Load
Sharding
Non-Relational
•Sharding key
•Bouncing
•Decreased resilience
Map/Reduce
Non-Relational
R1 R2 R3
Map/Reduce
Non-Relational
QueryQueryQuery
R1 R2 R3
Map/Reduce
Non-Relational
QueryQuery Query
R1 R2 R3
Map
Map/Reduce
Non-Relational
QueryQuery Query
R1 R2 R3
Result
ReduceMap
Denormalized
Non-Relationalread /
}write
Disk Space
Query
Non-RelationalData Model
Aggregation
}
Eventual-
Consistency
Non-Relational
X1
X1 X1
Eventual-
Consistency
Non-Relational
X2
W
X1 X1
X2
Eventual-
Consistency
Non-Relational
X2
W
X1 X1
R
X1
X2
Eventual-
Consistency
Non-Relational
X2
W
X2
X2
R
X2
X2
Locking
Non-RelationalDB Level (2.2)
}Document Level (3.0)
Backup
Non-Relational
Point-in-time
Denormalized data
Disk Space
Expertise
Complex Querying

Eventual Consistency
Resource Usage
DB-Level Locking
-
Schema-Les...
Random Profile

Search

(1M)
Full History
Retrieval

(6M)
Insert

(6M)
Aggregation

Map/Reduce

(6M)
0
0.003
0.005
0.008
0....
Persistence
Polyglot
Profile (Total Sum)
Leaderboards
…
Activity Details
Activity Aggregations
Master
Slave
Config Server
Shard 1
Replicaset
Shard 2
Replicaset
Data Access Layer
The
Challenge
Inconsistency1
Data Access Layer
Commit
2-Phase
Data Access Layer
W
Pending
Commit
2-Phase
Data Access Layer
Pending
W
Commit
2-Phase
Data Access Layer
Done
Commit
2-Phase
Consistency
vs.
Latency
2
Operations3
Availability4
Data Access Layer
Data Access Layer
Data Access Layer
Data Integrity5
Unit
{TEST Integration
Functional
DATA
Migration
Profile
Activity }3rd Party
{
profile:
{
GUID:…,
platform:
{
name:’xbox’,
xuid:…
}
},
activity:…,
duration:…,
datetime:…
}
1 One-Shot Migration
2 Profile Versioning
4
hrs
!!
Live
!!
+1
R
W
V
+1
1
R
W
V
+1
2
Final
Results
2011 2012 2013 2014
9x response time
3 years uptime
> Go Hybrid, wisely!
> Go Hybrid, wisely!
Denormalized data

Eventual Consistency
Evolving
Schema-Less
Large Scale Data
Sharding
Map/Reduce
ACI...
Thank
YOU!
@majidfn
http://joind.in/13728
PHP[tek] - 2015
PHP[tek] - 2015
Upcoming SlideShare
Loading in …5
×

PHP[tek] - 2015

427 views

Published on

NoSQL databases are everywhere and they're here to stay. For a successful scalable web application, it is vital to understand the capabilities of NoSQL databases. Relational databases are not dead, either, and they will not be. Understanding the abilities of each will help you to make better architectural decisions. Hybrid database solutions could use the best of the two worlds. In this talk, we will delve into the important decision making factors for each. As an example, we will study an online game at Ubisoft Montreal that switched from a SQL database to a polyglot persistence with no data loss and no downtime.

Published in: Internet
  • Be the first to comment

  • Be the first to like this

PHP[tek] - 2015

  1. 1. Polyglot Databases php[tek] 2015
  2. 2. @majidfn Majid Fatemian Ubisoft Montreal
  3. 3. Relational
  4. 4. DATA
  5. 5. BIG DATA
  6. 6. NoSQL
  7. 7. 1 2 3 Non-Relational Cluster Schemaless
  8. 8. 1. Document 2. Key-Value 3. Graph 4. Column MongoDB, CouchBase Memcache, Redis Neo4J, OrientDB Cassandra, HBase
  9. 9. A C P Availability Consistency Partition Tolerance
  10. 10. Consistency vailability A artition PConsistency vailability A artition P MongoDB MySQL Postgres CouchBase Cassandra Riak
  11. 11. Fitness Evolved
  12. 12. Fitness Evolved
  13. 13. Fitness Evolved
  14. 14. Fitness Evolved
  15. 15. 3M 15K 160K Players CCU Daily Ses. STATS
  16. 16. 2011 2012 2013 2014
  17. 17. 2011 2012 2013 201
  18. 18. Application
  19. 19. Normalized ACID Transaction{Consistency Relational
  20. 20. Replication Table Partitioning Availability {Relational
  21. 21. OPs / DBAs Tools Devs{DevOps Relational
  22. 22. 2011 2012 2013 201
  23. 23. Aggregated … Detailed Profile Community History Leaderboards
  24. 24. Single Point Failure Availability } Relational
  25. 25. Scalability Relational
  26. 26. Schema Relational {
 player_id: 1234,
 * calories: 40,
 * duration: 75, * activity: jumping_ropes,
 game_id: 4001,
 platform: WiiU, difficulty_level: medium,
 repetition: 178,
 score: 450,
 endurance:3
 } {
 player_id: 1234,
 calories: 75,
 duration: 120, activity: boxing
 }
  27. 27. Player Game Cals Duration Activity Difficulty Score Reps. 1000 1 100 80 Box - - - 2000 2 90 100 Jump H 4.5 180 1000 2 80 120 Pushup M 4 130 2000 1 120 110 Situps - - -
  28. 28. Player Game Cals Duration Activity Difficulty Score Reps. 1000 1 100 80 Box - - - 2000 2 90 100 Jump H 4.5 180 1000 2 80 120 Pushup M 4 130 2000 1 120 110 Situps - - -
  29. 29. Distribution Relational Data C Data B Data A Data Access Layer
  30. 30. ACID Transaction Partitioning Replication Tools OPs Data Volume Fixed Schema Scale Up Manual Sharding + -
  31. 31. NoSQL ...
  32. 32. mongoDB 1 2 3 OPs DEVs Community
  33. 33. Schemaless Non-Relational player_id: 1234,
 * calories: 40,
 * duration: 75, * activity: jumps, game_id: 4001,
 platform: WiiU, difficulty_level: 2,
 repetition: 178,
 score: 450,
 endurance:3 player_id: 1234,
 calories: 75,
 duration: 120, activity: boxing

  34. 34. Scalability Non-Relational
  35. 35. Scalability Non-Relational +
  36. 36. Sharding Non-Relational
  37. 37. Sharding Non-Relational Data A A Data B B Data CC
  38. 38. •DB Managed •Balanced Load Sharding Non-Relational •Sharding key •Bouncing •Decreased resilience
  39. 39. Map/Reduce Non-Relational R1 R2 R3
  40. 40. Map/Reduce Non-Relational QueryQueryQuery R1 R2 R3
  41. 41. Map/Reduce Non-Relational QueryQuery Query R1 R2 R3 Map
  42. 42. Map/Reduce Non-Relational QueryQuery Query R1 R2 R3 Result ReduceMap
  43. 43. Denormalized Non-Relationalread / }write Disk Space
  44. 44. Query Non-RelationalData Model Aggregation }
  45. 45. Eventual- Consistency Non-Relational X1 X1 X1
  46. 46. Eventual- Consistency Non-Relational X2 W X1 X1 X2
  47. 47. Eventual- Consistency Non-Relational X2 W X1 X1 R X1 X2
  48. 48. Eventual- Consistency Non-Relational X2 W X2 X2 R X2 X2
  49. 49. Locking Non-RelationalDB Level (2.2) }Document Level (3.0)
  50. 50. Backup Non-Relational Point-in-time
  51. 51. Denormalized data Disk Space Expertise Complex Querying
 Eventual Consistency Resource Usage DB-Level Locking - Schema-Less Aggregated Data Large Scale Data Sharding Map/Reduce Memory Storage Journaling +
  52. 52. Random Profile
 Search
 (1M) Full History Retrieval
 (6M) Insert
 (6M) Aggregation
 Map/Reduce
 (6M) 0 0.003 0.005 0.008 0.01 MySQL MongoDB
  53. 53. Persistence Polyglot
  54. 54. Profile (Total Sum) Leaderboards … Activity Details Activity Aggregations
  55. 55. Master Slave Config Server Shard 1 Replicaset Shard 2 Replicaset
  56. 56. Data Access Layer
  57. 57. The Challenge
  58. 58. Inconsistency1
  59. 59. Data Access Layer Commit 2-Phase
  60. 60. Data Access Layer W Pending Commit 2-Phase
  61. 61. Data Access Layer Pending W Commit 2-Phase
  62. 62. Data Access Layer Done Commit 2-Phase
  63. 63. Consistency vs. Latency 2
  64. 64. Operations3
  65. 65. Availability4
  66. 66. Data Access Layer
  67. 67. Data Access Layer
  68. 68. Data Access Layer
  69. 69. Data Integrity5
  70. 70. Unit {TEST Integration Functional
  71. 71. DATA Migration
  72. 72. Profile Activity }3rd Party { profile: { GUID:…, platform: { name:’xbox’, xuid:… } }, activity:…, duration:…, datetime:… }
  73. 73. 1 One-Shot Migration 2 Profile Versioning 4 hrs !! Live !!
  74. 74. +1
  75. 75. R W V +1 1
  76. 76. R W V +1 2
  77. 77. Final Results
  78. 78. 2011 2012 2013 2014
  79. 79. 9x response time
  80. 80. 3 years uptime
  81. 81. > Go Hybrid, wisely!
  82. 82. > Go Hybrid, wisely! Denormalized data
 Eventual Consistency Evolving Schema-Less Large Scale Data Sharding Map/Reduce ACID Transaction Replication Tools, OPs Data Volume Fixed Schema Scaling
  83. 83. Thank YOU! @majidfn http://joind.in/13728

×