SQL Considered Harmful Shannon Posniewski Cryptic Studios
<ul><li>Current Relational DBMSs are Inappropriate for Low-Latency, Controlled Working-Set Databases </li></ul><ul><li>and...
Database Fundamentals <ul><li>Structured collection of data </li></ul><ul><li>“ACID” rules </li></ul>
<ul><li>Atomicity </li></ul><ul><li>Consistency </li></ul><ul><li>Isolation </li></ul><ul><li>Durability </li></ul>
Database Types <ul><li>Relational </li></ul><ul><li>Object </li></ul><ul><li>Object-Relational </li></ul><ul><li>Hierarchi...
Relational Model <ul><li>Defined mathematically  </li></ul><ul><li>Based on </li></ul><ul><ul><li>set theory </li></ul></u...
Relational Database <ul><li>Tables </li></ul><ul><ul><li>Rows </li></ul></ul><ul><ul><ul><li>aka records </li></ul></ul></...
SQL <ul><li>computer language for manipulating relational databases </li></ul>
Unique and Foreign Keys <ul><li>Unique key in one table refers to other rows in another table. </li></ul><ul><li>Used to r...
Player ID PlayerName … AttribN 818 Warthog … foo 237 John … bar Player ID Item Count 818 Hat 1 818 Brooch 1 818 Pteradacty...
Class ID Name 1 Elf 2 Orc 3 Monkey Player ID Name … Class ID 818 Warthog … 2 237 John … 3
SQL DBMSs <ul><li>Provide all this and more </li></ul><ul><li>Have object extensions, BLOBs, etc </li></ul><ul><li>Used by...
<ul><li>Why not use it? </li></ul>
The Good <ul><li>SQL servers optimized for general case </li></ul><ul><li>Usually good at arbitrary queries </li></ul><ul>...
The Bad <ul><li>Long query set up time </li></ul><ul><li>Self-optimizes on the fly </li></ul><ul><li>Unpredictable latency...
The Ugly <ul><li>Inserts are SLOW </li></ul><ul><li>Per row overhead is high on updates </li></ul>
Object-Relational Impedance Mismatch <ul><li>Hierarchy and references are typical </li></ul><ul><li>Joins and normalizatio...
The Evils of Normalization <ul><li>Doing it the “the right way” can backfire </li></ul>
This was Bad Player ID Stat ID Value 818 Time Online 2,123 818 Fights Won 700 818 Fights Lost 3 … 818 Stat N Value N
This was Good Abandons relational DB concepts Player ID Stat 0 Value Stat 1 Value … Stat  N Value 818 2,123 200 … Value N ...
Solution: Caching <ul><li>Built a write-through cache </li></ul><ul><li>Reads come from the cache </li></ul><ul><li>Writes...
Not Enough <ul><li>Still too slow </li></ul><ul><li>Cache smoothes out latency issues </li></ul><ul><li>Throughput still a...
Reduce Throughput Needs <ul><li>Don’t save HP, location, etc </li></ul><ul><li>Snapshot instead </li></ul><ul><li>ACID is ...
Data Ownership <ul><li>A process “owns” an entity’s data </li></ul><ul><li>Vast majority of changes are local </li></ul><u...
Cross-Owner Transactions <ul><li>Flush both characters </li></ul><ul><li>Still a window of failure </li></ul><ul><li>We ju...
<ul><li>Optimized and played tricks on the database… </li></ul><ul><li>… and stopped getting the benefits of using one </l...
It’s just not the right tool
Cryptic DB
Requirements <ul><li>Regain ACID (as much as possible) </li></ul><ul><li>Low, predictable latency </li></ul><ul><li>high r...
Relaxations <ul><li>Offline character access can be slower </li></ul><ul><li>Domain integrity responsibility of programmer...
StructParser <ul><li>Structured text definition of data </li></ul><ul><li>Generated from C structs </li></ul><ul><li>Used ...
Example <ul><li>AUTO_STRUCT AST_CONTAINER; </li></ul><ul><li>typedef struct Entity </li></ul><ul><li>{ </li></ul><ul><li>E...
Struct Metadata <ul><li>Generates metadata on structs </li></ul><ul><ul><li>memory layout </li></ul></ul><ul><ul><li>data ...
Basic Operations <ul><li>Cryptic DB supports only a few basic operations </li></ul><ul><li>Simple transactions built-in  <...
Data and Logic Ownership <ul><li>A server “owns” the data for an entity </li></ul><ul><li>Knows how to modify these entiti...
Architecture Map Server Map Server Map Server Team Server Cryptic DB Transaction Server
Cryptic Engine Transactions <ul><li>Local if possible </li></ul><ul><li>Dispatched by Transaction Server if not </li></ul>...
AUTO_TRANSACTION <ul><li>C code plus special macros </li></ul>AUTO_TRANSACTION;  enumTransactionOutcome trGiveGold(ATR_ARG...
Step 1 Transaction Server Server 1 A Server 2 B DoTrade Trans(“DoTrade”, A, B)
Step 2 Transaction Server Server 1 Trans(“DoTrade”, A, B) A Server 2 B DoTrade 1 2 3 4 … 1 2 3 4 …
Step 3 Transaction Server Server 1 A Server 2 B DoTrade 1 2 3 4 … B
Step 4 Transaction Server Server 1 A Server 2 B DoTrade 1 2 3 4 … B A’ B’
Step 4 Transaction Server Server 1 A Server 2 B 1 2 3 4 … B A’ dB B’ dB B’
Step 4 Transaction Server Server 1 Server 2 1 2 3 4 … A’ B’
Abstract Cryptic DB <ul><li>Get transaction </li></ul><ul><li>Log it to journal </li></ul><ul><li>Apply change to in-memor...
We’re hiring!  Apply Online! Copyright 2008 Cryptic Studios, Inc.
Upcoming SlideShare
Loading in …5
×

Sql Considered Harmful

1,347 views
1,273 views

Published on

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,347
On SlideShare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
17
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • This presentation is copyright 2008 by Cryptic Studios. Please do not publically duplicate without permission. Foolish, provocative, title
  • Sql Considered Harmful

    1. 1. SQL Considered Harmful Shannon Posniewski Cryptic Studios
    2. 2. <ul><li>Current Relational DBMSs are Inappropriate for Low-Latency, Controlled Working-Set Databases </li></ul><ul><li>and what we did trying to circumvent that </li></ul><ul><li>and how that didn’t work out very well </li></ul><ul><li>and what we’re doing now </li></ul>Shannon Posniewski
    3. 3. Database Fundamentals <ul><li>Structured collection of data </li></ul><ul><li>“ACID” rules </li></ul>
    4. 4. <ul><li>Atomicity </li></ul><ul><li>Consistency </li></ul><ul><li>Isolation </li></ul><ul><li>Durability </li></ul>
    5. 5. Database Types <ul><li>Relational </li></ul><ul><li>Object </li></ul><ul><li>Object-Relational </li></ul><ul><li>Hierarchical </li></ul><ul><li>etc. </li></ul>
    6. 6. Relational Model <ul><li>Defined mathematically </li></ul><ul><li>Based on </li></ul><ul><ul><li>set theory </li></ul></ul><ul><ul><li>predicate logic </li></ul></ul>
    7. 7. Relational Database <ul><li>Tables </li></ul><ul><ul><li>Rows </li></ul></ul><ul><ul><ul><li>aka records </li></ul></ul></ul><ul><ul><ul><li>aka entities </li></ul></ul></ul><ul><ul><li>Columns </li></ul></ul><ul><ul><ul><li>values for that entity </li></ul></ul></ul>
    8. 8. SQL <ul><li>computer language for manipulating relational databases </li></ul>
    9. 9. Unique and Foreign Keys <ul><li>Unique key in one table refers to other rows in another table. </li></ul><ul><li>Used to represent hierarchical data </li></ul>
    10. 10. Player ID PlayerName … AttribN 818 Warthog … foo 237 John … bar Player ID Item Count 818 Hat 1 818 Brooch 1 818 Pteradactyl 1 237 Monkey Paw 1
    11. 11. Class ID Name 1 Elf 2 Orc 3 Monkey Player ID Name … Class ID 818 Warthog … 2 237 John … 3
    12. 12. SQL DBMSs <ul><li>Provide all this and more </li></ul><ul><li>Have object extensions, BLOBs, etc </li></ul><ul><li>Used by practically everyone </li></ul><ul><li>Safe, tested </li></ul><ul><li>Lots of experience </li></ul>
    13. 13. <ul><li>Why not use it? </li></ul>
    14. 14. The Good <ul><li>SQL servers optimized for general case </li></ul><ul><li>Usually good at arbitrary queries </li></ul><ul><li>Will always “work” </li></ul>
    15. 15. The Bad <ul><li>Long query set up time </li></ul><ul><li>Self-optimizes on the fly </li></ul><ul><li>Unpredictable latency </li></ul>
    16. 16. The Ugly <ul><li>Inserts are SLOW </li></ul><ul><li>Per row overhead is high on updates </li></ul>
    17. 17. Object-Relational Impedance Mismatch <ul><li>Hierarchy and references are typical </li></ul><ul><li>Joins and normalization are often slow </li></ul>
    18. 18. The Evils of Normalization <ul><li>Doing it the “the right way” can backfire </li></ul>
    19. 19. This was Bad Player ID Stat ID Value 818 Time Online 2,123 818 Fights Won 700 818 Fights Lost 3 … 818 Stat N Value N
    20. 20. This was Good Abandons relational DB concepts Player ID Stat 0 Value Stat 1 Value … Stat N Value 818 2,123 200 … Value N 237 5,235 60 … Value N 54 675 23 … Value N
    21. 21. Solution: Caching <ul><li>Built a write-through cache </li></ul><ul><li>Reads come from the cache </li></ul><ul><li>Writes go to cache and then to SQL </li></ul><ul><li>All active players in memory (fast) </li></ul>
    22. 22. Not Enough <ul><li>Still too slow </li></ul><ul><li>Cache smoothes out latency issues </li></ul><ul><li>Throughput still a problem </li></ul>
    23. 23. Reduce Throughput Needs <ul><li>Don’t save HP, location, etc </li></ul><ul><li>Snapshot instead </li></ul><ul><li>ACID is pretty much shot </li></ul>
    24. 24. Data Ownership <ul><li>A process “owns” an entity’s data </li></ul><ul><li>Vast majority of changes are local </li></ul><ul><li>Changes are periodically flushed to DB </li></ul>
    25. 25. Cross-Owner Transactions <ul><li>Flush both characters </li></ul><ul><li>Still a window of failure </li></ul><ul><li>We just did our best </li></ul>
    26. 26. <ul><li>Optimized and played tricks on the database… </li></ul><ul><li>… and stopped getting the benefits of using one </li></ul>
    27. 27. It’s just not the right tool
    28. 28. Cryptic DB
    29. 29. Requirements <ul><li>Regain ACID (as much as possible) </li></ul><ul><li>Low, predictable latency </li></ul><ul><li>high read/write throughput </li></ul><ul><li>minimize code-database impedance </li></ul><ul><li>friendly to schema changes </li></ul><ul><li>easy backup and restore </li></ul>
    30. 30. Relaxations <ul><li>Offline character access can be slower </li></ul><ul><li>Domain integrity responsibility of programmer </li></ul><ul><li>General ad hoc queries can be slower </li></ul><ul><li>SQL does not need to be supported </li></ul>
    31. 31. StructParser <ul><li>Structured text definition of data </li></ul><ul><li>Generated from C structs </li></ul><ul><li>Used for serializing static data </li></ul><ul><li>Unifies static and DB data </li></ul>
    32. 32. Example <ul><li>AUTO_STRUCT AST_CONTAINER; </li></ul><ul><li>typedef struct Entity </li></ul><ul><li>{ </li></ul><ul><li>EntityRef myRef; NO_AST </li></ul><ul><li>const ContainerID myContainerID; AST(PERSIST, KEY) </li></ul><ul><li>Vec3 pos; AST(PERSIST, SERVER_ONLY, NO_TRANSACT) </li></ul><ul><ul><li>… </li></ul></ul><ul><li>} Entity; </li></ul>
    33. 33. Struct Metadata <ul><li>Generates metadata on structs </li></ul><ul><ul><li>memory layout </li></ul></ul><ul><ul><li>data types </li></ul></ul><ul><li>Generic mutators are possible </li></ul><ul><ul><li>Cryptic DB </li></ul></ul>
    34. 34. Basic Operations <ul><li>Cryptic DB supports only a few basic operations </li></ul><ul><li>Simple transactions built-in </li></ul><ul><li>More complicated things need code </li></ul>
    35. 35. Data and Logic Ownership <ul><li>A server “owns” the data for an entity </li></ul><ul><li>Knows how to modify these entities </li></ul>
    36. 36. Architecture Map Server Map Server Map Server Team Server Cryptic DB Transaction Server
    37. 37. Cryptic Engine Transactions <ul><li>Local if possible </li></ul><ul><li>Dispatched by Transaction Server if not </li></ul><ul><li>AUTO_TRANSACTIONS in C </li></ul><ul><li>Field-level locking </li></ul>
    38. 38. AUTO_TRANSACTION <ul><li>C code plus special macros </li></ul>AUTO_TRANSACTION; enumTransactionOutcome trGiveGold(ATR_ARGS, NOCONST(Entity) *pEnt1, NOCONST(Entity) *pEnt2, int amount) { if (pEnt1->gold < amount) TRANSACTION_RETURN(TRANSACTION_OUTCOME_FAILURE, ” NotEnoughGold”); pEnt1->gold -= amount; pEnt2->gold += amount; return TRANSACTION_OUTCOME_SUCCESS; }
    39. 39. Step 1 Transaction Server Server 1 A Server 2 B DoTrade Trans(“DoTrade”, A, B)
    40. 40. Step 2 Transaction Server Server 1 Trans(“DoTrade”, A, B) A Server 2 B DoTrade 1 2 3 4 … 1 2 3 4 …
    41. 41. Step 3 Transaction Server Server 1 A Server 2 B DoTrade 1 2 3 4 … B
    42. 42. Step 4 Transaction Server Server 1 A Server 2 B DoTrade 1 2 3 4 … B A’ B’
    43. 43. Step 4 Transaction Server Server 1 A Server 2 B 1 2 3 4 … B A’ dB B’ dB B’
    44. 44. Step 4 Transaction Server Server 1 Server 2 1 2 3 4 … A’ B’
    45. 45. Abstract Cryptic DB <ul><li>Get transaction </li></ul><ul><li>Log it to journal </li></ul><ul><li>Apply change to in-memory DB </li></ul><ul><li>Occasionally write out updated DB file </li></ul><ul><li>Mark the journal resolved to that point </li></ul>
    46. 46. We’re hiring! Apply Online! Copyright 2008 Cryptic Studios, Inc.

    ×