Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Sql Considered Harmful

1,657 views

Published on

Published in: Technology
  • Be the first to comment

Sql Considered Harmful

  1. 1. SQL Considered Harmful Shannon Posniewski Cryptic Studios
  2. 2. <ul><li>Current Relational DBMSs are Inappropriate for Low-Latency, Controlled Working-Set Databases </li></ul><ul><li>and what we did trying to circumvent that </li></ul><ul><li>and how that didn’t work out very well </li></ul><ul><li>and what we’re doing now </li></ul>Shannon Posniewski
  3. 3. Database Fundamentals <ul><li>Structured collection of data </li></ul><ul><li>“ACID” rules </li></ul>
  4. 4. <ul><li>Atomicity </li></ul><ul><li>Consistency </li></ul><ul><li>Isolation </li></ul><ul><li>Durability </li></ul>
  5. 5. Database Types <ul><li>Relational </li></ul><ul><li>Object </li></ul><ul><li>Object-Relational </li></ul><ul><li>Hierarchical </li></ul><ul><li>etc. </li></ul>
  6. 6. Relational Model <ul><li>Defined mathematically </li></ul><ul><li>Based on </li></ul><ul><ul><li>set theory </li></ul></ul><ul><ul><li>predicate logic </li></ul></ul>
  7. 7. Relational Database <ul><li>Tables </li></ul><ul><ul><li>Rows </li></ul></ul><ul><ul><ul><li>aka records </li></ul></ul></ul><ul><ul><ul><li>aka entities </li></ul></ul></ul><ul><ul><li>Columns </li></ul></ul><ul><ul><ul><li>values for that entity </li></ul></ul></ul>
  8. 8. SQL <ul><li>computer language for manipulating relational databases </li></ul>
  9. 9. Unique and Foreign Keys <ul><li>Unique key in one table refers to other rows in another table. </li></ul><ul><li>Used to represent hierarchical data </li></ul>
  10. 10. Player ID PlayerName … AttribN 818 Warthog … foo 237 John … bar Player ID Item Count 818 Hat 1 818 Brooch 1 818 Pteradactyl 1 237 Monkey Paw 1
  11. 11. Class ID Name 1 Elf 2 Orc 3 Monkey Player ID Name … Class ID 818 Warthog … 2 237 John … 3
  12. 12. SQL DBMSs <ul><li>Provide all this and more </li></ul><ul><li>Have object extensions, BLOBs, etc </li></ul><ul><li>Used by practically everyone </li></ul><ul><li>Safe, tested </li></ul><ul><li>Lots of experience </li></ul>
  13. 13. <ul><li>Why not use it? </li></ul>
  14. 14. The Good <ul><li>SQL servers optimized for general case </li></ul><ul><li>Usually good at arbitrary queries </li></ul><ul><li>Will always “work” </li></ul>
  15. 15. The Bad <ul><li>Long query set up time </li></ul><ul><li>Self-optimizes on the fly </li></ul><ul><li>Unpredictable latency </li></ul>
  16. 16. The Ugly <ul><li>Inserts are SLOW </li></ul><ul><li>Per row overhead is high on updates </li></ul>
  17. 17. Object-Relational Impedance Mismatch <ul><li>Hierarchy and references are typical </li></ul><ul><li>Joins and normalization are often slow </li></ul>
  18. 18. The Evils of Normalization <ul><li>Doing it the “the right way” can backfire </li></ul>
  19. 19. This was Bad Player ID Stat ID Value 818 Time Online 2,123 818 Fights Won 700 818 Fights Lost 3 … 818 Stat N Value N
  20. 20. This was Good Abandons relational DB concepts Player ID Stat 0 Value Stat 1 Value … Stat N Value 818 2,123 200 … Value N 237 5,235 60 … Value N 54 675 23 … Value N
  21. 21. Solution: Caching <ul><li>Built a write-through cache </li></ul><ul><li>Reads come from the cache </li></ul><ul><li>Writes go to cache and then to SQL </li></ul><ul><li>All active players in memory (fast) </li></ul>
  22. 22. Not Enough <ul><li>Still too slow </li></ul><ul><li>Cache smoothes out latency issues </li></ul><ul><li>Throughput still a problem </li></ul>
  23. 23. Reduce Throughput Needs <ul><li>Don’t save HP, location, etc </li></ul><ul><li>Snapshot instead </li></ul><ul><li>ACID is pretty much shot </li></ul>
  24. 24. Data Ownership <ul><li>A process “owns” an entity’s data </li></ul><ul><li>Vast majority of changes are local </li></ul><ul><li>Changes are periodically flushed to DB </li></ul>
  25. 25. Cross-Owner Transactions <ul><li>Flush both characters </li></ul><ul><li>Still a window of failure </li></ul><ul><li>We just did our best </li></ul>
  26. 26. <ul><li>Optimized and played tricks on the database… </li></ul><ul><li>… and stopped getting the benefits of using one </li></ul>
  27. 27. It’s just not the right tool
  28. 28. Cryptic DB
  29. 29. Requirements <ul><li>Regain ACID (as much as possible) </li></ul><ul><li>Low, predictable latency </li></ul><ul><li>high read/write throughput </li></ul><ul><li>minimize code-database impedance </li></ul><ul><li>friendly to schema changes </li></ul><ul><li>easy backup and restore </li></ul>
  30. 30. Relaxations <ul><li>Offline character access can be slower </li></ul><ul><li>Domain integrity responsibility of programmer </li></ul><ul><li>General ad hoc queries can be slower </li></ul><ul><li>SQL does not need to be supported </li></ul>
  31. 31. StructParser <ul><li>Structured text definition of data </li></ul><ul><li>Generated from C structs </li></ul><ul><li>Used for serializing static data </li></ul><ul><li>Unifies static and DB data </li></ul>
  32. 32. Example <ul><li>AUTO_STRUCT AST_CONTAINER; </li></ul><ul><li>typedef struct Entity </li></ul><ul><li>{ </li></ul><ul><li>EntityRef myRef; NO_AST </li></ul><ul><li>const ContainerID myContainerID; AST(PERSIST, KEY) </li></ul><ul><li>Vec3 pos; AST(PERSIST, SERVER_ONLY, NO_TRANSACT) </li></ul><ul><ul><li>… </li></ul></ul><ul><li>} Entity; </li></ul>
  33. 33. Struct Metadata <ul><li>Generates metadata on structs </li></ul><ul><ul><li>memory layout </li></ul></ul><ul><ul><li>data types </li></ul></ul><ul><li>Generic mutators are possible </li></ul><ul><ul><li>Cryptic DB </li></ul></ul>
  34. 34. Basic Operations <ul><li>Cryptic DB supports only a few basic operations </li></ul><ul><li>Simple transactions built-in </li></ul><ul><li>More complicated things need code </li></ul>
  35. 35. Data and Logic Ownership <ul><li>A server “owns” the data for an entity </li></ul><ul><li>Knows how to modify these entities </li></ul>
  36. 36. Architecture Map Server Map Server Map Server Team Server Cryptic DB Transaction Server
  37. 37. Cryptic Engine Transactions <ul><li>Local if possible </li></ul><ul><li>Dispatched by Transaction Server if not </li></ul><ul><li>AUTO_TRANSACTIONS in C </li></ul><ul><li>Field-level locking </li></ul>
  38. 38. AUTO_TRANSACTION <ul><li>C code plus special macros </li></ul>AUTO_TRANSACTION; enumTransactionOutcome trGiveGold(ATR_ARGS, NOCONST(Entity) *pEnt1, NOCONST(Entity) *pEnt2, int amount) { if (pEnt1->gold < amount) TRANSACTION_RETURN(TRANSACTION_OUTCOME_FAILURE, ” NotEnoughGold”); pEnt1->gold -= amount; pEnt2->gold += amount; return TRANSACTION_OUTCOME_SUCCESS; }
  39. 39. Step 1 Transaction Server Server 1 A Server 2 B DoTrade Trans(“DoTrade”, A, B)
  40. 40. Step 2 Transaction Server Server 1 Trans(“DoTrade”, A, B) A Server 2 B DoTrade 1 2 3 4 … 1 2 3 4 …
  41. 41. Step 3 Transaction Server Server 1 A Server 2 B DoTrade 1 2 3 4 … B
  42. 42. Step 4 Transaction Server Server 1 A Server 2 B DoTrade 1 2 3 4 … B A’ B’
  43. 43. Step 4 Transaction Server Server 1 A Server 2 B 1 2 3 4 … B A’ dB B’ dB B’
  44. 44. Step 4 Transaction Server Server 1 Server 2 1 2 3 4 … A’ B’
  45. 45. Abstract Cryptic DB <ul><li>Get transaction </li></ul><ul><li>Log it to journal </li></ul><ul><li>Apply change to in-memory DB </li></ul><ul><li>Occasionally write out updated DB file </li></ul><ul><li>Mark the journal resolved to that point </li></ul>
  46. 46. We’re hiring! Apply Online! Copyright 2008 Cryptic Studios, Inc.

×