FleetDB A Schema-Free Database in Clojure

3,223 views
3,014 views

Published on

Published in: Technology, Business
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,223
On SlideShare
0
From Embeds
0
Number of Embeds
32
Actions
Shares
0
Downloads
22
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

FleetDB A Schema-Free Database in Clojure

  1. 1. FleetDB A Schema-Free Database in Clojure Mark McGranaghan January 8, 2010
  2. 2. Motivation Walkthrough Implementation QA
  3. 3. Motivation
  4. 4. Motivation optimize for agile development
  5. 5. Walkthrough Client Quickstart Basic Queries Concurrency Tools
  6. 6. Client Quickstart (use ’fleetdb.client) (def client (connect)) (client ["ping"]) => "pong"
  7. 7. Insert (client ["insert" "accounts" {"id" 1 "owner" "Eve" "credits" 100}])
  8. 8. Insert (client ["insert" "accounts" {"id" 1 "owner" "Eve" "credits" 100}]) => 1
  9. 9. Select (client ["select" "accounts" {"where" ["=" "id" 1]}])
  10. 10. Select (client ["select" "accounts" {"where" ["=" "id" 1]}]) => [{"id" 1 "owner" "Eve" "credits" 100}]
  11. 11. Select with Conditions (client ["select" "accounts" {"where" [">=" "credits" 150]}])
  12. 12. Select with Order and Limit (client ["select" "accounts" {"order" ["credits" "asc"] "limit" 2}])
  13. 13. Update (client ["update" "accounts" {"credits" 105} {"where" ["=" "owner" "Eve"]}])
  14. 14. Explain (I) (client ["explain" ["select" "accounts" {"where" ["=" "owner" "Eve"]}]])
  15. 15. Explain (I) (client ["explain" ["select" "accounts" {"where" ["=" "owner" "Eve"]}]]) => ["filter" ["=" "owner" "Eve"] ["collection-scan" "accounts"]]
  16. 16. Create Index (client ["create-index" "accounts" "owner"])
  17. 17. Explain (II) (client ["explain" ["select" "accounts" {"where" ["=" "owner" "Eve"]}]]) => ["index-lookup" ["accounts" "owner" "Eve"]]]
  18. 18. Multi-Write
  19. 19. Multi-Write (client ["multi-write" [write-query-1 write-query-2 ...]])
  20. 20. Multi-Write (client ["multi-write" [write-query-1 write-query-2 ...]]) => [result-1 result-2 ...]
  21. 21. Checked-Write
  22. 22. Checked-Write (client ["checked-write" read-query expected-read-result write-query])
  23. 23. Checked-Write (client ["checked-write" read-query expected-read-result write-query]) => [true write-result]
  24. 24. Checked-Write (client ["checked-write" read-query expected-read-result write-query]) => [true write-result] => [false actual-read-result]
  25. 25. Implementation Background Organization Key ideas
  26. 26. Background http://clojure.org/data_structures http://clojure.org/sequences http://clojure.org/state
  27. 27. Organization
  28. 28. Organization fleetdb.core: pure functions
  29. 29. Organization fleetdb.core: pure functions fleetdb.embedded: identity and durability
  30. 30. Organization fleetdb.core: pure functions fleetdb.embedded: identity and durability fleetdb.server: network interface
  31. 31. fleetdb.core Pure functions
  32. 32. fleetdb.core Pure functions Databases as Clojure data structures
  33. 33. fleetdb.core Pure functions Databases as Clojure data structures Read: db + query → result
  34. 34. fleetdb.core Pure functions Databases as Clojure data structures Read: db + query → result ‘Write’: db + query → result + new db
  35. 35. fleetdb.core Query Planner
  36. 36. fleetdb.core Query Planner Implements declarative queries
  37. 37. fleetdb.core Query Planner Implements declarative queries Database + query → plan
  38. 38. fleetdb.core Query Planner Implements declarative queries Database + query → plan ["select" "accounts" {"where" ["=" "owner" "Eve"]}]
  39. 39. fleetdb.core Query Planner Implements declarative queries Database + query → plan ["select" "accounts" {"where" ["=" "owner" "Eve"]}] ["filter" ["=" "owner" "Eve"] ["record-scan" "accounts"]]
  40. 40. fleetdb.core Query Planner Implements declarative queries Database + query → plan ["select" "accounts" {"where" ["=" "owner" "Eve"]}] ["filter" ["=" "owner" "Eve"] ["record-scan" "accounts"]] ["index-lookup" ["accounts" "owner" "Eve"]]
  41. 41. fleetdb.core Query Executor
  42. 42. fleetdb.core Query Executor Database + plan → result
  43. 43. fleetdb.core Query Executor Database + plan → result ["filter" ["=" "owner" "Eve"] ["record-scan" "accounts"]]
  44. 44. fleetdb.core Query Executor Database + plan → result ["filter" ["=" "owner" "Eve"] ["record-scan" "accounts"]] (filter (fn [r] (= (r "owner") "Eve")) (vals (:rmap (db "accounts"))))
  45. 45. fleetdb.embedded Wraps fleetdb.core
  46. 46. fleetdb.embedded Wraps fleetdb.core Adds identity and durability
  47. 47. fleetdb.embedded Wraps fleetdb.core Adds identity and durability Databases in atoms
  48. 48. fleetdb.embedded Wraps fleetdb.core Adds identity and durability Databases in atoms Append-only log
  49. 49. fleetdb.embedded Read Path
  50. 50. fleetdb.embedded Read Path Dereference database atom
  51. 51. fleetdb.embedded Read Path Dereference database atom Pass to fleetdb.core
  52. 52. fleetdb.embedded Read Path Dereference database atom Pass to fleetdb.core Return result
  53. 53. fleetdb.embedded Write Path
  54. 54. fleetdb.embedded Write Path Enter fair lock
  55. 55. fleetdb.embedded Write Path Enter fair lock Dereference database atom
  56. 56. fleetdb.embedded Write Path Enter fair lock Dereference database atom Pass to fleetdb.core
  57. 57. fleetdb.embedded Write Path Enter fair lock Dereference database atom Pass to fleetdb.core Append query to log
  58. 58. fleetdb.embedded Write Path Enter fair lock Dereference database atom Pass to fleetdb.core Append query to log Swap in new database value
  59. 59. fleetdb.embedded Write Path Enter fair lock Dereference database atom Pass to fleetdb.core Append query to log Swap in new database value Return result
  60. 60. fleetdb.server Wraps fleetdb.embedded
  61. 61. fleetdb.server Wraps fleetdb.embedded Adds JSON client API
  62. 62. Aside: Source Size
  63. 63. Aside: Source Size Lines of Code core 630 embedded 130 server 120 lint 280 utilities 140 total 1300
  64. 64. Takeaways
  65. 65. Takeaways Clojure’s data structures are awesome
  66. 66. Takeaways Clojure’s data structures are awesome Clojure is viable for infrastructure software
  67. 67. Takeaways Clojure’s data structures are awesome Clojure is viable for infrastructure software Try FleetDB!
  68. 68. Thanks for listening! Questions?
  69. 69. http://FleetDB.org http://github.com/mmcgrana

×