Your SlideShare is downloading. ×
0
FleetDB
A Schema-Free Database in Clojure

         Mark McGranaghan


           January 8, 2010
Motivation


Walkthrough


Implementation


QA
Motivation
Motivation



       optimize for agile development
Walkthrough



     Client Quickstart
     Basic Queries
     Concurrency Tools
Client Quickstart



  (use ’fleetdb.client)
  (def client (connect))

  (client ["ping"])
  => "pong"
Insert



  (client
    ["insert" "accounts"
      {"id" 1 "owner" "Eve" "credits" 100}])
Insert



  (client
    ["insert" "accounts"
      {"id" 1 "owner" "Eve" "credits" 100}])

  => 1
Select



  (client
    ["select" "accounts" {"where" ["=" "id" 1]}])
Select



  (client
    ["select" "accounts" {"where" ["=" "id" 1]}])

  => [{"id" 1 "owner" "Eve" "credits" 100}]
Select with Conditions



  (client
    ["select" "accounts"
      {"where" [">=" "credits" 150]}])
Select with Order and Limit



  (client
    ["select" "accounts"
      {"order" ["credits" "asc"] "limit" 2}])
Update



  (client
    ["update" "accounts" {"credits" 105}
      {"where" ["=" "owner" "Eve"]}])
Explain (I)


  (client
    ["explain" ["select" "accounts"
                 {"where" ["=" "owner" "Eve"]}]])
Explain (I)


  (client
    ["explain" ["select" "accounts"
                 {"where" ["=" "owner" "Eve"]}]])

  => ["filt...
Create Index



  (client
    ["create-index" "accounts" "owner"])
Explain (II)



  (client
    ["explain" ["select" "accounts"
                 {"where" ["=" "owner" "Eve"]}]])

  => ["in...
Multi-Write
Multi-Write



  (client
    ["multi-write"
     [write-query-1 write-query-2 ...]])
Multi-Write



  (client
    ["multi-write"
     [write-query-1 write-query-2 ...]])

  => [result-1 result-2 ...]
Checked-Write
Checked-Write


  (client
    ["checked-write"
     read-query
     expected-read-result
     write-query])
Checked-Write


  (client
    ["checked-write"
     read-query
     expected-read-result
     write-query])

  => [true wr...
Checked-Write


  (client
    ["checked-write"
     read-query
     expected-read-result
     write-query])

  => [true wr...
Implementation



     Background
     Organization
     Key ideas
Background



    http://clojure.org/data_structures
    http://clojure.org/sequences
    http://clojure.org/state
Organization
Organization



     fleetdb.core: pure functions
Organization



     fleetdb.core: pure functions
     fleetdb.embedded: identity and durability
Organization



     fleetdb.core: pure functions
     fleetdb.embedded: identity and durability
     fleetdb.server: netw...
fleetdb.core



    Pure functions
fleetdb.core



    Pure functions
    Databases as Clojure data structures
fleetdb.core



    Pure functions
    Databases as Clojure data structures
    Read: db + query → result
fleetdb.core



    Pure functions
    Databases as Clojure data structures
    Read: db + query → result
    ‘Write’: db ...
fleetdb.core Query Planner
fleetdb.core Query Planner

    Implements declarative queries
fleetdb.core Query Planner

    Implements declarative queries
    Database + query → plan
fleetdb.core Query Planner

      Implements declarative queries
      Database + query → plan

  ["select" "accounts"
   ...
fleetdb.core Query Planner

      Implements declarative queries
      Database + query → plan

  ["select" "accounts"
   ...
fleetdb.core Query Planner

      Implements declarative queries
      Database + query → plan

  ["select" "accounts"
   ...
fleetdb.core Query Executor
fleetdb.core Query Executor


    Database + plan → result
fleetdb.core Query Executor


      Database + plan → result

  ["filter" ["=" "owner" "Eve"]
    ["record-scan" "accounts...
fleetdb.core Query Executor


      Database + plan → result

  ["filter" ["=" "owner" "Eve"]
    ["record-scan" "accounts...
fleetdb.embedded



    Wraps fleetdb.core
fleetdb.embedded



    Wraps fleetdb.core
    Adds identity and durability
fleetdb.embedded



    Wraps fleetdb.core
    Adds identity and durability
    Databases in atoms
fleetdb.embedded



    Wraps fleetdb.core
    Adds identity and durability
    Databases in atoms
    Append-only log
fleetdb.embedded Read Path
fleetdb.embedded Read Path



    Dereference database atom
fleetdb.embedded Read Path



    Dereference database atom
    Pass to fleetdb.core
fleetdb.embedded Read Path



    Dereference database atom
    Pass to fleetdb.core
    Return result
fleetdb.embedded Write Path
fleetdb.embedded Write Path


    Enter fair lock
fleetdb.embedded Write Path


    Enter fair lock
    Dereference database atom
fleetdb.embedded Write Path


    Enter fair lock
    Dereference database atom
    Pass to fleetdb.core
fleetdb.embedded Write Path


    Enter fair lock
    Dereference database atom
    Pass to fleetdb.core
    Append query ...
fleetdb.embedded Write Path


    Enter fair lock
    Dereference database atom
    Pass to fleetdb.core
    Append query ...
fleetdb.embedded Write Path


    Enter fair lock
    Dereference database atom
    Pass to fleetdb.core
    Append query ...
fleetdb.server



    Wraps fleetdb.embedded
fleetdb.server



    Wraps fleetdb.embedded
    Adds JSON client API
Aside: Source Size
Aside: Source Size


                Lines of Code
              core         630
              embedded 130
             ...
Takeaways
Takeaways



     Clojure’s data structures are awesome
Takeaways



     Clojure’s data structures are awesome
     Clojure is viable for infrastructure software
Takeaways



     Clojure’s data structures are awesome
     Clojure is viable for infrastructure software
     Try FleetD...
Thanks for listening!

    Questions?
http://FleetDB.org

http://github.com/mmcgrana
Upcoming SlideShare
Loading in...5
×

FleetDB A Schema-Free Database in Clojure

2,864

Published on

Published in: Technology, Business
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,864
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
21
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Transcript of "FleetDB A Schema-Free Database in Clojure"

  1. 1. FleetDB A Schema-Free Database in Clojure Mark McGranaghan January 8, 2010
  2. 2. Motivation Walkthrough Implementation QA
  3. 3. Motivation
  4. 4. Motivation optimize for agile development
  5. 5. Walkthrough Client Quickstart Basic Queries Concurrency Tools
  6. 6. Client Quickstart (use ’fleetdb.client) (def client (connect)) (client ["ping"]) => "pong"
  7. 7. Insert (client ["insert" "accounts" {"id" 1 "owner" "Eve" "credits" 100}])
  8. 8. Insert (client ["insert" "accounts" {"id" 1 "owner" "Eve" "credits" 100}]) => 1
  9. 9. Select (client ["select" "accounts" {"where" ["=" "id" 1]}])
  10. 10. Select (client ["select" "accounts" {"where" ["=" "id" 1]}]) => [{"id" 1 "owner" "Eve" "credits" 100}]
  11. 11. Select with Conditions (client ["select" "accounts" {"where" [">=" "credits" 150]}])
  12. 12. Select with Order and Limit (client ["select" "accounts" {"order" ["credits" "asc"] "limit" 2}])
  13. 13. Update (client ["update" "accounts" {"credits" 105} {"where" ["=" "owner" "Eve"]}])
  14. 14. Explain (I) (client ["explain" ["select" "accounts" {"where" ["=" "owner" "Eve"]}]])
  15. 15. Explain (I) (client ["explain" ["select" "accounts" {"where" ["=" "owner" "Eve"]}]]) => ["filter" ["=" "owner" "Eve"] ["collection-scan" "accounts"]]
  16. 16. Create Index (client ["create-index" "accounts" "owner"])
  17. 17. Explain (II) (client ["explain" ["select" "accounts" {"where" ["=" "owner" "Eve"]}]]) => ["index-lookup" ["accounts" "owner" "Eve"]]]
  18. 18. Multi-Write
  19. 19. Multi-Write (client ["multi-write" [write-query-1 write-query-2 ...]])
  20. 20. Multi-Write (client ["multi-write" [write-query-1 write-query-2 ...]]) => [result-1 result-2 ...]
  21. 21. Checked-Write
  22. 22. Checked-Write (client ["checked-write" read-query expected-read-result write-query])
  23. 23. Checked-Write (client ["checked-write" read-query expected-read-result write-query]) => [true write-result]
  24. 24. Checked-Write (client ["checked-write" read-query expected-read-result write-query]) => [true write-result] => [false actual-read-result]
  25. 25. Implementation Background Organization Key ideas
  26. 26. Background http://clojure.org/data_structures http://clojure.org/sequences http://clojure.org/state
  27. 27. Organization
  28. 28. Organization fleetdb.core: pure functions
  29. 29. Organization fleetdb.core: pure functions fleetdb.embedded: identity and durability
  30. 30. Organization fleetdb.core: pure functions fleetdb.embedded: identity and durability fleetdb.server: network interface
  31. 31. fleetdb.core Pure functions
  32. 32. fleetdb.core Pure functions Databases as Clojure data structures
  33. 33. fleetdb.core Pure functions Databases as Clojure data structures Read: db + query → result
  34. 34. fleetdb.core Pure functions Databases as Clojure data structures Read: db + query → result ‘Write’: db + query → result + new db
  35. 35. fleetdb.core Query Planner
  36. 36. fleetdb.core Query Planner Implements declarative queries
  37. 37. fleetdb.core Query Planner Implements declarative queries Database + query → plan
  38. 38. fleetdb.core Query Planner Implements declarative queries Database + query → plan ["select" "accounts" {"where" ["=" "owner" "Eve"]}]
  39. 39. fleetdb.core Query Planner Implements declarative queries Database + query → plan ["select" "accounts" {"where" ["=" "owner" "Eve"]}] ["filter" ["=" "owner" "Eve"] ["record-scan" "accounts"]]
  40. 40. fleetdb.core Query Planner Implements declarative queries Database + query → plan ["select" "accounts" {"where" ["=" "owner" "Eve"]}] ["filter" ["=" "owner" "Eve"] ["record-scan" "accounts"]] ["index-lookup" ["accounts" "owner" "Eve"]]
  41. 41. fleetdb.core Query Executor
  42. 42. fleetdb.core Query Executor Database + plan → result
  43. 43. fleetdb.core Query Executor Database + plan → result ["filter" ["=" "owner" "Eve"] ["record-scan" "accounts"]]
  44. 44. fleetdb.core Query Executor Database + plan → result ["filter" ["=" "owner" "Eve"] ["record-scan" "accounts"]] (filter (fn [r] (= (r "owner") "Eve")) (vals (:rmap (db "accounts"))))
  45. 45. fleetdb.embedded Wraps fleetdb.core
  46. 46. fleetdb.embedded Wraps fleetdb.core Adds identity and durability
  47. 47. fleetdb.embedded Wraps fleetdb.core Adds identity and durability Databases in atoms
  48. 48. fleetdb.embedded Wraps fleetdb.core Adds identity and durability Databases in atoms Append-only log
  49. 49. fleetdb.embedded Read Path
  50. 50. fleetdb.embedded Read Path Dereference database atom
  51. 51. fleetdb.embedded Read Path Dereference database atom Pass to fleetdb.core
  52. 52. fleetdb.embedded Read Path Dereference database atom Pass to fleetdb.core Return result
  53. 53. fleetdb.embedded Write Path
  54. 54. fleetdb.embedded Write Path Enter fair lock
  55. 55. fleetdb.embedded Write Path Enter fair lock Dereference database atom
  56. 56. fleetdb.embedded Write Path Enter fair lock Dereference database atom Pass to fleetdb.core
  57. 57. fleetdb.embedded Write Path Enter fair lock Dereference database atom Pass to fleetdb.core Append query to log
  58. 58. fleetdb.embedded Write Path Enter fair lock Dereference database atom Pass to fleetdb.core Append query to log Swap in new database value
  59. 59. fleetdb.embedded Write Path Enter fair lock Dereference database atom Pass to fleetdb.core Append query to log Swap in new database value Return result
  60. 60. fleetdb.server Wraps fleetdb.embedded
  61. 61. fleetdb.server Wraps fleetdb.embedded Adds JSON client API
  62. 62. Aside: Source Size
  63. 63. Aside: Source Size Lines of Code core 630 embedded 130 server 120 lint 280 utilities 140 total 1300
  64. 64. Takeaways
  65. 65. Takeaways Clojure’s data structures are awesome
  66. 66. Takeaways Clojure’s data structures are awesome Clojure is viable for infrastructure software
  67. 67. Takeaways Clojure’s data structures are awesome Clojure is viable for infrastructure software Try FleetDB!
  68. 68. Thanks for listening! Questions?
  69. 69. http://FleetDB.org http://github.com/mmcgrana
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×