1. FleetDB
A Schema-Free Database in Clojure
Mark McGranaghan
January 8, 2010

2. Motivation
Walkthrough
Implementation
QA

3. Motivation

4. Motivation
optimize for agile development

5. Walkthrough
Client Quickstart
Basic Queries
Concurrency Tools

6. Client Quickstart
(use ’fleetdb.client)
(def client (connect))
(client ["ping"])
=> "pong"

7. Insert
(client
["insert" "accounts"
{"id" 1 "owner" "Eve" "credits" 100}])

8. Insert
(client
["insert" "accounts"
{"id" 1 "owner" "Eve" "credits" 100}])
=> 1

9. Select
(client
["select" "accounts" {"where" ["=" "id" 1]}])

10. Select
(client
["select" "accounts" {"where" ["=" "id" 1]}])
=> [{"id" 1 "owner" "Eve" "credits" 100}]

11. Select with Conditions
(client
["select" "accounts"
{"where" [">=" "credits" 150]}])

12. Select with Order and Limit
(client
["select" "accounts"
{"order" ["credits" "asc"] "limit" 2}])

13. Update
(client
["update" "accounts" {"credits" 105}
{"where" ["=" "owner" "Eve"]}])

14. Explain (I)
(client
["explain" ["select" "accounts"
{"where" ["=" "owner" "Eve"]}]])

15. Explain (I)
(client
["explain" ["select" "accounts"
{"where" ["=" "owner" "Eve"]}]])
=> ["filter" ["=" "owner" "Eve"]
["collection-scan" "accounts"]]

16. Create Index
(client
["create-index" "accounts" "owner"])

17. Explain (II)
(client
["explain" ["select" "accounts"
{"where" ["=" "owner" "Eve"]}]])
=> ["index-lookup" ["accounts" "owner" "Eve"]]]

18. Multi-Write

19. Multi-Write
(client
["multi-write"
[write-query-1 write-query-2 ...]])

20. Multi-Write
(client
["multi-write"
[write-query-1 write-query-2 ...]])
=> [result-1 result-2 ...]

21. Checked-Write

22. Checked-Write
(client
["checked-write"
read-query
expected-read-result
write-query])

23. Checked-Write
(client
["checked-write"
read-query
expected-read-result
write-query])
=> [true write-result]

24. Checked-Write
(client
["checked-write"
read-query
expected-read-result
write-query])
=> [true write-result]
=> [false actual-read-result]

25. Implementation
Background
Organization
Key ideas

26. Background
http://clojure.org/data_structures
http://clojure.org/sequences
http://clojure.org/state

27. Organization

28. Organization
fleetdb.core: pure functions

29. Organization
fleetdb.core: pure functions
fleetdb.embedded: identity and durability

30. Organization
fleetdb.core: pure functions
fleetdb.embedded: identity and durability
fleetdb.server: network interface

31. fleetdb.core
Pure functions

32. fleetdb.core
Pure functions
Databases as Clojure data structures

33. fleetdb.core
Pure functions
Databases as Clojure data structures
Read: db + query → result

34. fleetdb.core
Pure functions
Databases as Clojure data structures
Read: db + query → result
‘Write’: db + query → result + new db

35. fleetdb.core Query Planner

36. fleetdb.core Query Planner
Implements declarative queries

37. fleetdb.core Query Planner
Implements declarative queries
Database + query → plan

38. fleetdb.core Query Planner
Implements declarative queries
Database + query → plan
["select" "accounts"
{"where" ["=" "owner" "Eve"]}]

39. fleetdb.core Query Planner
Implements declarative queries
Database + query → plan
["select" "accounts"
{"where" ["=" "owner" "Eve"]}]
["filter" ["=" "owner" "Eve"]
["record-scan" "accounts"]]

40. fleetdb.core Query Planner
Implements declarative queries
Database + query → plan
["select" "accounts"
{"where" ["=" "owner" "Eve"]}]
["filter" ["=" "owner" "Eve"]
["record-scan" "accounts"]]
["index-lookup" ["accounts" "owner" "Eve"]]

41. fleetdb.core Query Executor

42. fleetdb.core Query Executor
Database + plan → result

43. fleetdb.core Query Executor
Database + plan → result
["filter" ["=" "owner" "Eve"]
["record-scan" "accounts"]]

44. fleetdb.core Query Executor
Database + plan → result
["filter" ["=" "owner" "Eve"]
["record-scan" "accounts"]]
(filter (fn [r] (= (r "owner") "Eve"))
(vals (:rmap (db "accounts"))))

45. fleetdb.embedded
Wraps fleetdb.core

46. fleetdb.embedded
Wraps fleetdb.core
Adds identity and durability

47. fleetdb.embedded
Wraps fleetdb.core
Adds identity and durability
Databases in atoms

48. fleetdb.embedded
Wraps fleetdb.core
Adds identity and durability
Databases in atoms
Append-only log

49. fleetdb.embedded Read Path

50. fleetdb.embedded Read Path
Dereference database atom

51. fleetdb.embedded Read Path
Dereference database atom
Pass to fleetdb.core

52. fleetdb.embedded Read Path
Dereference database atom
Pass to fleetdb.core
Return result

53. fleetdb.embedded Write Path

54. fleetdb.embedded Write Path
Enter fair lock

55. fleetdb.embedded Write Path
Enter fair lock
Dereference database atom

56. fleetdb.embedded Write Path
Enter fair lock
Dereference database atom
Pass to fleetdb.core

57. fleetdb.embedded Write Path
Enter fair lock
Dereference database atom
Pass to fleetdb.core
Append query to log

58. fleetdb.embedded Write Path
Enter fair lock
Dereference database atom
Pass to fleetdb.core
Append query to log
Swap in new database value

59. fleetdb.embedded Write Path
Enter fair lock
Dereference database atom
Pass to fleetdb.core
Append query to log
Swap in new database value
Return result

60. fleetdb.server
Wraps fleetdb.embedded

61. fleetdb.server
Wraps fleetdb.embedded
Adds JSON client API

62. Aside: Source Size

63. Aside: Source Size
Lines of Code
core 630
embedded 130
server 120
lint 280
utilities 140
total 1300

64. Takeaways

65. Takeaways
Clojure’s data structures are awesome

66. Takeaways
Clojure’s data structures are awesome
Clojure is viable for infrastructure software

67. Takeaways
Clojure’s data structures are awesome
Clojure is viable for infrastructure software
Try FleetDB!

68. Thanks for listening!
Questions?

69. http://FleetDB.org
http://github.com/mmcgrana