Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Exploring four Datomic superpowers

3,993 views

Published on

https://www.youtube.com/watch?v=7lm3K8zVOdY

This session will explore four common problems, and the unique and surprising tools Datomic provides to solve them elegantly: HTTP caching - How to generically generate and validate Last-Modified and If-Modified-Since headers Audit trail - how to extend Datomic’s immutable transaction log to include arbitrary audit related metadata Mobile database sync - trivial implementation of an incremental update API for high latency/low bandwidth clients Authorization - easily determine resource ownership, and centrally isolate users from data they are not allowed to see

These problems have certainly been solved before using other databases, but Datomic provides features that make the proposed implementations concise, generic, and purely functional

Published in: Software
  • Be the first to comment

Exploring four Datomic superpowers

  1. 1. Exploring Four Datomic Superpowers Lucas Cavalcanti & Edward Wible
  2. 2. 2 Why build a bank from scratch in Brazil using Clojure and Datomic?
  3. 3. 3
  4. 4. 3
  5. 5. 4 1 Audit Trail
  6. 6. 5 event stream 01 NOV 10:00 Customer joins waiting list for a card
  7. 7. 5 event stream 01 NOV 11:00 Robot 437aae3 approves R$3K limit 01 NOV 10:00 Customer joins waiting list for a card
  8. 8. 5 event stream 09 NOV 08:00 Mastercard purchase, Starbucks, R$100 01 NOV 11:00 Robot 437aae3 approves R$3K limit 01 NOV 10:00 Customer joins waiting list for a card
  9. 9. 5 event stream 15 NOV 15:00 Support agent increases limit to R$5K 09 NOV 08:00 Mastercard purchase, Starbucks, R$100 01 NOV 11:00 Robot 437aae3 approves R$3K limit 01 NOV 10:00 Customer joins waiting list for a card
  10. 10. 5 event stream 15 NOV 17:05 Customer blocks card 15 NOV 15:00 Support agent increases limit to R$5K 09 NOV 08:00 Mastercard purchase, Starbucks, R$100 01 NOV 11:00 Robot 437aae3 approves R$3K limit 01 NOV 10:00 Customer joins waiting list for a card
  11. 11. facts over time 6 15 NOV 17:05 15 NOV 15:00 09 NOV 08:00 01 NOV 11:00 01 NOV 10:00 [<card> :card/status :card.status/blocked] [<card> :card/status :card.status/active] [<account> :account/limit 5000] [<account> :account/limit 3000] [<purchase> :purchase/card <card>] [<purchase> :purchase/amount 100] [<purchase> :purchase/merchant “Starbucks”] [<account> :account/customer <customer>] [<account> :account/limit 3000] [<card> :card/account <account>] [<card> :card/status :card.status/active] [<customer> :customer/id #uuid “b2c90…”]
  12. 12. facts over time 6 15 NOV 17:05 15 NOV 15:00 09 NOV 08:00 01 NOV 11:00 01 NOV 10:00 [<card> :card/status :card.status/blocked] [<card> :card/status :card.status/active] [<account> :account/limit 5000] [<account> :account/limit 3000] [<purchase> :purchase/card <card>] [<purchase> :purchase/amount 100] [<purchase> :purchase/merchant “Starbucks”] [<account> :account/customer <customer>] [<account> :account/limit 3000] [<card> :card/account <account>] [<card> :card/status :card.status/active] [<customer> :customer/id #uuid “b2c90…”] entity attribute value
  13. 13. 7
  14. 14. 7 don’t lose data
  15. 15. 7 don’t lose data what changed when
  16. 16. 7 don’t lose data what changed when as-of since
  17. 17. 7 don’t lose data what changed when as-of since fork merge
  18. 18. 8 What was the initial limit for the card?
  19. 19. 8 What was the initial limit for the card? At the time the Starbucks transaction occurred, which fraud triggers would have activated?
  20. 20. 8 What was the initial limit for the card? At the time the Starbucks transaction occurred, which fraud triggers would have activated? How long did the customer spend on each stage of the acquisition funnel?
  21. 21. 8 What was the initial limit for the card? At the time the Starbucks transaction occurred, which fraud triggers would have activated? How long did the customer spend on each stage of the acquisition funnel? How frequently do we see amount changes on Starbucks transactions?
  22. 22. 9 What sequence of events resulted in the Starbucks transaction being persisted?
  23. 23. 9 What sequence of events resulted in the Starbucks transaction being persisted? iOS http-in kafka-out kafka-in iZb iZb.jnA iZb.jnA.9Cd iZb.jnA.9Cd.l9A
  24. 24. 9 iOS http-in kafka-out kafka-in iZb iZb.jnA iZb.jnA.9Cd iZb.jnA.9Cd.l9A What sequence of events resulted in the Starbucks transaction being persisted? correlation-id
  25. 25. Who was responsible for the credit limit increase? 9 iOS http-in kafka-out kafka-in iZb iZb.jnA iZb.jnA.9Cd iZb.jnA.9Cd.l9A What sequence of events resulted in the Starbucks transaction being persisted? correlation-id
  26. 26. Who was responsible for the credit limit increase? 9 iOS http-in kafka-out kafka-in iZb iZb.jnA iZb.jnA.9Cd iZb.jnA.9Cd.l9A What sequence of events resulted in the Starbucks transaction being persisted? correlation-id #uuid “b2c90…” “lucas@nubank.com.br” :kafka-LIMIT-CHANGED :robot-437aae3
  27. 27. Who was responsible for the credit limit increase? 9 iOS http-in kafka-out kafka-in iZb iZb.jnA iZb.jnA.9Cd iZb.jnA.9Cd.l9A What sequence of events resulted in the Starbucks transaction being persisted? correlation-id user #uuid “b2c90…” “lucas@nubank.com.br” :kafka-LIMIT-CHANGED :robot-437aae3
  28. 28. Who was responsible for the credit limit increase? 9 What sequence of events resulted in the Starbucks transaction being persisted? Why was the customer’s card blocked? correlation-id iOS http-in kafka-out kafka-in iZb iZb.jnA iZb.jnA.9Cd iZb.jnA.9Cd.l9A user #uuid “b2c90…” “lucas@nubank.com.br” :kafka-LIMIT-CHANGED :robot-437aae3
  29. 29. Who was responsible for the credit limit increase? 9 What sequence of events resulted in the Starbucks transaction being persisted? Why was the customer’s card blocked? correlation-id iOS http-in kafka-out kafka-in iZb iZb.jnA iZb.jnA.9Cd iZb.jnA.9Cd.l9A :fraud-preventative :recurring-scheduled :late-payment :data-migration user #uuid “b2c90…” “lucas@nubank.com.br” :kafka-LIMIT-CHANGED :robot-437aae3
  30. 30. Who was responsible for the credit limit increase? 9 What sequence of events resulted in the Starbucks transaction being persisted? Why was the customer’s card blocked? correlation-id iOS http-in kafka-out kafka-in iZb iZb.jnA iZb.jnA.9Cd iZb.jnA.9Cd.l9A user #uuid “b2c90…” “lucas@nubank.com.br” :kafka-LIMIT-CHANGED :robot-437aae3 :fraud-preventative reason :recurring-scheduled :late-payment :data-migration
  31. 31. (defn block-card [card reason] (d/transact (conn) [[:db/add card :card/status :card.status/blocked] ])) 10 storing transaction metadata
  32. 32. (defn block-card [card reason] (d/transact (conn) [[:db/add card :card/status :card.status/blocked] {:db/id (d/tempid :db.part/tx) :audit/user "lucas@nubank.com.br" :audit/cid "iZb.jnA.9Cd.l9A" :audit/tags :fraud-preventative} ])) 10 storing transaction metadata
  33. 33. 11 transactions over time with metadata [<card> :card/status :card.status/blocked] [<card> :card/status :card.status/active] 15 NOV 17:05 [<account> :account/limit 5000M] 15 NOV 15:00 [<account> :account/limit 3000M] [<purchase> :purchase/card <card>] [<purchase> :purchase/amount 100M] [<purchase> :purchase/merchant “Starbucks”] 09 NOV 08:00 [<account> :account/customer <customer>] [<account> :account/limit 3000M] [<card> :card/account <account>] [<card> :card/status :card.status/active] 01 NOV 11:00 01 NOV 10:00 [<customer> :customer/id #uuid “b2c90…”]
  34. 34. 11 transactions over time with metadata [<card> :card/status :card.status/blocked] [<card> :card/status :card.status/active] 15 NOV 17:05 [<account> :account/limit 5000M] 15 NOV 15:00 [<account> :account/limit 3000M] [<purchase> :purchase/card <card>] [<purchase> :purchase/amount 100M] [<purchase> :purchase/merchant “Starbucks”] 09 NOV 08:00 [<account> :account/customer <customer>] [<account> :account/limit 3000M] [<card> :card/account <account>] [<card> :card/status :card.status/active] 01 NOV 11:00 01 NOV 10:00 [<customer> :customer/id #uuid “b2c90…”]
  35. 35. 12 querying transaction metadata (defn who-blocked-the-card? [card] (d/q '{:find [?user ?cid ?tags] :in [$ ?status ?card] :where [[?card :card/status ?status ?tx] [?tx :audit/user ?user] [?tx :audit/cid ?cid] [?tx :audit/tags ?tags]]} db :card.status/blocked card))
  36. 36. 12 querying transaction metadata (defn who-blocked-the-card? [card] (d/q '{:find [?user ?cid ?tags] :in [$ ?status ?card] :where [[?entity attribute value txn [?card :card/status ?status ?tx] [?tx :audit/user ?user] [?tx :audit/cid ?cid] [?tx :audit/tags ?tags]]} db :card.status/blocked card))
  37. 37. 12 querying transaction metadata (defn who-blocked-the-card? [card] (d/q '{:find [?user ?cid ?tags] :in [$ ?status ?card] :where [[?card :card/status ?status ?tx] [?tx :audit/user ?user] [?tx :audit/cid ?cid] [?tx :audit/tags ?tags]]} db :card.status/blocked card))
  38. 38. 12 querying transaction metadata (defn who-blocked-the-card? [card] (d/q '{:find [?user ?cid ?tags] :in [$ ?status ?card] :where [[?card :card/status ?status ?tx] [?tx :audit/user ?user] [?tx :audit/cid ?cid] [?tx :audit/tags ?tags]]} db :card.status/blocked card))
  39. 39. 12 querying transaction metadata (defn who-blocked-the-card? [card] (d/q '{:find [?user ?cid ?tags] :in [$ ?status ?card] :where [[?card :card/status ?status ?tx] [?tx :audit/user ?user] [?tx :audit/cid ?cid] [?tx :audit/tags ?tags]]} db :card.status/blocked card))
  40. 40. 13 2 Authorization
  41. 41. should we return data? GET /purchases/1337/comments
  42. 42. should we return data? GET /purchases/1337/comments 200 OK 401 Unauthorized
  43. 43. customer account payment card purchase comment should we return data? GET /purchases/1337/comments 200 OK 401 Unauthorized
  44. 44. customer account payment card purchase comment should we return data? GET /purchases/1337/comments 200 OK 401 Unauthorized
  45. 45. customer account payment card purchase comment should we return data? GET /purchases/1337/comments 200 OK 401 Unauthorized
  46. 46. 15 explicit relationship traversal (defn owns? [customer-id purchase-id db] (d/q '{:find [?pur .] :in [$ ?customer-id ?purchase-id] :where [[?pur :purchase/id ?purchase-id] [?pur :purchase/account ?acc] [?acc :account/customer ?cus] [?cus :customer/id ?customer-id]]} db customer-id purchase-id))
  47. 47. 15 explicit relationship traversal (defn owns? [customer-id purchase-id db] (d/q '{:find [?pur .] :in [$ ?customer-id ?purchase-id] :where [[?pur :purchase/id ?purchase-id] [?pur :purchase/account ?acc] [?acc :account/customer ?cus] [?cus :customer/id ?customer-id]]} db customer-id purchase-id))
  48. 48. generic relationship traversal 16 recursive rule (logical OR)
  49. 49. generic relationship traversal 16 recursive rule (logical OR) (def owner-rules '[[(owns? ?cus-id ?e) [?e :customer/id ?cus-id]] [(owns? ?cus-id ?e) [?e ?ref-attr ?r] (owns? ?cus-id ?r)]])
  50. 50. generic relationship traversal 16 recursive rule (logical OR) (def owner-rules '[[(owns? ?cus-id ?e) [?e :customer/id ?cus-id]] [(owns? ?cus-id ?e) [?e ?ref-attr ?r] (owns? ?cus-id ?r)]])
  51. 51. generic relationship traversal 16 recursive rule (logical OR) (def owner-rules '[[(owns? ?cus-id ?e) [?e :customer/id ?cus-id]] [(owns? ?cus-id ?e) [?e ?ref-attr ?r] (owns? ?cus-id ?r)]])
  52. 52. generic relationship traversal 16 recursive rule (logical OR) (def owner-rules '[[(owns? ?cus-id ?e) [?e :customer/id ?cus-id]] [(owns? ?cus-id ?e) [?e ?ref-attr ?r] (owns? ?cus-id ?r)]])
  53. 53. generic relationship traversal 16 recursive rule (logical OR) (def owner-rules '[[(owns? ?cus-id ?e) [?e :customer/id ?cus-id]] [(owns? ?cus-id ?e) [?e ?ref-attr ?r] (owns? ?cus-id ?r)]])
  54. 54. generic relationship traversal 16 recursive rule (logical OR) (def owner-rules '[[(owns? ?cus-id ?e) [?e :customer/id ?cus-id]] [(owns? ?cus-id ?e) [?e ?ref-attr ?r] (owns? ?cus-id ?r)]])
  55. 55. generic relationship traversal 16 recursive rule (logical OR) query using rule (def owner-rules '[[(owns? ?cus-id ?e) [?e :customer/id ?cus-id]] [(owns? ?cus-id ?e) [?e ?ref-attr ?r] (owns? ?cus-id ?r)]])
  56. 56. generic relationship traversal :in [$ ?cus-id ?e %] :where [(owns? ?cus-id ?e)]} db customer-id entity owner-rules)) 16 recursive rule (logical OR) query using rule (def owner-rules '[[(owns? ?cus-id ?e) [?e :customer/id ?cus-id]] [(owns? ?cus-id ?e) [?e ?ref-attr ?r] (owns? ?cus-id ?r)]]) (defn owns? [customer-id entity db] (d/q '{:find [?e .] (owns? customer-id [:purchase/id id] db)
  57. 57. generic relationship traversal :in [$ ?cus-id ?e %] :where [(owns? ?cus-id ?e)]} db customer-id entity owner-rules)) 16 recursive rule (logical OR) query using rule (def owner-rules '[[(owns? ?cus-id ?e) [?e :customer/id ?cus-id]] [(owns? ?cus-id ?e) [?e ?ref-attr ?r] (owns? ?cus-id ?r)]]) (defn owns? [customer-id entity db] (d/q '{:find [?e .] (owns? customer-id [:purchase/id id] db)
  58. 58. 17 Recursion rules! But can we go further? Can we prevent queries from returning entities owned by other customers?
  59. 59. all owned entities 18 query binding attribute
  60. 60. all owned entities 18 query binding attribute (defn owns? [customer-id entity db] (d/q '{:find [?e .] :in [$ ?cus-id ?e %] :where [(owns? ?cus-id ?e)]} db customer-id entity] owner-rules))
  61. 61. all owned entities 18 query binding attribute (defn owns? [customer-id entity db] (d/q '{:find [?e .] :in [$ ?cus-id ?e %] :where [(owns? ?cus-id ?e)]} db customer-id entity] owner-rules))
  62. 62. all owned entities 18 query binding attribute attribute unbound (defn owns? [customer-id entity db] (d/q '{:find [?e .] :in [$ ?cus-id ?e %] :where [(owns? ?cus-id ?e)]} db customer-id entity] owner-rules))
  63. 63. all owned entities 18 query binding attribute attribute unbound (defn owns? [customer-id entity db] (d/q '{:find [?e .] :in [$ ?cus-id ?e %] :where [(owns? ?cus-id ?e)]} db customer-id entity] owner-rules)) (defn owned-entities [customer-id db] (d/q '{:find [[?e ...]] :in [$ ?cus-id %] :where [(owns? ?cus-id ?e)]} db customer-id owner-rules))
  64. 64. all owned entities 18 query binding attribute attribute unbound (defn owns? [customer-id entity db] (d/q '{:find [?e .] :in [$ ?cus-id ?e %] :where [(owns? ?cus-id ?e)]} db customer-id entity] owner-rules)) (defn owned-entities [customer-id db] (d/q '{:find [[?e ...]] :in [$ ?cus-id %] :where [(owns? ?cus-id ?e)]} db customer-id owner-rules))
  65. 65. all owned entities 18 query binding attribute attribute unbound (defn owns? [customer-id entity db] (d/q '{:find [?e .] :in [$ ?cus-id ?e %] :where [(owns? ?cus-id ?e)]} db customer-id entity] owner-rules)) (defn owned-entities [customer-id db] (d/q '{:find [[?e ...]] :in [$ ?cus-id %] :where [(owns? ?cus-id ?e)]} db customer-id owner-rules))
  66. 66. filtered db (as value) 19 (defn owned-db [customer-id db] (let [owned (set (owned-entities customer-id db))] (d/filter db (fn [_ [e a v t :as datom]] (when (or (= e t) (contains? owned e)) datom)))))
  67. 67. filtered db (as value) 19 (defn owned-db [customer-id db] (let [owned (set (owned-entities customer-id db))] (d/filter db (fn [_ [e a v t :as datom]] (when (or (= e t) (contains? owned e)) datom)))))
  68. 68. filtered db (as value) 19 (defn owned-db [customer-id db] (let [owned (set (owned-entities customer-id db))] (d/filter db (fn [_ [e a v t :as datom]] (when (or (= e t) (contains? owned e)) datom)))))
  69. 69. filtered db (as value) 19 (defn owned-db [customer-id db] (let [owned (set (owned-entities customer-id db))] (d/filter db (fn [_ [e a v t :as datom]] (when (or (= e t) (contains? owned e)) datom)))))
  70. 70. filtered db (as value) 19 (defn owned-db [customer-id db] (let [owned (set (owned-entities customer-id db))] (d/filter db (fn [_ [e a v t :as datom]] (when (or (= e t) (contains? owned e)) datom)))))
  71. 71. filtered db (as value) (let [owned (set (owned-entities customer-id db))] (d/filter db 19 (defn owned-db [customer-id db] (fn [_ [e a v t :as datom]] (when (or (= e t) (contains? owned e)) datom))))) filtered db will not contain datoms from other owners!
  72. 72. passing a filtered db is transparent for queries 20 (defn all-purchases [account-id db] (d/q '{:find [[(pull ?p [*]) ...]] :in [$ ?acc] :where [[?p :purchase/account ?acc]]} db [:account/id account-id])) (all-purchases account-id db) (all-purchases account-id (owned-db customer-id db))
  73. 73. passing a filtered db is transparent for queries 20 (defn all-purchases [account-id db] (d/q '{:find [[(pull ?p [*]) ...]] :in [$ ?acc] :where [[?p :purchase/account ?acc]]} db [:account/id account-id])) (all-purchases account-id db) (all-purchases account-id (owned-db customer-id db))
  74. 74. 21 We’re Mobile! Bandwidth is costly
  75. 75. 22 3 HTTP Cache
  76. 76. 23 http last modified header GET /accounts/1234/purchases 200 OK Last-Modified: Fri, 14 Nov 2014 14:28:50 UTC {"purchases": [...]}
  77. 77. http last modified header If-Modified-Since: Fri, 14 Nov 2014 14:28:50 UTC GET /accounts/1234/purchases 304 Not Modified 23 GET /accounts/1234/purchases 200 OK Last-Modified: Fri, 14 Nov 2014 14:28:50 UTC {"purchases": [...]}
  78. 78. 24 How to keep track of a good last-modified date for a URL?
  79. 79. 24 How to keep track of a good last-modified date for a URL? The last time any owned entity changed!
  80. 80. If no customer-owned data changed, 304 25 (defn last-modified [customer-id db] (d/q '{:find [(max ?time) .] :in [$ ?cus-id %] :where [(owns? ?cus-id ?e) [?e ?a ?v ?tx] [?tx :db/txInstant ?time]]} db customer-id owner-rules))
  81. 81. If no customer-owned data changed, 304 25 (defn last-modified [customer-id db] (d/q '{:find [(max ?time) .] :in [$ ?cus-id %] :where [(owns? ?cus-id ?e) [?e ?a ?v ?tx] [?tx :db/txInstant ?time]]} db customer-id owner-rules))
  82. 82. If no customer-owned data changed, 304 25 (defn last-modified [customer-id db] (d/q '{:find [(max ?time) .] :in [$ ?cus-id %] :where [(owns? ?cus-id ?e) [?e ?a ?v ?tx] [?tx :db/txInstant ?time]]} db customer-id owner-rules))
  83. 83. If no customer-owned data changed, 304 25 (defn last-modified [customer-id db] (d/q '{:find [(max ?time) .] :in [$ ?cus-id %] :where [(owns? ?cus-id ?e) [?e ?a ?v ?tx] [?tx :db/txInstant ?time]]} db customer-id owner-rules))
  84. 84. If no customer-owned data changed, 304 25 (defn last-modified [customer-id db] (d/q '{:find [(max ?time) .] :in [$ ?cus-id %] :where [(owns? ?cus-id ?e) [?e ?a ?v ?tx] [?tx :db/txInstant ?time]]} db customer-id owner-rules))
  85. 85. If no customer-owned data changed, 304 25 (defn last-modified [customer-id db] (d/q '{:find [(max ?time) .] :in [$ ?cus-id %] :where [(owns? ?cus-id ?e) [?e ?a ?v ?tx] [?tx :db/txInstant ?time]]} db customer-id owner-rules))
  86. 86. If no customer-owned data changed, 304 25 (defn last-modified [customer-id db] (d/q '{:find [(max ?time) .] :in [$ ?cus-id %] :where [(owns? ?cus-id ?e) [?e ?a ?v ?tx] [?tx :db/txInstant ?time]]} db customer-id owner-rules))
  87. 87. If no customer-owned data changed, 304 25 (defn last-modified [customer-id db] (d/q '{:find [(max ?time) .] :in [$ ?cus-id %] :where [(owns? ?cus-id ?e) [?e ?a ?v ?tx] [?tx :db/txInstant ?time]]} db customer-id owner-rules))
  88. 88. If no customer-owned data changed, 304 25 (defn last-modified [customer-id db] (d/q '{:find [(max ?time) .] :in [$ ?cus-id %] :where [(owns? ?cus-id ?e) [?e ?a ?v ?tx] [?tx :db/txInstant ?time]]} db customer-id owner-rules))
  89. 89. 26 Cache hits are awesome, but what about cache misses?
  90. 90. 27 4 Mobile Sync
  91. 91. desired hypermedia from the API 28 GET /accounts/1223/purchases 200 OK {"purchases": [...] "_links": { "updates": "https://...?since=2014-11-10T11:12:13Z" } }
  92. 92. serve purchases added or modified since our last sync 29 (defn updated-purchases [account-id db since] (d/q '{:find [[(pull ?pur [*]) ...]] :in [$ $since ?acc] :where [[?pur :purchase/account ?acc] [$since ?pur]]} db (d/history (d/since db since)) [:account/id account-id])) (updated-purchases account-id db #inst "2014-11-14"))
  93. 93. serve purchases added or modified since our last sync 29 (defn updated-purchases [account-id db since] (d/q '{:find [[(pull ?pur [*]) ...]] :in [$ $since ?acc] :where [[?pur :purchase/account ?acc] [$since ?pur]]} db (d/history (d/since db since)) [:account/id account-id])) (updated-purchases account-id db #inst "2014-11-14"))
  94. 94. serve purchases added or modified since our last sync 29 (defn updated-purchases [account-id db since] (d/q '{:find [[(pull ?pur [*]) ...]] :in [$ $since ?acc] :where [[?pur :purchase/account ?acc] [$since ?pur]]} db (d/history (d/since db since)) [:account/id account-id])) (updated-purchases account-id db #inst "2014-11-14"))
  95. 95. serve purchases added or modified since our last sync 29 (defn updated-purchases [account-id db since] (d/q '{:find [[(pull ?pur [*]) ...]] :in [$ $since ?acc] :where [[?pur :purchase/account ?acc] [$since ?pur]]} db (d/history (d/since db since)) [:account/id account-id])) (updated-purchases account-id db #inst "2014-11-14"))
  96. 96. serve purchases added or modified since our last sync 29 (defn updated-purchases [account-id db since] (d/q '{:find [[(pull ?pur [*]) ...]] :in [$ $since ?acc] :where [[?pur :purchase/account ?acc] [$since ?pur]]} db (d/history (d/since db since)) [:account/id account-id])) (updated-purchases account-id db #inst "2014-11-14"))
  97. 97. serve purchases added or modified since our last sync 29 (defn updated-purchases [account-id db since] (d/q '{:find [[(pull ?pur [*]) ...]] :in [$ $since ?acc] :where [[?pur :purchase/account ?acc] [$since ?pur]]} db (d/history (d/since db since)) [:account/id account-id])) (updated-purchases account-id db #inst "2014-11-14"))
  98. 98. serve purchases added or modified since our last sync 29 (defn updated-purchases [account-id db since] (d/q '{:find [[(pull ?pur [*]) ...]] :in [$ $since ?acc] :where [[?pur :purchase/account ?acc] [$since ?pur]]} db (d/history (d/since db since)) [:account/id account-id])) (updated-purchases account-id db #inst "2014-11-14"))
  99. 99. 30 Bonus
  100. 100. 31 5 Future DBs
  101. 101. generating a virtual db (defn db-with-virtual-charges [purchases db] (:db-after (d/with db (mapcat purchase->virtual-charges purchases)))) 32
  102. 102. generating a virtual db (defn db-with-virtual-charges [purchases db] (:db-after (d/with db (mapcat purchase->virtual-charges purchases)))) 32
  103. 103. generating a virtual db (defn db-with-virtual-charges [purchases db] (:db-after (d/with db (mapcat purchase->virtual-charges purchases)))) 32
  104. 104. generating a virtual db (defn db-with-virtual-charges [purchases db] (:db-after (d/with db (mapcat purchase->virtual-charges purchases)))) 32
  105. 105. generating a virtual db (defn db-with-virtual-charges [purchases db] (:db-after (d/with db (mapcat purchase->virtual-charges purchases)))) 32
  106. 106. 33 6 Testing
  107. 107. db testing - locally scoped 34 (defn virtual-db [updates db] (:db-after (d/with db updates))) (let [one-account {...} one-card {...} one-purchase {...} wrong-purchase {...} db (virtual-db [one-account one-card one-purchase wrong-purchase] db)] (my-weird-db-query db) => [one-purchase])
  108. 108. db testing - locally scoped 34 (defn virtual-db [updates db] (:db-after (d/with db updates))) (let [one-account {...} one-card {...} one-purchase {...} wrong-purchase {...} db (virtual-db [one-account one-card one-purchase wrong-purchase] db)] (my-weird-db-query db) => [one-purchase])
  109. 109. db testing - locally scoped 34 (defn virtual-db [updates db] (:db-after (d/with db updates))) (let [one-account {...} one-card {...} one-purchase {...} wrong-purchase {...} db (virtual-db [one-account one-card one-purchase wrong-purchase] db)] (my-weird-db-query db) => [one-purchase])
  110. 110. db testing - locally scoped 34 (defn virtual-db [updates db] (:db-after (d/with db updates))) (let [one-account {...} one-card {...} one-purchase {...} wrong-purchase {...} db (virtual-db [one-account one-card one-purchase wrong-purchase] db)] (my-weird-db-query db) => [one-purchase])
  111. 111. db testing - locally scoped 34 (defn virtual-db [updates db] (:db-after (d/with db updates))) (let [one-account {...} one-card {...} one-purchase {...} wrong-purchase {...} db (virtual-db [one-account one-card one-purchase wrong-purchase] db)] (my-weird-db-query db) => [one-purchase])
  112. 112. db testing - locally scoped 34 (defn virtual-db [updates db] (:db-after (d/with db updates))) (let [one-account {...} one-card {...} one-purchase {...} wrong-purchase {...} db (virtual-db [one-account one-card one-purchase wrong-purchase] db)] (my-weird-db-query db) => [one-purchase])
  113. 113. db testing - vector of datoms 35 (fact "on last-modification-for" (let [customer-id (uuid) db [[42 :account/customer-id customer-id 99] [99 :db/txInstant #inst "2014-10-10T12:00:00.909Z" 99] [1337 :purchase/account 42 100] [1337 :purchase/id (uuid) 100] [1337 :purchase/merchant-name "Coiso" 100] [100 :db/txInstant #inst "2014-10-11T12:00:00.888Z" 100] [9001 :line-item/account 42 101] [9001 :charge/id (uuid) 101] [9001 :line-item/precise-amount 100.00M 101] [101 :db/txInstant #inst "2014-10-12T13:00:00.777Z" 101] [45 :account/customer-id (uuid) 102] [102 :db/txInstant #inst "2014-10-15T12:00:00.909Z" 102]]] (last-modification-for customer-id db) => #nu/time "2014-10-12T13:00:00.777Z"))
  114. 114. db testing - vector of datoms 35 (fact "on last-modification-for" (let [customer-id (uuid) db [[42 :account/customer-id customer-id 99] [99 :db/txInstant #inst "2014-10-10T12:00:00.909Z" 99] [1337 :purchase/account 42 100] [1337 :purchase/id (uuid) 100] [1337 :purchase/merchant-name "Coiso" 100] [100 :db/txInstant #inst "2014-10-11T12:00:00.888Z" 100] [9001 :line-item/account 42 101] [9001 :charge/id (uuid) 101] [9001 :line-item/precise-amount 100.00M 101] [101 :db/txInstant #inst "2014-10-12T13:00:00.777Z" 101] [45 :account/customer-id (uuid) 102] [102 :db/txInstant #inst "2014-10-15T12:00:00.909Z" 102]]] (last-modification-for customer-id db) => #nu/time "2014-10-12T13:00:00.777Z"))
  115. 115. 36 7 Schema Extension
  116. 116. schema is data! 37 (d/transact conn [{:db/valueType :db.type/string, :db.install/_attribute :db.part/db, :db/id (d/tempid :db.part/db), :db/cardinality :db.cardinality/one, :db/ident :customer/name }])
  117. 117. :nubank/transform :pii schema is data! 37 (d/transact conn [{:db/valueType :db.type/string, :db.install/_attribute :db.part/db, :db/id (d/tempid :db.part/db), :db/cardinality :db.cardinality/one, :db/ident :customer/name }])
  118. 118. 38 8 Sharding Reads
  119. 119. 39 sharding to the transactor customers accounts notification acquisition processor auth
  120. 120. 39 sharding to the transactor customers accounts notification acquisition processor auth
  121. 121. customers 1 2 … 39 sharding to the transactor customers accounts notification acquisition processor auth
  122. 122. customers 1 2 … 39 sharding to the transactor customers accounts notification acquisition processor auth shard-specific peer cache
  123. 123. customers 1 2 … 39 sharding to the transactor customers accounts notification acquisition processor auth shard-specific peer cache inter-shard ACID transactions
  124. 124. 40 9 Db Aggregation
  125. 125. 41 data science aggregation customers accounts notification acquisition processor auth data science
  126. 126. 42 querying multiple databases (defn multi-join [customer-id cus-db acc-db acq-db ntf-db] (d/q '{:find [...] :in [$cus $acc $acq $ntf ?cus-id] :where [[$cus ?cus :customer/id ?cus-id] [$acc ?acc :account/customer-id ?cus-id] [$acq ?ar :request/customer-id ?cus-id] [$ntf ?evt :event/customer-id ?cus-id] [...]]} cus-db acc-db acq-db ntf-db customer-id))
  127. 127. 42 querying multiple databases (defn multi-join [customer-id cus-db acc-db acq-db ntf-db] (d/q '{:find [...] :in [$cus $acc $acq $ntf ?cus-id] :where [[$cus ?cus :customer/id ?cus-id] [$acc ?acc :account/customer-id ?cus-id] [$acq ?ar :request/customer-id ?cus-id] [$ntf ?evt :event/customer-id ?cus-id] [...]]} cus-db acc-db acq-db ntf-db customer-id))
  128. 128. 43 1 Audit Trail 2 Authorization 3 HTTP Cache 4 Mobile Sync 5 Future DBs 6 Testing 7 Schema Extension 8 Sharding Reads 9 Db Aggregation
  129. 129. 44 We’re hiring
  130. 130. 45 We’re hiring
  131. 131. Thanks! Lucas Cavalcanti & Edward Wible @lucascs lucas@nubank.com.br edward@nubank.com.br

×