Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

The algebra of library design

261 views

Published on

Talk at the Sydney Clojure User Group, October 2014 exploring the principles behind the deign of imminent - https://github.com/leonardoborges/imminent - a Clojure library for composable futures

Published in: Technology
  • Be the first to comment

  • Be the first to like this

The algebra of library design

  1. 1. The algebra of library design #cljsyd - October 2014 Leonardo Borges @leonardo_borges www.leonardoborges.com www.thoughtworks.com
  2. 2. Algebra?
  3. 3. More specifically… • Algebraic structures, studied in Abstract Algebra • a set with one or more operations on it • Category theory is another way to study these structures
  4. 4. Overview • The problem with Clojure futures • A better futures library • Functors, Applicative Functors and Monads • What about core.async (and others?)
  5. 5. The beginning… (def age (future (do (Thread/sleep 2000)! 31)))! (prn "fetching age...")! ! (def double-age (* 2 @age))! (prn "age, doubled: " double-age)! ! (prn "doing something else important...")
  6. 6. Do you see any issues?
  7. 7. The beginning… (def age (future (do (Thread/sleep 2000)! 31)))! (prn "fetching age...")! ! (def double-age (* 2 @age))! (prn "age, doubled: " double-age)! ! (prn "doing something else important...")! ! ;; "fetching age..."! ;; "age, doubled: " 62! ;; "doing something else important..."!
  8. 8. It looks like we would like to execute something once the Future has completed
  9. 9. A first glance at imminent (require '[imminent.core :as i])! ! (def age (i/future (do (Thread/sleep 2000)! 31)))! (prn "fetching age...")! (def double-age (i/map age #(* % 2)))! (i/on-success double-age #(prn "age, doubled: " %))! ! (prn "doing something else important...")
  10. 10. A first glance at imminent (require '[imminent.core :as i])! ! (def age (i/future (do (Thread/sleep 2000)! 31)))! (prn "fetching age...")! (def double-age (i/map age #(* % 2)))! (i/on-success double-age #(prn "age, doubled: " %))! ! (prn "doing something else important...")! ! ;; "fetching age..."! ;; "doing something else important..."! ;; "age, doubled: " 62!
  11. 11. Functor class Functor f where! fmap :: (a -> b) -> f a -> f b
  12. 12. Another example (def age (future (do (Thread/sleep 2000)! 31)))! (def name (future (do (Thread/sleep 2000)! "Leonardo")))! (prn "fetching name...")! (prn (format "%s is %s years old" @name @age))! (prn "more important things going on...")
  13. 13. Same as before, only this time we want to execute the code once both futures have completed
  14. 14. Rewriting it in imminent (def age (i/future (do (Thread/sleep 2000)! 31)))! (def name (i/future (do (Thread/sleep 2000)! "Leonardo")))! (prn "fetching name...")! (def both (i/sequence [name age]))! (i/on-success both ! (fn [[name age]]! (prn (format "%s is %s years old" name age))))! ! (prn "more important things going on...")! !
  15. 15. Rewriting it in imminent (def age (i/future (do (Thread/sleep 2000)! 31)))! (def name (i/future (do (Thread/sleep 2000)! "Leonardo")))! (prn "fetching name...")! (def both (i/sequence [name age]))! (i/on-success both ! (fn [[name age]]! (prn (format "%s is %s years old" name age))))! ! (prn "more important things going on...")! ! ;; "fetching name..."! ;; "more important things going on..."! ;; "Leonardo is 31 years old"!
  16. 16. Monad class Monad m where! (>>=) :: m a -> (a -> m b) -> m b! return :: a -> m a
  17. 17. Monad class Monad m where! (>>=) :: m a -> (a -> m b) -> m b! return :: a -> m a (>>=) is also called bind, flatmap, selectMany and mapcat…
  18. 18. Monad - derived functions liftM2 :: (Monad m) => (a1 -> a2 -> r) -> m a1 -> m a2 -> m r
  19. 19. Monad - derived functions liftM2 :: (Monad m) => (a1 -> a2 -> r) -> m a1 -> m a2 -> m r (defn mlift2! [f]! (fn [ma mb]! (flatmap ma! (fn [a]! (flatmap mb! (fn [b]! (pure (f a b))))))))
  20. 20. Monad - derived functions sequence :: Monad m => [m a] -> m [a]
  21. 21. Monad - derived functions sequence :: Monad m => [m a] -> m [a] (defn sequence! [ms]! (reduce (mlift2 conj)! (pure [])! ms))
  22. 22. Monad - derived functions mapM :: Monad m => (a -> m b) -> [a] -> m [b]
  23. 23. Monad - derived functions mapM :: Monad m => (a -> m b) -> [a] -> m [b] (defn mmap! [f vs]! (sequence (map f vs)))
  24. 24. Monad - derived functions mapM :: Monad m => (a -> m b) -> [a] -> m [b] (defn mmap! [f vs]! (sequence (map f vs))) plus a bunch of others…
  25. 25. Let’s look at a more concrete example
  26. 26. Aggregating movie data (defn cast-by-movie [name]! (future (do (Thread/sleep 5000)! (:cast movie))))! ! (defn movies-by-actor [name]! (do (Thread/sleep 2000)! (->> actor-movies! (filter #(= name (:name %)))! first)))! ! (defn spouse-of [name]! (do (Thread/sleep 2000)! (->> actor-spouse! (filter #(= name (:name %)))! first)))! ! (defn top-5 []! (future (do (Thread/sleep 5000)! top-5-movies)))!
  27. 27. The output ({:name "Cate Blanchett",! :spouse "Andrew Upton",! :movies! ("Lord of The Rings: The Fellowship of The Ring - (top 5)"...)}! {:name "Elijah Wood",! :spouse "Unknown",! :movies! ("Eternal Sunshine of the Spotless Mind"...)}! ...)
  28. 28. One possible solution (let [cast (cast-by-movie "Lord of The Rings: The Fellowship of The Ring")! movies (pmap movies-by-actor @cast)! spouses (pmap spouse-of @cast)! top-5 (top-5)]! (prn "Fetching data...")! (pprint (aggregate-actor-data spouses movies @top-5)))
  29. 29. One possible solution (let [cast (cast-by-movie "Lord of The Rings: The Fellowship of The Ring")! movies (pmap movies-by-actor @cast)! spouses (pmap spouse-of @cast)! top-5 (top-5)]! (prn "Fetching data...")! (pprint (aggregate-actor-data spouses movies @top-5))) ;; "Elapsed time: 10049.334 msecs"
  30. 30. We face the same problems as before
  31. 31. Re-arranging the code would improve it
  32. 32. But the point is not having to do so
  33. 33. In imminent (defn cast-by-movie [name]! (i/future (do (Thread/sleep 5000)! (:cast movie))))! ! (defn movies-by-actor [name]! (i/future (do (Thread/sleep 2000)! (->> actor-movies! (filter #(= name (:name %)))! first))))! ! (defn spouse-of [name]! (i/future (do (Thread/sleep 2000)! (->> actor-spouse! (filter #(= name (:name %)))! first))))! ! (defn top-5 []! (i/future (do (Thread/sleep 5000)! top-5-movies)))
  34. 34. In imminent (let [cast (cast-by-movie "Lord of The Rings: The Fellowship of The Ring")! movies (i/flatmap cast #(i/map-future movies-by-actor %))! spouses (i/flatmap cast #(i/map-future spouse-of %))! result (i/sequence [spouses movies (top-5)])]! (prn "Fetching data...")! (pprint (apply aggregate-actor-data! (i/dderef (i/await result)))))
  35. 35. In imminent (let [cast (cast-by-movie "Lord of The Rings: The Fellowship of The Ring")! movies (i/flatmap cast #(i/map-future movies-by-actor %))! spouses (i/flatmap cast #(i/map-future spouse-of %))! result (i/sequence [spouses movies (top-5)])]! (prn "Fetching data...")! (pprint (apply aggregate-actor-data! (i/dderef (i/await result))))) ;; "Elapsed time: 7087.603 msecs"
  36. 36. Everything is asynchronous by default
  37. 37. We can optionally block and wait for a result using the “i/await” function
  38. 38. Ok, that’s cool! But what about Applicative Functors?
  39. 39. Writing this sucks (defn int-f [n]! (i/future (do (Thread/sleep 2000)! (* 2 n))))! ! ! (-> (i/bind! (int-f 10)! (fn [a] (i/bind! (int-f a)! (fn [b] (i/bind! (int-f b)! (fn [c] ! (i/pure (+ a b c)))))))
  40. 40. Monadic “do” notation (i/mdo [a (int-f 10)! b (int-f a)! c (int-f b)]! (i/pure (+ a b c)))
  41. 41. Monadic “do” notation (i/mdo [a (int-f 10)! b (int-f a)! c (int-f b)]! (i/pure (+ a b c))) How long does this computation take?
  42. 42. Monadic “do” notation (i/mdo [a (int-f 10)! b (int-f a)! c (int-f b)]! (i/pure (+ a b c))) How long does this computation take? ;; "Elapsed time: 6002.39 msecs"
  43. 43. What if the computations don’t depend on each other?
  44. 44. Monadic “do” notation (i/mdo [a (int-f 10)! b (int-f 20)! c (int-f 30)]! (i/pure (+ a b c)))
  45. 45. Monadic “do” notation (i/mdo [a (int-f 10)! b (int-f 20)! c (int-f 30)]! (i/pure (+ a b c))) How long does this take now?
  46. 46. Monadic “do” notation (i/mdo [a (int-f 10)! b (int-f 20)! c (int-f 30)]! (i/pure (+ a b c))) How long does this take now? ;; "Elapsed time: 6002.39 msecs"
  47. 47. ?
  48. 48. The ‘do-notation’ is used to sequence monadic steps so we remain serial, but still asynchronous
  49. 49. Applicative Functor class Functor f => Applicative f where! pure :: a -> f a! (<*>) :: f (a -> b) -> f a -> f b!
  50. 50. Previous example, rewritten in applicative style (i/<*> (i/map (int-f 10) (curry + 3))! (int-f 20)! (int-f 30)) ;;"Elapsed time: 2001.509 msecs"
  51. 51. It turns out this pattern is a derived function from Applicative Functors
  52. 52. Applicative Functor - derived functions liftA2 :: Applicative f => (a -> b -> c) -> f a -> f b -> f c
  53. 53. Applicative Functor - derived functions liftA2 :: Applicative f => (a -> b -> c) -> f a -> f b -> f c (defn alift! [f]! (fn [& as]! {:pre [(seq as)]}! (let [curried (curry f (count as))]! (apply <*>! (fmap curried (first as))! (rest as)))))
  54. 54. “alift” assumes every Applicative Functor is also Functor
  55. 55. Previous example, rewritten using “alift" ;;"Elapsed time: 2001.509 msecs" ((i/alift +) (int-f 10) (int-f 20) (int-f 30))
  56. 56. It looks a lot like function application :)
  57. 57. Applicatives give us the parallel semantics without giving up convenience
  58. 58. Ok, this is awesome! But we have core.async!
  59. 59. The movies example, revisited (defn cast-by-movie [name]! (let [c (chan)]! (go (<! (timeout 5000))! (>! c (:cast movie))! (close! c))! c))! ! (defn movies-by-actor [name]! (let [c (chan)]! (go (<! (timeout 2000))! (>! c (->> actor-movies! (filter #(= name (:name %)))! first))! (close! c))! c))
  60. 60. The movies example, revisited (defn spouse-of [name]! (let [c (chan)]! (go (<! (timeout 2000))! (>! c (->> actor-spouse! (filter #(= name (:name %)))! first))! (close! c))! c))! ! (defn top-5 []! (let [c (chan)]! (go (<! (timeout 5000))! (>! c top-5-movies)! (close! c))! c))! ! ! (defn async-pmap [f source]! (go (->> (map f (<! source))! async/merge! (async/into [])! <!)))
  61. 61. The movies example, revisited (let [cast (cast-by-movie "Lord of The Rings: The Fellowship of The Ring")! m-cast (mult cast)! movies (async-pmap movies-by-actor (tap m-cast (chan)))! spouses (async-pmap spouse-of (tap m-cast (chan)))! top-5 (top-5)]! (prn "Fetching data...")! (pprint (<!! (go (aggregate-actor-data (<! spouses)! (<! movies)! (<! top-5))))))! ! ;; "Elapsed time: 7088.834 msecs"
  62. 62. “mult” and “tap” have no business being in that code
  63. 63. they are necessary because channels are single-take containers
  64. 64. Exceptions core.async swallows them by default (defn movies-by-actor [name]! (let [c (chan)]! (go (<! (timeout 2000))! (throw (Exception. (str "Error fetching movies for actor " name)))! (>! c (->> actor-movies! (filter #(= name (:name %)))! first))! (close! c))! c))
  65. 65. Exceptions core.async swallows them by default (defn movies-by-actor [name]! (let [c (chan)]! (go (<! (timeout 2000))! (throw (Exception. (str "Error fetching movies for actor " name)))! (>! c (->> actor-movies! (filter #(= name (:name %)))! first))! (close! c))! c)) (let [cast (cast-by-movie "Lord of The Rings: The Fellowship of The Ring")! m-cast (mult cast)! movies (async-pmap movies-by-actor (tap m-cast (chan)))! spouses (async-pmap spouse-of (tap m-cast (chan)))! top-5 (top-5)]! (prn "Fetching data...")! (pprint (<!! (go (aggregate-actor-data (<! spouses)! (<! movies)! (<! top-5))))))! ! ;; nil - WTF???"
  66. 66. Exceptions - workaround (defn throw-err [e]! (when (instance? Throwable e) (throw e))! e)! ! (defmacro <? [ch]! `(throw-err (<! ~ch)))! ! ! (defn movies-by-actor [name]! (let [c (chan)]! (go (try (do (<! (timeout 2000))! (throw (Exception. (str "Error fetching movies for actor " name)))! (>! c (->> actor-movies! (filter #(= name (:name %)))! first))! (close! c))! (catch Exception e! (do (>! c e)! (close! c)))))! c))
  67. 67. Exceptions - workaround (let [cast (cast-by-movie "Lord of The Rings: The Fellowship of The Ring")! m-cast (mult cast)! movies (async-pmap movies-by-actor (tap m-cast (chan)))! spouses (async-pmap spouse-of (tap m-cast (chan)))! top-5 (top-5)]! (prn "Fetching data...")! (pprint (<!! (go (try! (aggregate-actor-data (<? spouses)! (<? movies)! (<? top-5))! (catch Exception e! e)))))))
  68. 68. A lot of effort for not much benefit
  69. 69. Exceptions - the imminent way * * not really unique to imminent. Other libraries use this same approach (defn movies-by-actor [name]! (i/future (do (Thread/sleep 2000)! (throw (Exception. (str "Error fetching movies for actor " name)))! (->> actor-movies! (filter #(= name (:name %)))! first))))
  70. 70. Exceptions - the imminent way # 1 (let [cast (cast-by-movie "Lord of The Rings: The Fellowship of The Ring")! movies (i/flatmap cast #(i/map-future movies-by-actor %))! spouses (i/flatmap cast #(i/map-future spouse-of %))! results (i/sequence [spouses movies (top-5)])! _ (prn "Fetching data...")! result (deref (i/await results))]! ! (i/map result #(pprint (apply aggregate-actor-data %)))! (i/map-failure result #(pprint (str "Oops: " %))))) ;; Oops: java.lang.Exception: Error fetching movies for actor Cate Blanchett
  71. 71. Exceptions - the imminent way # 1 (let [cast (cast-by-movie "Lord of The Rings: The Fellowship of The Ring")! movies (i/flatmap cast #(i/map-future movies-by-actor %))! spouses (i/flatmap cast #(i/map-future spouse-of %))! results (i/sequence [spouses movies (top-5)])! _ (prn "Fetching data...")! result (deref (i/await results))]! ! (i/map result #(pprint (apply aggregate-actor-data %)))! (i/map-failure result #(pprint (str "Oops: " %))))) imminent results are themselves functors :) ;; Oops: java.lang.Exception: Error fetching movies for actor Cate Blanchett
  72. 72. Exceptions - the imminent way # 2 (let [cast (cast-by-movie "Lord of The Rings: The Fellowship of The Ring")! movies (i/flatmap cast #(i/map-future movies-by-actor %))! spouses (i/flatmap cast #(i/map-future spouse-of %))! result (i/sequence [spouses movies (top-5)])]! (prn "Fetching data...")! (i/on-success result #(prn-to-repl (apply aggregate-actor-data %)))! (i/on-failure result #(prn-to-repl (str "Oops: " %))))! ;; Oops: java.lang.Exception: Error fetching movies for actor Cate Blanchett
  73. 73. Final Thoughts Category Theory helps you design libraries with higher code reuse due to well known and understood abstractions
  74. 74. Final Thoughts Imminent takes advantage of that and provides an asynchronous and composable Futures library for Clojure
  75. 75. Final Thoughts Even when compared to core.async, imminent still makes sense unless you need the low level granularity of channels and/or coordination via queues
  76. 76. Questions? Leonardo Borges @leonardo_borges www.leonardoborges.com www.thoughtworks.com

×