3 years with Clojure

21,401 views

Published on

An experience report from 3 years of using Clojure

Published in: Software

3 years with Clojure

  1. 1. Life of an immutant 3 years with Clojure
  2. 2. About me • @michaelklishin • github.com/michaelklishin
  3. 3. Setting expectations
  4. 4. Setting expectations • This is an experience report
  5. 5. Setting expectations • This is an experience report • Not a language primer
  6. 6. Setting expectations • This is an experience report • Not a language primer • Assumes no familiarity except basic facts
  7. 7. Setting expectations • This is an experience report • Not a language primer • Assumes no familiarity except basic facts • ~200 slides.This is gonna be fast.
  8. 8. Background
  9. 9. Background • Data processing
  10. 10. Background • Data processing • Scheduling systems
  11. 11. Background • Data processing • Scheduling systems • Specialized Web crawlers
  12. 12. Background • Data processing • Scheduling systems • Specialized Web crawlers • Billing, accounting
  13. 13. Background • Data processing • Scheduling systems • Specialized Web crawlers • Billing, accounting • Infrastructure automation
  14. 14. Wanted
  15. 15. Wanted • An environment that does not suck
  16. 16. Wanted • An environment that does not suck is operationally sane
  17. 17. Wanted • An environment that does not suck is operationally sane • A runtime that has decent tooling (profilers, concurrency-related metrics, GC metrics)
  18. 18. Wanted • An environment that does not suck is operationally sane • A runtime that has decent tooling (profilers, concurrency-related metrics, GC metrics) • A runtime that yields good baseline performance
  19. 19. Wanted • An environment that does not suck is operationally sane • A runtime that has decent tooling (profilers, concurrency-related metrics, GC metrics) • A runtime that yields good baseline performance • Painless upgrade paths
  20. 20. Wanted • Community
  21. 21. Wanted • Community • Benefit from existing libraries and tools
  22. 22. Wanted • Community • Benefit from existing libraries and tools • Concurrency and parallelism
  23. 23. Wanted • Community • Benefit from existing libraries and tools • Concurrency and parallelism • A productive, concise language
  24. 24. Wanted JVM
  25. 25. JVM
  26. 26. JVM • One of the most mature runtimes you can find
  27. 27. JVM • One of the most mature runtimes you can find • Excellent tooling
  28. 28. JVM • One of the most mature runtimes you can find • Excellent tooling (VisualVM,YourKit, jstack, jhat,VM/+XX flags, BTrace, …)
  29. 29. JVM • One of the most mature runtimes you can find • Excellent tooling • Excellent baseline performance
  30. 30. JVM • One of the most mature runtimes you can find • Excellent tooling • Excellent baseline performance • Multilingual: Clojure, Java, JRuby, Scala, JS implementations
  31. 31. Clojure
  32. 32. Clojure • Very high level, concise, expressive
  33. 33. Clojure • Very high level, concise, expressive • Excellent concurrency/parallelism story
  34. 34. Clojure • Very high level, concise, expressive • Excellent concurrency/parallelism story • Very good baseline performance
  35. 35. Clojure • Very high level, concise, expressive • Excellent concurrency/parallelism story • Very good baseline performance • Consistent, well designed
  36. 36. Clojure • Very high level, concise, expressive • Excellent concurrency/parallelism story • Very good baseline performance • Consistent, well designed • Decent tooling
  37. 37. Clojure • Very high level, concise, expressive • Excellent concurrency/parallelism story • Very good baseline performance • Consistent, well designed • Decent tooling • Fun to work with (motivation)
  38. 38. Clojure • No bored academics, built by a practicioner
  39. 39. Clojure: concurrency/ parallelism story
  40. 40. Clojure: concurrency/ parallelism story • Immutable data structures by default
  41. 41. Clojure: concurrency/ parallelism story • Immutable data structures by default • Identity/values separation
  42. 42. Clojure: concurrency/ parallelism story • Immutable data structures by default • Identity/values separation • Concurrency-aware reference types
  43. 43. Clojure: concurrency/ parallelism story • Immutable data structures by default • Identity/values separation • Concurrency-aware reference types • java.util.concurrent
  44. 44. Clojure: concurrency/ parallelism story When I find myself in times of trouble Prof. Doug Lea comes to me Coding lines of wisdom j.u.c.
  45. 45. Clojure: concurrency/ parallelism story • Immutable data structures by default • Identity/values separation • Concurrency-aware reference types • java.util.concurrent • Reactor,Akka, Disruptor, RxJava…
  46. 46. Clojure: concurrency/ parallelism story • Async I/O: NIO, Netty,Vert.x…
  47. 47. Clojure: concurrency/ parallelism story • You have choice
  48. 48. Clojure: concurrency/ parallelism story • You have choice • Runtime parallelism (on JVM, CLR)
  49. 49. Clojure: concurrency/ parallelism story • You have choice • Runtime parallelism (on JVM, CLR) • Hundreds, thousands of threads at the same time
  50. 50. Clojure: baseline performance
  51. 51. Clojure: baseline performance • What matters the most is what bytecode the compiler produces
  52. 52. contextneeded.com
  53. 53. Clojure: baseline performance • What matters the most is what bytecode the compiler produces • Java interop bytecode is typically the same as produced by javac
  54. 54. Clojure: baseline performance • What matters the most is what bytecode the compiler produces • Java interop bytecode is typically the same as produced by javac • Watch out for reflective calls, primitive boxing
  55. 55. Clojure: baseline performance • Optional type hints
  56. 56. Clojure: baseline performance • Optional type hints • Type inference in simple cases
  57. 57. Clojure: baseline performance (let [d (java.util.Date.)] d)
  58. 58. Clojure: baseline performance (let [d (java.util.Date.)] d)
  59. 59. Clojure: baseline performance • Optional type hints • Type inference in simple cases
  60. 60. Clojure: baseline performance • Optional type hints • Type inference in simple cases • If JVM can optimize your bytecode, things run screaming fast
  61. 61. Clojure: baseline performance • Optional type hints • Type inference in simple cases • If JVM can optimize your bytecode, things run screaming fast • From 2% (Java lib wrappers) to 200% (naïve numerics heavy code, tight loops) penalty compared to Java
  62. 62. Clojure: baseline performance • Immutable data structures have O(log32 n) access complexity
  63. 63. Clojure: baseline performance • Immutable data structures have O(log32 n) access complexity • Immutable data structures have associated GC tax
  64. 64. Clojure: baseline performance • Immutable data structures have O(log32 n) access complexity • Immutable data structures have associated GC tax • Focus on exploiting parallelism over squeezing µs from a single thread
  65. 65. Clojure: baseline performance • Poor startup time, excellent execution performance
  66. 66. Clojure: baseline performance • Poor startup time, excellent execution performance • Anecdotal evidence: 10 — 50 times throughput improvements compared to Ruby in real apps
  67. 67. Clojure: stability
  68. 68. Clojure: stability • Have been using 1.3 since alpha1
  69. 69. Clojure: stability • Have been using 1.3 since alpha1 • Hit by 4 Clojure bugs in ~3 years
  70. 70. Clojure: stability • Have been using 1.3 since alpha1 • Hit by 4 Clojure bugs in ~3 years • 1 bug is an edge case that is not worth fixing (I did something really stupid)
  71. 71. Clojure: stability • Have been using 1.3 since alpha1 • Hit by 4 Clojure bugs in ~3 years • 1 bug is an edge case that is not worth fixing (I did something really stupid) • Alphas more solid than GA releases of almost all other languages I’ve worked with
  72. 72. Clojure: stability • Hit by 4 Clojure bugs in ~3 years • 1 bug is an edge case that is not worth fixing (I did something really stupid) • Alphas more solid than GA releases of almost all other languages I’ve worked with • Others (e.g.ThoughtWorks) report the same experience
  73. 73. Clojure: stability • Upgraded 28 apps & libraries from 1.3 to 1.4. Just 1 change due to a tricky 3rd party library.The rest was a drop-in replacement.
  74. 74. Clojure: stability • Upgraded 28 apps & libraries from 1.3 to 1.4. Just 1 change due to a tricky 3rd party library.The rest was a drop-in replacement. • When upgrading is this easy, people do it happily
  75. 75. Clojure: stability • Upgraded 28 apps & libraries from 1.3 to 1.4. Just 1 change due to a tricky 3rd party library.The rest was a drop-in replacement. • When upgrading is this easy, people do it happily • No library binary compatibility hell some other JVM languages suffer from
  76. 76. Clojure: stability • Many active developers are conservative with versioning & calling things “done”
  77. 77. Clojure: stability • Many active developers are conservative with versioning & calling things “done” • Fits JVM ecosystem maturity really well
  78. 78. Clojure: stability • The real reason: Clojure is small and was designed to be simple
  79. 79. Clojure: stability • No monkey patching (there are features that give you all the good parts of teh)
  80. 80. Clojure: consistency
  81. 81. Clojure: consistency • Collections/sequences library is very unified
  82. 82. Clojure: consistency • Collections/sequences library is very unified • Easy to implement your own data types that work just like core Clojure types
  83. 83. Clojure: consistency • Collections/sequences library is very unified • Easy to implement your own data types that work just like core Clojure types • Almost no “bad baggage” in the core
  84. 84. Clojure: consistency • Collections/sequences library is very unified • Easy to implement your own data types that work just like core Clojure types • Almost no “bad baggage” in the core • The language was actually designed
  85. 85. Clojure: deployment
  86. 86. Clojure: deployment • Pretty standard JVM deployment options
  87. 87. Clojure: deployment • Pretty standard JVM deployment options • Compile everything AOT into a single jar (“überjar”), including dependencies, assets, etc
  88. 88. Clojure: deployment • Pretty standard JVM deployment options • Compile everything AOT into a single jar (“überjar”), including dependencies, assets, etc • A tiny shell script + start-stop-daemon
  89. 89. Clojure: deployment • Pretty standard JVM deployment options • Compile everything AOT into a single jar (“überjar”), including dependencies, assets, etc • A tiny shell script + start-stop-daemon • Chef or Capistrano
  90. 90. Clojure: deployment • CloudFoundry
  91. 91. Clojure: deployment • CloudFoundry • Heroku
  92. 92. Clojure: deployment • CloudFoundry • Heroku • Immutant/JBoss AS
  93. 93. Clojure: deployment • CloudFoundry • Heroku • Immutant/JBoss AS • OpenShift
  94. 94. Clojure: operations
  95. 95. Clojure: operations • VisualVM, jstack, jheap, …
  96. 96. Clojure: operations • VisualVM, jstack, jheap, … • nREPL
  97. 97. Clojure: operations • VisualVM, jstack, jheap, … • nREPL • Remote REPL over HTTP(S)
  98. 98. Clojure: operations • VisualVM, jstack, jheap, … • nREPL • Remote REPL over HTTP(S) • Not quite Erlang but still impressive
  99. 99. Clojure: operations • VisualGC gives real insight into what is going on with object allocation, tenuring, death rate
  100. 100. Clojure: operations • VisualGC gives real insight into what is going on with object allocation, tenuring, death rate • JVM DTrace support (Solaris, SmartOS, FreeBSD, OS X)
  101. 101. Clojure: operations • VisualGC gives real insight into what is going on with object allocation, tenuring, death rate • JVM DTrace support (Solaris, SmartOS, FreeBSD, OS X) • BTrace
  102. 102. Clojure: community
  103. 103. Clojure: community • Full of intelligent engineers who don’t give two shits about Hacker News
  104. 104. Clojure: community • Full of intelligent engineers who don’t give two shits about Hacker News • Over 10 conferences to date
  105. 105. Clojure: community • Full of intelligent engineers who don’t give two shits about Hacker News • Over 10 conferences to date • ~10K people on the mailing list
  106. 106. Clojure: community • Full of intelligent engineers who don’t give two shits about Hacker News • Over 10 conferences to date • ~10K people on the mailing list • Users from small and not-so-small startups to Fortune 50 corporations in finance, biotech, retail
  107. 107. Clojure: fun to work with
  108. 108. Clojure: fun to work with • Profoundly changes the way you think
  109. 109. Clojure: changes the way you think
  110. 110. Clojure: changes the way you think • About managing your program state
  111. 111. Clojure: changes the way you think • About managing your program state • About the value of experimenting (REPL driven development)
  112. 112. Clojure: changes the way you think • About managing your program state • About the value of experimenting (REPL driven development) • About metaprogramming
  113. 113. Clojure: changes the way you think • About managing your program state • About the value of experimenting (REPL driven development) • About metaprogramming • About isolating side effects
  114. 114. Clojure: changes the way you think
  115. 115. Clojure: changes the way you think • Program = pure core + parts that communicate with the outside world (network, disk I/O)
  116. 116. Clojure: changes the way you think • Program = pure core + parts that communicate with the outside world (network, disk I/O) • Pure parts are ridiculously easy to reason about, test, experiment with and reuse
  117. 117. Clojure: changes the way you think • Program = pure core + parts that communicate with the outside world (network, disk I/O) • Pure parts are ridiculously easy to reason about, test, experiment with and reuse • Typical Rails app: 95% of methods have side effects
  118. 118. Clojure: changes the way you think • Typical Clojure app: 40% of functions have side effects
  119. 119. Clojure: changes the way you think • Typical Clojure app: 40% of functions have side effects • Anecdotal evidence: 3-4 times less time spent on tests
  120. 120. Clojure: changes the way you think • Typical Clojure app: 40% of functions have side effects • Anecdotal evidence: 3-4 times less time spent on tests • Moral of the story: make more of your code not have side effects
  121. 121. Clojure: fun to work with • Profoundly changes the way you think
  122. 122. Clojure: fun to work with • Profoundly changes the way you think • Motivates you to learn
  123. 123. Clojure: fun to work with • Profoundly changes the way you think • Motivates you to learn • Easy things are easy, hard things are possible
  124. 124. Clojure: fun to work with • Profoundly changes the way you think • Motivates you to learn • Easy things are easy, hard things are possible • Wheels are already invented in clojure.core or JDK
  125. 125. Clojure: fun to work with • Stable: fewer distractions, get in the flow
  126. 126. Clojure: fun to work with • Stable: fewer distractions, get in the flow • REPL driven development keeps you in the flow for longer periods
  127. 127. Clojure: fun to work with • Stable: fewer distractions, get in the flow • REPL driven development keeps you in the flow for longer periods • Polymorphism done right <3
  128. 128. Clojure: fun to work with “Clojure demands that you raise your game and rewards you greatly for it…”
  129. 129. Clojure: fun to work with • The community is full of smart and down to earth people.
  130. 130. Clojure: use cases
  131. 131. Clojure: use cases • Data processing
  132. 132. Clojure: use cases • Data processing • Stream processing
  133. 133. Clojure: use cases • Data processing • Stream processing • Micro-services
  134. 134. Clojure: use cases • Data processing • Stream processing • Micro-services • Machine learning
  135. 135. Clojure: use cases • Data processing • Stream processing • Micro-services • Machine learning • Web development
  136. 136. Clojure: misc
  137. 137. Clojure: misc • Leiningen (leiningen.org) is excellent
  138. 138. Clojure: misc • Leiningen (leiningen.org) is excellent • Deps management, build tool, custom tasks are just Clojure functions
  139. 139. Clojure: misc • Leiningen (leiningen.org) is excellent • Deps management, build tool, custom tasks are just Clojure functions • Packages your code + all deps into a single jar for deployment
  140. 140. Clojure: misc • Leiningen (leiningen.org) is excellent • Deps management, build tool, custom tasks are just Clojure functions • Packages your code + all deps into a single jar for deployment • Makes it trivial to develop against multiple Clojure versions
  141. 141. Clojure: misc • clojars.org
  142. 142. Clojure: misc • clojars.org • Deploy via SSH (SCP) or HTTPS
  143. 143. Clojure: misc • clojars.org • Deploy via SSH (SCP) or HTTPS • Very easy to deploy snapshot releases
  144. 144. Clojure: misc • clojars.org • Deploy via SSH (SCP) or HTTPS • Very easy to deploy snapshot releases • Clojars and Leiningen are maintained by the same core team
  145. 145. Clojure: misc • travis-ci.org support
  146. 146. Clojure: misc • travis-ci.org support • Test against multiple JDKs
  147. 147. Clojure: ecosystem • Feels a bit like Ruby circa 2009
  148. 148. Clojure: ecosystem • Feels a bit like Ruby circa 2009 • But with the entire JVM ecosystem at your reach
  149. 149. Clojure: ecosystem • Feels a bit like Ruby circa 2009 • But with the entire JVM ecosystem at your reach • And Leiningen + clojars.org
  150. 150. Clojure: ecosystem • Feels a bit like Ruby circa 2009 • But with the entire JVM ecosystem at your reach • And Leiningen + clojars.org • Community ~ doubles every year in size
  151. 151. Clojure: ecosystem • Feels a bit like Ruby circa 2009 • But with the entire JVM ecosystem at your reach • And Leiningen + clojars.org • Community ~ doubles every year in size • Google for “The State of Clojure” survey
  152. 152. Clojure: ecosystem • ~35% of people come from Java, ~35% from Ruby or Python
  153. 153. Clojure: ecosystem • ~35% of people come from Java, ~35% from Ruby or Python • Libraries, not frameworks
  154. 154. Clojure: ecosystem • ~35% of people come from Java, ~35% from Ruby or Python • Libraries, not frameworks • Decent libraries just for almost every problem
  155. 155. Clojure: ecosystem • ~35% of people come from Java, ~35% from Ruby or Python • Libraries, not frameworks • Decent libraries just for almost every problem • 15 books, 2 outdated (cover 1.2)
  156. 156. Clojure: ecosystem • ~35% of people come from Java, ~35% from Ruby or Python • Libraries, not frameworks • Decent libraries just for almost every problem • 15 books, 2 outdated (cover 1.2) • clojure-doc.org, tryclj.com, …
  157. 157. Clojure: ecosystem
  158. 158. Clojure: what sucks
  159. 159. Clojure: what sucks • clojure.org
  160. 160. Clojure: what sucks • clojure.org • Documentation (guides, the ref is mostly OK, books are great)
  161. 161. Clojure: what sucks • clojure.org • Documentation (guides, the ref is mostly OK, books are great) • Compiler messages can be cryptic
  162. 162. Clojure: what sucks • clojure.org • Documentation (guides, the ref is mostly OK, books are great) • Compiler messages can be cryptic • JVM startup time + code compilation on each run (without REPL or tools such as Drip)
  163. 163. Clojure: what sucks • Paper contributor agreement
  164. 164. ClojureWerkz
  165. 165. ClojureWerkz “A growing collection of open source Clojure libraries…”
  166. 166. ClojureWerkz “It just werkz…”
  167. 167. ClojureWerkz • Modern targets (Clojure 1.5+, JDK 6)
  168. 168. ClojureWerkz • Modern targets (Clojure 1.5+, JDK 6) • Feature rich
  169. 169. ClojureWerkz • Modern targets (Clojure 1.5+, JDK 6) • Feature rich • Well documented
  170. 170. ClojureWerkz • Modern targets (Clojure 1.5+, JDK 6) • Feature rich • Well documented • Friendly license (EPL, the same as Clojure)
  171. 171. ClojureWerkz • Modern targets (Clojure 1.5+, JDK 6) • Feature rich • Well documented • Friendly license (EPL, the same as Clojure) • Tested against multiple Clojure versions, 3 JDKs on travis-ci.org
  172. 172. ClojureWerkz clojurewerkz.org
  173. 173. ClojureWerkz • Projects reuse a lot of Java libraries
  174. 174. ClojureWerkz • Projects reuse a lot of Java libraries • Which are often officially supported (e.g. RabbitMQ, ElasticSearch, MongoDB, Riak, Reactor)
  175. 175. “ClojureWerkz stuff is really improving my Clojure experience, which is rapidly becoming my language of choice…”
  176. 176. Clojure: the takeaway
  177. 177. Clojure: the takeaway • A lot of merit of its own
  178. 178. Clojure: the takeaway • A lot of merit of its own • Great stability
  179. 179. Clojure: the takeaway • A lot of merit of its own • Great stability • Hosted (symbiotic) language is a great thing
  180. 180. Clojure: the takeaway • A lot of merit of its own • Great stability • Hosted (symbiotic) language is a great thing • Don’t fear the JVM
  181. 181. Clojure: the takeaway • A lot of merit of its own • Great stability • Hosted (symbiotic) language is a great thing • Don’t fear the JVM • Immutability is essential for sane concurrent programming
  182. 182. Clojure: the takeaway • Clojure greatly rewards those who choose to use it
  183. 183. The real reason?
  184. 184. The real reason? ???
  185. 185. clojure.core/lazy-cat
  186. 186. Thank you • @michaelklishin • github.com/michaelklishin • clojurewerkz.org • michael@clojurewerkz.org

×