Clojure has always been good at manipulating data. With the release of spec and Onyx (“a masterless, cloud scale, fault tolerant, high performance distributed computation system”) good became best. In this talk you will learn about a data layer architecture build around Kafka and Onyx that is self-describing, declarative, scalable and convenient to work with for the end user. The focus will be on the power and elegance of describing data and computation with data; and the inferences and automations that can be built on top of that.

1. Spec + Onyx:
an experience report
@sbelak
simon@goopti.com

2. Onyxa masterless, cloud scale, fault tolerant, high
performance distributed computation system
… written entirely in Clojure

3. Onyx at
• In production for almost a year
• ETL
• online machine learning
• offline (batch) machine learning
• ad-hoc analysis

4. Self-service infrastructure
for data scientists

5. 1.Onyx at a glance
2.How Onyx rewired my brain
3.Building on top of spec

6. Onyx at a glance

7. Job =
[[:input :processing-1]
[:input :processing-2]
[:processing-1 :output-1]
[:processing-2 :output-2]]
[{:flow/from :input-stream
:flow/to [:process-adults]
:flow/predicate :my.ns/adult?
:flow/doc "Emits segment if an adult.”}]
workflow
+ flow conditions
+ catalogue[{:onyx/name :add-5
:onyx/fn :my/adder
:onyx/type :function
:my/n 5
:onyx/params [:my/n]
:onyx/batch-size batch-size}
{:onyx/name :in
:onyx/plugin :onyx.plugin.core-async/input
:onyx/type :input
:onyx/medium :core.async
:onyx/batch-size batch-size
:onyx/max-peers 1
:onyx/doc "Reads segments from a core.async channel"}
{:onyx/name :out
:onyx/plugin :onyx.plugin.core-async/output
:onyx/type :output
:onyx/medium :core.async
:onyx/batch-size batch-size
:onyx/max-peers 1
:onyx/doc "Writes segments to a core.async channel"}]

8. Catalogue
[{:onyx/name :add-5
:onyx/fn :my/adder
:onyx/type :function
:my/n 5
:onyx/params [:my/n]
:onyx/batch-size batch-size}
{:onyx/name :in
:onyx/plugin :onyx.plugin.core-async/input
:onyx/type :input
:onyx/medium :core.async
:onyx/batch-size batch-size
:onyx/max-peers 1
:onyx/doc "Reads segments from a core.async channel"}
{:onyx/name :out
:onyx/plugin :onyx.plugin.core-async/output
:onyx/type :output
:onyx/medium :core.async
:onyx/batch-size batch-size
:onyx/max-peers 1
:onyx/doc "Writes segments to a core.async channel"}]
Vanilla Clojure function
(defn adder [n {:keys [x] :as segment}]
(assoc segment :x (+ n x))))
Plugins (I/O)
seq, async, Kafka,
Datomic, SQL,…
parameter
self-documenting

9. Computation entirely
described with data
data
is
code!

10. Everything can be run
locally!

11. Testing without
mocking

12. How Onyx rewired my
brain

13. It’s not about scaling,
but clean architecture

14. My goto architecture
KafkaDB Events
Onyx Onyx
Onyx
Persist all
messages to S3
(time travel!)

15. Decomplect
everything

16. Computation graphs

17. Building on top of
spec

18. Queryable data descriptions
• s/registry, s/form
• Build a graph (Datomic)
Interact with your type system!
code
is
data!

19. Case study: autogenerating materialised views
Kafka
Materialised
views
Events
External data
Automatic view generation
• Event & attribute ontology
• Manual (via spec)
• Inferred
• Statistical analysis (seasonality
detection, outlier removal, …)
Onyx Onyx
Onyx

20. Automatic view generation
1. Walk spec registry
2. Apply rules
1. Define new view (spec)
2. Trigger Onyx job that creates the view
⤾

21. Code is data
or
data is code?

22. Takeouts

23. Onyx
is
production
ready

24. Everything should be
live and interactive

25. Computation graphs are
a great way to structure
data processing code

26. Queryable data and
computation descriptions
supercharge interactive
development and are a
great building block for
automation

27. Questions
@sbelak
simon@goopti.com

28. viebel.github.io/klipse/examples/onyx.html
onyxplatform.org
onyxplatform.org/jekyll/update/2017/02/08/Pyroclast-
Preview-Simulation.html