@sbelak Simon Belak Using Onyx in anger Clojure has always been good at manipulating data. With the release of spec and Onyx ("masterless, cloud scale, fault tolerant, high performance distributed computation system") good became best. In this talk I will walk you through a data layer architecture build around Kafka an Onyx that is self-describing, declarative, scalable and convenient to work with for the end user. The focus will be on the power and elegance of describing data and computation with data; and the inferences and automations that can be built on top of that.

1. Using Onyx in
anger
@sbelak
simon@goopti.com

2. Onyxa masterless, cloud scale, fault tolerant, high
performance distributed computation system
… written entirely in Clojure

3. Onyx at
• In production for almost a year
• ETL
• online machine learning
• offline (batch) machine learning
• ad-hoc analysis

4. Self-service infrastructure
for data scientists

5. 1.Onyx at a glance
2.How Onyx rewired my brain
3.Putting “data is code” to work

6. 1.Onyx at a glance
2.How Onyx rewired my brain
3.Putting “data is code” to work
Describing computation
with data

7. Onyx at a glance

8. Job =
[[:input :processing-1]
[:input :processing-2]
[:processing-1 :output-1]
[:processing-2 :output-2]]
[{:flow/from :input-stream
:flow/to [:process-adults]
:flow/predicate :my.ns/adult?
:flow/doc "Emits segment if an adult.”}]
workflow
+ flow conditions
+ catalogue[{:onyx/name :add-5
:onyx/fn :my/adder
:onyx/type :function
:my/n 5
:onyx/params [:my/n]}
{:onyx/name :in
:onyx/plugin :onyx.plugin.core-async/input
:onyx/type :input
:onyx/medium :core.async
:onyx/batch-size batch-size
:onyx/max-peers 1
:onyx/doc "Reads segments from a core.async channel"}
{:onyx/name :out
:onyx/plugin :onyx.plugin.core-async/output
:onyx/type :output
:onyx/medium :core.async
:onyx/doc "Writes segments to a core.async channel"}]

9. Catalogue
[{:onyx/name :add-5
:onyx/fn :my/adder
:onyx/type :function
:my/n 5
:onyx/params [:my/n]}
{:onyx/name :in
:onyx/plugin :onyx.plugin.core-async/input
:onyx/type :input
:onyx/medium :core.async
:onyx/batch-size batch-size
:onyx/max-peers 1
:onyx/doc "Reads segments from a core.async channel"}
{:onyx/name :out
:onyx/plugin :onyx.plugin.core-async/output
:onyx/type :output
:onyx/medium :core.async
:onyx/doc "Writes segments to a core.async channel"}]
Vanilla Clojure function
(defn adder [n {:keys [x] :as segment}]
(assoc segment :x (+ n x))))
Plugins (I/O)
seq, async, Kafka,
Datomic, SQL, S3,
SQS, …
parameter
self-documenting

10. Computation entirely
described with data
data
is
code!

11. Everything can be run
locally!

12. Testing without
mocking

13. How Onyx rewired my
brain

14. It’s not about scaling,
but clean architecture

15. My goto architecture
KafkaDB Events
Onyx Onyx
Onyx
Persist all events to S3
• time travel
• query with AWS Athena

16. Decomplect
everything

17. Computation graphs

18. Putting “data is code”
to work

19. Interlude:
queryable data descriptions
with spec
• s/registry, s/form
• Build a graph (Datomic)
Interact with your type system!
code
is
data!

20. Case study: autogenerating materialised views
Kafka
Materialised
views
Events
External data
Automatic view generation
• Event & attribute ontology
• Manual (via spec)
• Inferred
• Statistical analysis (seasonality
detection, outlier removal, …)
Onyx Onyx
Onyx

21. Automatic view generation
1. Walk spec registry
2. Apply rules
1. Define new view (spec)
2. Trigger Onyx job that creates the view
⤾

22. Code is data
or
data is code?

23. Takeouts

24. Onyx
is
production
ready

25. Everything should be
live and interactive

26. Computation graphs are
a great way to structure
data processing code

27. Queryable data and
computation descriptions
supercharge interactive
development and are a
great building block for
automation

28. Questions
@sbelak
simon@goopti.com

29. viebel.github.io/klipse/examples/onyx.html
onyxplatform.org
onyxplatform.org/jekyll/update/2017/02/08/Pyroclast-
Preview-Simulation.html