Clojure has always been good at manipulating data. With the release of spec and Onyx (“a masterless, cloud scale, fault tolerant, high performance distributed computation system”) good became best. In this talk you will learn about a streaming data layer architecture build around Kafka and Onyx that is self-describing, declarative, scalable and convenient to work with for the end user. The focus will be on the power and elegance of describing data and computation with data; the inferences and automations that can be built on top of that; and how and why Clojure is a natural choice for tasks that involve a lot of data manipulation, touching both on functional programming and lisp-specifics such as code-is-data. We will look at how such an approach can be used to manage a data warehouse by automatically inferring materialized views from raw incoming data or other views based on a combination of heuristics, statistical analysis (seasonality, outlier removal, ...) and predefined ontologies. Doing so is a practical way to maintain a large number of views, increasing their availability and abstracting the complexity into declarative rules, rather than having an ETL pipeline with dozens or even hundreds of hand crafted tasks. The system described requires relatively little effort upfront but can easily grow with one's needs both in terms of scale as well as scope. With its good introspection capabilities and strong decoupling it is for instance an excellent substrate for putting machine learning algorithms in production, which is the final use-case we will dive into.

1. A data layer in
Clojure
@sbelak
simon@goopti.com

2. • Started in machine learning
• Turned to data science and
helped 20+ companies become
data-driven
• Now leading data science
department at GoOpti

3. Self-service infrastructure
for data scientists

4. The analytics chasm
Ideal. Almost real-time, can
be done during brainstorming
without disrupting flow
< 2min < 20min project
squeeze in
somewhere
in the day
fail
roadmap
ahoy!

5. My goto architecture
KafkaDB Events
Onyx Onyx
Onyx
Persist all events to S3
• time travel
• query with AWS Athena

6. Onyxa masterless, cloud scale, fault tolerant, high
performance distributed computation system
… written entirely in Clojure

7. Clojure at a glance
• Lisp running on JVM
• Functional, dynamic, immutable
• Excellent concurrency and state management
support
• Unparalleled data manipulation
• Good Java interoperability

8. Onyx at
• In production for almost a year
• ETL
• online machine learning
• offline (batch) machine learning
• ad-hoc analysis

9. Onyx at a glance

10. Job =
[[:input :processing-1]
[:input :processing-2]
[:processing-1 :output-1]
[:processing-2 :output-2]]
[{:flow/from :input-stream
:flow/to [:process-adults]
:flow/predicate :my.ns/adult?
:flow/doc "Emits segment if an adult.”}]
workflow
+ flow conditions
+ catalogue[{:onyx/name :add-5
:onyx/fn :my/adder
:onyx/type :function
:my/n 5
:onyx/params [:my/n]}
{:onyx/name :in
:onyx/plugin :onyx.plugin.core-async/input
:onyx/type :input
:onyx/medium :core.async
:onyx/batch-size batch-size
:onyx/max-peers 1
:onyx/doc "Reads segments from a core.async channel"}
{:onyx/name :out
:onyx/plugin :onyx.plugin.core-async/output
:onyx/type :output
:onyx/medium :core.async
:onyx/doc "Writes segments to a core.async channel"}]

11. Catalogue
[{:onyx/name :add-5
:onyx/fn :my/adder
:onyx/type :function
:my/n 5
:onyx/params [:my/n]}
{:onyx/name :in
:onyx/plugin :onyx.plugin.core-async/input
:onyx/type :input
:onyx/medium :core.async
:onyx/batch-size batch-size
:onyx/max-peers 1
:onyx/doc "Reads segments from a core.async channel"}
{:onyx/name :out
:onyx/plugin :onyx.plugin.core-async/output
:onyx/type :output
:onyx/medium :core.async
:onyx/doc "Writes segments to a core.async channel"}]
Vanilla Clojure function
(defn adder [n {:keys [x] :as segment}]
(assoc segment :x (+ n x))))
Plugins (I/O)
seq, async, Kafka,
Datomic, SQL, S3,
SQS, …
parameter
self-documenting

12. Computation entirely
described with data
data
is
code!

13. Everything can be run
locally!

14. Testing without
mocking

15. Resilience and handling
state
• Activity log
• Window and trigger states checkpointed
• Resume points
• Configurable flux policies

16. How Onyx rewired my
brain

17. It’s not about scaling,
but clean architecture

18. Decomplect
everything

19. Computation graphs

20. Machine learning with Onyx
• Hyperparameter server build on top of Onyx
parameters
• Batch & streaming mode
• Monitoring: performance metrics, side channels for
partial results/introspection into computiation
• Everything is data so easy to build tools around

21. Onyx/Pyroclast

22. Putting “data is code”
to work

23. Describing data with
clojure.spec
composing smaller
parts into the whole}
code
is
data!

24. Queryable data
descriptions
Turn spec into a graph
A fully interactive and open type system!
order
promo code
user
account age
country
always always
alwaysmaybe

25. “Composition is about
decomposing.”
— E. Normand

26. Case study: autogenerating materialised views
Kafka
Materialised
views
Events
External data
Automatic view generation
• Event & attribute ontology
• Manual (via spec)
• Inferred
• Statistical analysis (seasonality
detection, outlier removal, …)
Onyx Onyx
Onyx

27. Automatic view generation
1. Walk spec registry
2. Apply rules
1. Define new view (spec)
2. Trigger Onyx job that creates the view
⤾

28. Takeouts

29. Everything should be
live and interactive

30. Computation graphs are
a great way to structure
data processing code

31. Queryable data and
computation descriptions
supercharge interactive
development and are a
great building block for
automation

32. Questions
@sbelak
simon@goopti.com

33. viebel.github.io/klipse/examples/onyx.html
onyxplatform.org
onyxplatform.org/jekyll/update/2017/02/08/Pyroclast-
Preview-Simulation.html