You can do cool and unexpected things if your entire type system is a first class citizen and accessible at runtime. With the introduction of spec, Clojure got its own distinct spin on a type system. Just as macros add another -time (runtime and compile time) where the full power of the language can be used, spec does to describing data. The result is an entire additional type system that is a first class citizen and accessible at runtime that facilitates validation, generative testing (a la QuickCheck), destructuring (pattern matching into deeply nested data), data macros (recursive transformations of data) and a pluginable error system. And then you can start building on top of it. The talk will be half introduction to spec and the ideas packed within it, and half experience report instrumenting 15k loc production codebase (primarily ETL and analytics) with spec.

1. clojure.spec:
a lisp-flavoured
type system
@sbelak
simon@goopti.com

2. Clojure at a glance
• (lisp (running-on :JVM))
• Functional, dynamic, immutable
• Excellent concurrency and state management
support
• Unparalleled data manipulation

4. Motivation
• Communication (docs are not enough)
• A lot of computation in Clojure is encoded with
(naked) data
• Generative (property based) testing
• Manual parsing and error reporting is tedious and
error prone

5. Enter clojure.spec

6. Writing a spec should enable automatic:
• Validation
• Error reporting
• Destructuring
• Instrumentation
• Test-data generation
• Generative test generation
*http://clojure.org/about/spec

7. Parsing with Derivatives
A Functional Pearl
Matthew Might David Darais
University of Utah
might@cs.utah.edu, david.darais@gmail.com
Daniel Spiewak
University of Wisconsin, Milwaukee
dspiewak@uwm.edu
Abstract
We present a functional approach to parsing unrestricted context-
free grammars based on Brzozowski’s derivative of regular expres-
sions. If we consider context-free grammars as recursive regular ex-
pressions, Brzozowski’s equational theory extends without modifi-
cation to context-free grammars (and it generalizes to parser combi-
nators). The supporting actors in this story are three concepts famil-
iar to functional programmers—laziness, memoization and fixed
points; these allow Brzozowski’s original equations to be translit-
erated into purely functional code in about 30 lines spread over
three functions.
Yet, this almost impossibly brief implementation has a draw-
back: its performance is sour—in both theory and practice. The
culprit? Each derivative can double the size of a grammar, and with
it, the cost of the next derivative.
Fortunately, much of the new structure inflicted by the derivative
is either dead on arrival, or it dies after the very next derivative.
To eliminate it, we once again exploit laziness and memoization
The derivative of regular expressions [1], if gently tempered
with laziness, memoization and fixed points, acts immediately
as a pure, functional technique for generating parse forests from
arbitrary context-free grammars. Despite—even because of—its
simplicity, the derivative transparently handles ambiguity, left-
recursion, right-recursion, ill-founded recursion or any combina-
tion thereof.
1.1 Outline
• After a review of formal languages, we introduce Brzozowski’s
derivative for regular languages. A brief implementation high-
lights its rugged elegance.
• As our implementation of the derivative engages context-free
languages, non-termination emerges as a problem.
• Three small, surgical modifications to the implementation (but
not the theory)—laziness, memoization and fixed points—
guarantee termination. Termination means the derivative can

8. Parsing with derivates
• Brzozowski derivative u−1S = { v ∈ Σ*: uv ∈ S }
the set of all rest-strings obtainable from a string in S by cutting off its prefix u
• code along:
blog.klipse.tech/clojure/2016/10/02/parsing-with-derivatives-regular.html

9. Regular expressions (for data)
+
arbitrary predicates
+
data transformations (conformers)
clojure.spec

11. Two schools of
thinking

12. System paradigmLanguage paradigm
infoq.com/presentations/Mixin-based-Inheritance
realworldclojure.com/the-system-paradigm
[Language:] a formal system of signs governed by
grammatical rules of combination to communicate
meaning
[System:] a set of interacting or interdependent
components forming an integrated whole

13. The system paradigm
1. Nibble at the problem from different directions
2. Compose partial solutions into the final solution
In lisp you build systems by altering the living, running one right in front of you
— R. Gabriel
Pascal is for building pyramids—imposing, breathtaking, static structures built
by armies pushing heavy blocks into place. Lisp is for building organisms.
— A. Perlis

14. The Artificial Intelligence people were endeavouring to
write programs that no one knew how to write. The idea
that you could sit down and say: 'Well, here is my
problem, here are the requirements, let me come up with
a specification and now code that up' (was) completely
crazy as far as the AI people were concerned' The only
way to write AI programs, then (as it still is now) was by
taking an exploratory approach to development. The
only way to do it was to experiment. 'Let me try this, let
me add that. Let me try to add this fuzzy concept, let me
try to add a scheduler, let me add agendas, let me add
resources, let me have resource-limitations' ... you're not
constructing it like making a ton of source code and
compiling it periodically, you're constructing it the way
you construct a city: build some of it, it's running all the
time, so it's kind of like a live programming language.
— R. Gabriel

15. Live programming

16. No clear delineation
between environment
and work code

17. spec: a fully interactive
à la carte type system?

18. “Composition is about
decomposing.”
— E. Normand
“Good design is not about making
grand plans, but about taking
things apart.”
— R. Hickey

19. Destructuring
• Pull apart and name
• Factor out data shape

20. Data macros
• Recursive transformations into canonical form
• Do more without code macros
* juxt.pro/blog/posts/data-macros.html

23. Decomplect validation
and encoding

24. Generative testing
• Limitations
• sequences with internal structure (time series
etc.)
• generic higher-order functions (e.g. map)
• Uncovers numerical instabilities
• Better mocking in the REPL

25. TDD
vs.
REPL-driven development
vs.
spec-driven development?

26. Everything is data
(aka. the lisp way)
• generation
• introspection
• parsable errors

27. Building on top of
spec

28. Queryable data
descriptions
Turn spec into a graph
order
promo code
user
account age
country
always always
alwaysmaybe

29. Case study: autogenerating materialised views
Kafka
Materialised
views
Events
External data
Automatic view generation
• Event & attribute ontology
• Manual (via spec)
• Inferred
• Statistical analysis (seasonality
detection, outlier removal, …)
Onyx Onyx
Onyx

30. Automatic view generation
1. Walk spec registry
2. Apply rules
1. Define new view (spec)
2. Trigger Onyx job that creates the view
⤾

31. Missing pieces
github.com/arohner/spectrum — static analysis of
(spec annotated) Clojure code
github.com/stathissideris/spec-provider — infer spec
from examples
github.com/typedclojure/auto-annotation — infer
spec from tests

32. Takeouts

33. Decomplect
everything

34. Everything should be
live and interactive

35. Blurring the line between
environment and work is
a powerful idea

36. Queryable data
descriptions supercharge
interactive development
and are a great building
block for automation

37. Questions
@sbelak
simon@goopti.com

38. clojure.org/about/spec
matt.might.net/papers/might2011derivatives.pdf
infoq.com/presentations/Mixin-based-Inheritance
realworldclojure.com/the-system-paradigm
juxt.pro/blog/posts/data-macros.html
blog.klipse.tech/clojure/2016/10/02/parsing-with-derivatives-regular.html
indiegogo.com/projects/typed-clojure-clojure-spec-auto-annotations#/
github.com/arohner/spectrum
github.com/stathissideris/spec-provider
purelyfunctional.tv/issues/clojure-gazette-192-composition-is-about-decomposing