This document describes how a company switched from monitoring their large infrastructure using a Python-based system to using Clojure. The new Clojure-based system loads monitoring data from XML into RabbitMQ, stores the time-series data in JRobin, processes the data to perform checks, computations, and aggregations, and allows querying the historical data using a custom DSL. The Clojure system leverages many Java libraries, allows full utilization of processors via agents, and enables automated corrective actions and making the application smarter using real-time monitoring data.

1. Monitoring a Large-Scale Infrastructure with Clojure

2. Who am I?
Dennis Rowe
Senior Software Developer
Dell MessageOne - DevOps
2 Oracle OpenWorld 2011

3. MessageOne
E-Mail Continuity
E-Mail Archive
E-Mail Search
3 Oracle OpenWorld 2011

4. The Basics
2646 Servers
3 Countries
3 Billion E-Mail
5 Million Users
12 Tired People
4 Oracle OpenWorld 2011

5. We got to have a
way to monitor all
that stuff…
Maybe not the people
5 Oracle OpenWorld 2011

6. So, we came up with a solution…
6 Oracle OpenWorld 2011

7. “Kneel before Zod”
-- General Zod
7 Oracle OpenWorld 2011

8. A Bit of History
Initially written in Python
Utilized Twisted framework
Historical Data stored in relational database
It worked, but it did not perform
8 Oracle OpenWorld 2011

9. Why?
Global Interpreter Lock (GIL)
caused performance problems
Relational Database not efficient
for time-series data
9 Oracle OpenWorld 2011

10. So…
Why switch to Clojure?
It is hip
It was designed with multi-threading in mind
It is a functional language
It uses the JVM
We can use all the Java libraries lying around
Homoiconic
10 Oracle OpenWorld 2011

11. “And there was much rejoicing”
-- Monty Python and the Holy Grail
11 Oracle OpenWorld 2011

12. So, this is how we did it
12 Oracle OpenWorld 2011

13. Loader
Takes XML and dumps it on a
Message Bus (RabbitMQ)
Nothing much to see here but…
13 Oracle OpenWorld 2011

14. Data is Code
So, how do we store the configurations we want for the various datacenters?
As code … data … code …
[“dc1” “url1” “type1”
“dc2” “url2” “type2”]
The configs are just Clojure code and they make sense
14 Oracle OpenWorld 2011

15. RabbitMQ
That is easy
We will just use the RabbitMQ Java API
We will create Clojure centric data structures
This whole Java interoperability is kind of nice …
things just kind of work
15 Oracle OpenWorld 2011

16. Also!
If code is data … then we can just send the code over
RabbitMQ
16 Oracle OpenWorld 2011

17. Wait – What?
We don’t need any funky configurations?
We don’t need to use XML?
We don’t need to use JSON?
If it is Clojure talking to Clojure we can just use data (or is it
code, I am confused)
17 Oracle OpenWorld 2011

18. Persister
Takes the data off the bus and writes it to disk
The Java ecosystem has tools for that, too
Jrobin
We now have our own little timeseries database and we didn’t
really have to work for it.
18 Oracle OpenWorld 2011

19. Consumer
Takes metrics and does stuff with them
Checks
Computes
Aggregates
Historical Aggregates
19 Oracle OpenWorld 2011

20. Examples
Check
(check “mta-delay” :degraded
(above (* 3600 72)) :fmt “%,.1f secs”)
Compute
(compute “mem-swap-used”
:using [swap_total swap_free]
:as (- swap_total swap_free))
20 Oracle OpenWorld 2011

21. Aggregate
(aggregate “cfg-anomalies”)
Historical Aggregate
(hist-aggregate “index-percent-failed” “index-percent-
failed#hist-1h” 3600 :agg-fn avg)
21 Oracle OpenWorld 2011

22. Threading
All those metrics are Clojure Agents, so I don’t
have to worry about it
All 16 of my processors get used
Life is easy
22 Oracle OpenWorld 2011

23. Look
23 Oracle OpenWorld 2011

24. WWW
We are not web developer types, which is fine,
Clojure (plus some libraries) makes that easy, too
Compojure
Hiccup
So, no HTML. Just code
[:a {:href “/”}]
24 Oracle OpenWorld 2011

25. Query
We need a way to query the data in real time
Clojure is homiconic
So…
We will just create our own DSL
25 Oracle OpenWorld 2011

26. The DSL
It is just code
We can use existing Clojure functions plus new ones like:
where
select
pivot
filter
sort
format
sum-by and agg-by
26 Oracle OpenWorld 2011

27. Query Example
where :metric [“qsize” “qsize-2h-old” “rate”] |
pivot |
filter (> :qsize 50000) |
select :host
:qsize
[(* 100 (- 1 (/ :qsize-2h-old :qsize))) :pct-recent]
:rate |
sort :pct-recent
27 Oracle OpenWorld 2011

28. Explanation
Looks a lot like Linux pipes
Which is a good way to think about it
Clojure way of reading it is:
(sort (select (filter (pivot (where)))))
28 Oracle OpenWorld 2011

29. Output
29 Oracle OpenWorld 2011

30. DevOps
What we needed (and what we got)
Reports
Ad-hoc Queries
Corrective actions?
Make the app smarter?
30 Oracle OpenWorld 2011

31. Corrective Actions
Write little python scripts that
pull data and take actions
This was so easy that we had to do it
Simple, repetitive actions are now fully automated
Life is better
31 Oracle OpenWorld 2011

32. App Smarter
App now uses the monitoring to feed intelligently
Less operator interaction needed
More time spent solving real problems
32 Oracle OpenWorld 2011

33. Q and A
33