Video: http://youtu.be/a1r2bpGQbBQ A talk about using Riemann for stream processing. This is the metric aficionados on-premise tool of choice currently. Ali will talk about how they are using it to process the metrics coming out of their cloud service. For more info see : http://riemann.io Join DevOps Exchange London here: http://www.meetup.com/DevOps-Exchange-London/ Follow DOXLON on twitter: twitter.com/doxlon

1. Metrics Insight with Riemann
!
Ali Asad Lotia | ali.asad.lotia@gmail.com | @aalotia
!!

2. Clarification
!
• Riemann usage at Beamly is WIP

3. Platform visibility
• Dashboards
– Plots
– Summary values (min, max, mean)
• Alerts

4. Dashboard Limitations
• Dashboard overload
• Hard to be proactive
• Mostly reactive

5. Alerting challenges
• Service interdependencies
• Multiple services + multiple metrics
• Alerting before outage

6. If only…
• A tool to gain insight from the constant
stream of metrics
• Analyse incoming metrics in near real
time

7. Well, there is…
• http://riemann.io/
• Started by Aphyr (Kyle Kingsbury)
• Named for Bernhard Riemann
– http://en.wikipedia.org/wiki/Bernhard_Riemann

8. Riemann
http://riemann.io/concepts.html

9. Requirements
• Java 7 or greater
• Sufficient CPU and RAM
• Curiosity
• Patience

10. Concepts
• Events
• Streams
• Index
• Event manipulation (via functions)

11. Event Struct/Dictionary/Map
{:host "foo"!
:service "My test service"!
:state "good"!
:time 1414687710!
:description "A test that I am running"!
:tags ["foo", "bar"]!
:metric 1.0!
:ttl 900}

12. Stream
http://riemann.io/concepts.html

13. Configuration
• Written in clojure
– Pro: Power of a programming language
– Con: Power of a programming language
• Updatable on the fly

14. Beamly metrics flow
Sensu
Server
Server
Server
Server
Graphite
Riemann

15. Accepting graphite metrics
!
(let [host "0.0.0.0"!
port 2003]!
(graphite-server {:host host!
:port port})!
(tcp-server {:host host})!
(udp-server {:host host!
:max-size 65000})!
(repl-server {:host "127.0.0.1"})!
(ws-server {:host host}))!

16. Transforming graphite metrics !!
(defn process-graphite-metric-name!
"Split the graphite metric name and set the event map values!
accordingly. Currently a somewhat naive function since it assumes!
the format of the graphite metric key."!
[{:keys [service metric time]}]!
(let [[component source environment region aws-host metric-details]!
(cs/split service #"." 6)!
host-ip!
(extract-host-ip-from-aws-hostname aws-host)!
new-service!
(cs/join "." (vector component source metric-details))]!
{:service new-service!
:environment environment!
:region region!
:host host-ip!
:component component!
:metric (or metric 0)!
:time time!
:state "ok"!
:description service!
:tags ["from-sensu" component]}))!

17. Indexing select events
(let [index (default :ttl 900 (index))]!
(streams!
(where (= (:component event) "comp-a")!
#(info (format "received event from component %s" (:component %)) %)!
index)!
(where (= (:component event) "comp-b")!
(with :ttl 300 index!
#(info (format "received event from component %s" (:component %)) %)))!
(where (= (:component event) "comp-c")!
(with :ttl 300 index!
#(info (format "received event from component %s" (:component %)) %)))!
(where (and (not (nil? service)) (service #".sys.swapused$") (> metric 0))!
(with :state "swapping"!
#(info (format "swap used on component %s" (:component %)) %)!
index))))!

18. Output
• Email
• Pagerduty
• Graphite, InfluxDB
• custom plugin

19. Riemann Dashboard
http://riemann.io/dashboard.html
!
Can query riemann index

20. Gotchas
• Stateful
• Dashboard docs sparse
• Scaling possible by cascading to other
Riemann instances

21. Resources
• Website: http://riemann.io
• IRC: #riemann on freenode
• Riemann Users Google Group
https://groups.google.com/forum/#!forum/riemann-users
• Guardian riemann-config repo on Github
https://github.com/guardian/riemann-config

22. Thank you
• Questions?
• @aalotia
• ali.asad.lotia@gmail.com

23. LONDON
!
Drury House
34-43 Russell Street
London
WC2B 5HA
NEW YORK CITY
!
84 Wooster Street
Suite 703
New York
NY 10012
SYDNEY
!
22-36 Mountain Street
Suite 1.10
Ultimo
NSW 2007
© Beamly 2014. All Rights Reserved.