Presented in SoCal Code Camp 2016 on DevOps topic. This is the story of how our team fought (the good fight against) live issues and how we evolved over time and got better. We evolved our operational toolset from Graphite to Riemann to NewRelic tools, to help us react-to and resolve live issues quickly. We progressed from knowing 'what just broke' to 'whats about to break' to 'whats about to break and why'. More important than the technology & toolset, we focused on building a good team culture and tackled this as a team.

1. Java DevOps
Fighting the Good fight
2016 SoCal Code Camp

2. Urban Dictionary

3. Werner Vogels from Amazon
“The traditional model is that you take your
software to the wall that separates development
and operations, and throw it over and then
forget about it.
Not at Amazon. You build it, you run it.”

4. Dev Ops
You need to be able to deliver software at speed
– because your customers want it
You need to be able to react to issues at speed
– because your customers will

5. The focus of this talk is on
You need to be able to deliver software at speed
– because your customers want it
You need to be able to react to issues at speed
– because your customers will

6. A DevOps Story: Team vs Live Issue
• We have never lost a fight
• How did we go down so fast? (Metrics & Graphite)
• The punches are coming left, right & fast (Riemann)
• These new gloves are so powerful (New Relic)
• Let's fight this as a team (Culture & Pager Duty)

7. • We have never lost a fight
• How did we go down so fast? (Metrics & Graphite)
• The punches are coming left, right & fast (Riemann)
• These new gloves are so powerful (New Relic)
• Let's fight this as a team (Culture & Pager Duty)
A DevOps Story: Team vs Live Issue

8. Can you find points of failure?

9. Fallacies of Distributed Computing
• The network is reliable
• Latency is zero
• Bandwidth is infinite
• The network is secure
• Topology doesn't change
• There is one administrator
• Transport cost is zero
• The network is homogeneous

10. Shout Bingo !
X
X
X X
X
X X
X X
XX
X X X X

11. A DevOps Story: Team vs Live Issue
• We have never lost a fight
• How did we go down so fast? (Metrics & Graphite)
• The punches are coming left, right & fast (Riemann)
• These new gloves are so powerful (New Relic)
• Let's fight this as a team (Culture & Pager Duty)

12. Yammer Metrics
Java library that provides ways to measure behavior of critical
components in production.
Highly recommend watching Coda Hale’s presentation. Just google
“Metrics Metrics Everywhere” – there’s a deck and video.

13. Yammer Metrics
Gauges The instantaneous value of something
Ex: # of cities; The service has 589 cities registered.
Counters An incrementing and decrementing value
Ex: # of open connections; There are 594 active sessions.
Meters The average rate of events over a period of time
Ex: # of requests/sec; rate of requests for 1m, 5m, 15m period.
Histograms The statistical distribution of values in a stream of data
Ex: # of cities returned in autocomplete; min, max, mean, median, 75th, 95th, 99th
Timers A histogram of durations and a meter of calls
Ex: # of ms to respond; At ~2,000 req/sec, our 99% latency is 500ms.

14. Timer Metric Publish Code Example

15. Graphite
Stores numeric time-series data
<metric name> <metric value> <metric time>
Render graphs of this data on-demand
Name Value Time
service.active.thread.count 20 10:00 AM
service.active.thread.count 25 10:01 AM
service.active.thread.count 30 10:02 AM
service.active.thread.count 120 10:03 AM
service.active.thread.count 60 10:04 AM
service.active.thread.count 40 10:05 AM
20
25
30
120
60
40
0
20
40
60
80
100
120
140
10:00 10:01 10:02 10:03 10:04 10:05
ACTIVE THREAD COUNTS

16. Data published to Graphite couple of ways
Graphite
Apps
Host
Agents

17. Demo 1
Rate and duration of Deal loads
apps.dms-service.la1crm-dmsbatch01.p9775.consumed.dms_item_fi_load.rate.1m
apps.dms-service.la1crm-dmsbatch01.p9775.consumed.dms_item_fi_load.duration.p95

18. Demo 2
A stacked graph of thread states in all the servers in services layer.
groupByNode(crm.crm-services*.dealer.ddc.application.activeThreadStates.*.value, 6,'sumSeries')
Dashboard link

19. We built lots of dashboards
• Monitoring Dashboard
• CRM Requests
• CRM Services GC
• CRM Vertica

20. When customers experienced an issue,
we could tell what just happened that caused the issue.
But, we couldn’t tell when things are about to break.

21. A DevOps Story: Team vs Live Issue
• We have never lost a fight
• How did we go down so fast? (Metrics & Graphite)
• The punches are coming left, right & fast (Riemann)
• These new gloves are so powerful (New Relic)
• Let's fight this as a team (Culture & Pager Duty)

22. Riemann
Riemann is an Event stream processor.
Every time something important happens
in your system, submit an event to
Riemann.
Riemann filters, combines, and acts on
flows of events to understand your
systems.

23. Riemann - working with streams

24. Riemann - working with streams

25. Riemann - working with streams

26. Riemann - working with streams

27. Data published to Riemann couple of ways
Riemann
Apps
Riemann
Agent App

28. Rollup Email Notification

29. Dashboard - Services Response Times

30. Dashboard - State of Queues

31. Before customers experienced an issue,
we could tell what was about to break,
but we couldn’t figure out quickly what was causing it to break.
We could revert it in some cases or inform the customer before hand.

32. A DevOps Story: Team vs Live Issue
• We have never lost a fight
• How did we go down so fast? (Metrics & Graphite)
• The punches are coming left, right & fast (Riemann)
• These new gloves are so powerful (New Relic)
• Let's fight this as a team (Culture & Pager Duty)

33. New Relic APM

34. New Relic Browser

35. New Relic Server

36. Live Issue Examples
- The network oopsie
- NFS Mounts got slow

37. Publish Data with Context
def your_application_filter() {
…
if (account) {
NewRelic.addCustomParameter("account", account)
}
if (username) {
NewRelic.addCustomParameter("username", username)
}
…
}

38. New Relic Insights

39. Before customers experienced an issue,
we could tell what was about to break,
In most cases we could also figure out what was causing it to break.
We could revert it in some cases or inform the customer before hand.

40. A DevOps Story: Team vs Live Issue
• We have never lost a fight
• How did we go down so fast? (Metrics & Graphite)
• The punches are coming left, right & fast (Riemann)
• These new gloves are so powerful (New Relic)
• Let's fight this as a team (Culture & Pager Duty)

41. Think about failures…
• Build fault tolerant solutions
• Isolate points of failures
• Prevent cascading failures
• Build resilient solutions

42. Pager Duty
Icinga

43. On Call Schedule

44. Escalation Policy

45. Team Culture
• Resist the “Pulp Fiction - Send in the Wolf” mentality
• Get the whole team involved
• Build alert & solution documentation
• Provide training and be there to help
• Have an on-call rotation and escalation policy
• Force yourself to explain every Live Issue

46. In Conclusion
Speed and accuracy is a hard game to play,
but that’s the game.
Wish you happy live issues and hope you
fight the good fight well !