Effective DevOps Practices for collecting, correlating and analyzing DevOps data. First presented by Splunk's Andi Mann at DevOps Summit at Cloud Expo.

1. Copyright © 2016 Splunk Inc.
Listen to Your Machines:
DevOps Analytics for
Better Feedback Loops
Andi Mann, Chief Technology Advocate
Splunk
@andimann

2. ‘Known good
practices’ for
collecting, correlating,
and analyzing DevOps
data
3

3. Effective DevOps Practices
● Improve collaboration and sharing between dev and ops
● Build trust and accountability between teams
● Break down barriers and improve handoffs between silos
● Establish trust and transparency between Dev. and Ops.
● Streamline flow of code from idea to cash
● Creating feedback loops at every stage
● Focus on impact on business goals and customer experience
4

4. FROM EVERY TOOL, EVERY PROCESS, EVERY COMPONENT, ON-PREM OR OFF
One Constant -
Machine Data
5

5. Common Data Fabric
6
API
SDKs UI
Server, Storage.
N/W
Server
Virtualization
Operating
Systems
Infrastructure
Applications
Mobile
Applications
Cloud Services
Other Tools
Ticketing/Help
Desk
Custom
Applications
Visibility Across the Whole Ops Environment
API Services

6. Common Data Fabric
7
API
SDKs UI
Other Tools
Escalation/
Collaboration
Visibility Across the Whole Dev Lifecycle
Plan Code Build Test/QA Stage Release Config Monitor

7. Important DevOps
data and metrics for
different DevOps
teams
8

8. Computing UK’s ‘Metrics that Matter’
Source: Computing Research UK, DevOps Review 2016: Accelerating Innovation, July 2016
9

9. More DevOps Metrics that Might Matter
10
Culture
e.g.
• Retention
• Work hours
• Callouts
Process
e.g.
• Idea-to-cash
• MTTR
• Deliver time
Quality
e.g.
• Tests passed
• Tests failed
• Best/worst
Systems
e.g.
• Throughput
• Uptime
• Build times
Activity
e.g.
• Commits
• Tests run
• Releases
Impact
e.g.
• Signups
• Checkouts
• Revenue

10. Specific Metrics For Each Stakeholder
11
BizOpsStageBuildSecQADevPMOBiz
• time to deliver
• idea to cash
• ROI
• process times
• team efficiency
• unplanned work
• code volume
• commit volume
• release speed
• test volume
• code coverage
• exception counts
• access attempts
• remediation time
• code quality
• build speed
• failure rates
• manual exceptions
• performance
• latency
• scalability
• response time
• uptime/availability
• resource usage
• revenue
• signups
• satisfaction

11. Shared Metrics for Multiple Stakeholders
12
BizOpsStageBuildSecQADevPMOBiz
• time to deliver
• scalability
• ROI
• time to deliver
• team efficiency
• ROI
• team efficiency
• scalability
• release speed
• remediation time
• code quality
• performance
• remediation time
• code quality
• manual exceptions
• scalability
• performance
• manual exceptions
• release speed
• code quality
• scalability
• remediation time
• performance
• scalability
• performance
• release speed
• ROI

12. What About Just Dev and Ops
Velocity
time to deploy
time to build
‘idea to customer’
story throughout
build failure rate
Quality
number of defects
downtime per release
code coverage
response time impacts
build failures
Business impact
customer satisfaction
application usage
user signup/cancel
transaction failures
sales volumes
Human impact
employee satisfaction
team productivity
staff retention
work hours
‘work from home’ days
Source: Computing Research UK, DevOps Review 2016: Accelerating Innovation, July 2016
13

13. How different orgs
and teams benefit
from using DevOps
data in multiple
feedback loops
14

14. State of DevOps 2016: ‘Metrics that Matter’
15
Source: 2016 State of DevOps Report, DevOps Research and Assessment

15. Objective data enables continuous improvement
Defect
Information
Capacity
Planning
Quality
Standards
Enhancement
Requests
Integration
Requirements
Acceptance
Metrics
Service Levels
and KPIs
Application Development Test and Acceptance Production
BuildCodePlan Test/QA Stage Release Config Monitor
Infrastructure
Dependencies
16

16. Shared Data Helps Find and Fix Issues Faster
Real-time dashboards show error rate
in production and impact of pushing
new builds
17
Developers can search and visualize
web logs, Java logs—without
production access
Alerts notify developers as soon as a
problem arises
17

17. Shared Data Increases App Delivery Velocity
18
DevOps Teams iterate with
continuous insights
Product Managers
identify new
opportunities
Code continuously delivered
to market
Auditors
have visibility
Customers
are happy

18. Shared Data Improves Code Quality
19
Code Quality Scans Static Security Scans
White BoxDevelopers
check in code
Automated
Acceptance Tests
Dynamic Security
Scans
Black Box
“Chaos Monkey”
Tests
Test Fail:
Return
Test Fail:
Return
Production
QA Prod Pattern
QA Pattern Library
Test Pass:
Promote
Test Pass:
Promote to
Production
Pattern
library used
for test and
QA

19. Shared Data Aligns DevOps With Business Impact
20

20. Some Real-World ‘Metrics That Matter’
21
“Developers can focus on innovation and not on
building monitoring tools.”
“Web Ops can measure performance of
releases in pre-prod, prod and in QA.”
“Gather all data, and it starts looking like one big system,
instead of a bazillion teeny ones that hate each other.”
“We measure customer sentiment on Google
Play in real time and can correlate it with code
releases and app performance.”

21. Enable Improved DevOps Agility
22
Key Customer Benefits
-Robert Gonsalves,
Web Operations
“It’s like we were working
without peripheral vision
before and now we have it.”
• Increased success rate of deployments
• Ability to detect issues before they affect broad
production
• Monitoring deployment process several times per day

22. Deliver Better Code Quality
23
Key Customer Benefits
-Principal Engineer,
Apollo Group
“Developers are now able to look
for errors and troubleshoot issues
five to 10 times faster by having all
their event data centralized in
Splunk.”
• Provide full visibility into QA sanity and load testing
before production
• Exceed SLA thresholds with full visibility and benchmark
key infrastructure metrics and errors
• Easily troubleshoot if tests do not contain the expected
results

23. Enable Data-Driven Continuous Delivery
24
-Alison Perkins,
Senior Systems Engineer
“Dump all the logs into Splunk,
and it starts looking like one big
system, instead of a bazillion
teeny ones that hate each
other.”
Key Customer Benefits
• Quickly validate and troubleshoot code pushes to
production
• Ensure that new code does not negatively impact
performance or user experience
• Reduced one application’s error rate by 2 orders of
magnitude in a matter of weeks

24. Enable Closer Business-IT Alignment
Allows DevOps to ensure quality of releases &
avoid negative impact on service performance.
Analyze which new website features are being
adopted, and how, by end users.
Insight fed back into the development
cycle to improve customer engagement.
25

25. Data-oriented service
delivery visualization
to help leaders see
value in opaque
DevOps processes
26

26. Data From Dev and Ops Tools
27

27. Data From Provisioning and Config
28

28. Data from Release Servers
29

29. Data from Infrastructure Systems
30

30. Data from Database Servers
31

31. Data From Automation Servers
32

32. Data From DevOps Systems
33

33. Advanced analytics to
enable data-driven
DevOps decisions
34

34. DevOps VSM In a Glass Table View – all good
35

35. Glass Table view – threshold exceeded
36

36. Deep Dive – Build Service Status
37

37. Deep Dive – Build Server Error Message
38

38. Machine-learning &
predictive analytics in
a DevOps-driven
service delivery
culture
39

39. Apply Advanced Algorithms to Your Data
40

40. Track and Predict Anomalies
41

41. Summary
42

42. FROM EVERY TOOL, EVERY PROCESS, EVERY COMPONENT, ON-PREM OR OFF
One Constant -
Machine Data
43

43. Common Data Fabric
44
API
SDKs UI
Server, Storage.
N/W
Server
Virtualization
Operating
Systems
Infrastructure
Applications
Mobile
Applications
Cloud Services
Other Tools
Ticketing/Help
Desk
Custom
Applications
Visibility Across the Whole Ops Environment
API Services

44. Common Data Fabric
45
API
SDKs UI
Other Tools
Escalation/
Collaboration
Visibility Across the Whole Dev Lifecycle
Plan Code Build Test/QA Stage Release Config Monitor

45. Code Repository Automation
Systems
Application
Monitoring
CI/Build
Servers
Project & Issue
Tracking
Dev/Test/
Staging Servers InfrastructureCloud
46
Explore Visualize Customize ShareAnalyze
Common Data Fabric
API
SDKs UI

46. INCREASE APP
DELIVERY
VELOCITY
IMPROVE CODE
QUALITY
INCREASE
BUSINESS IMPACT
Improve the Impact of Application Delivery
47

47. For more, please visit Splunk.com
• Splunk for DevOps - www.splunk.com/DevOps
• Splunk DevOps Ecosystem Apps: splunkbase.splunk.com
• Splunk blogs: blogs.splunk.com
• Splunk community: www.splunk.com/community
• DevOps demo available

48. Sources/Additional Reading
● splunk.com/DevOps - Resources on Splunk for DevOps incl. case studies, customer stories, partners, products, videos, etc.
● dev.splunk.com – Resources for developing with or on ther Splunk platform, incl. SDKs, API Docs, guides, etc.
● blogs.splunk.com – Check the ‘DevOps’ and ‘Ansible’ tags for specifics, including how to deploy Spunk w/ Ansible
● splunkbase.splunk.com – Splunk add-ons and applications incl. Ansible Tower App for Splunk and 1000+ more
● DevOps Review 2016: Accelerating Innovation, Computing Research UK, July 2016
● 2016 State of DevOps Report, DevOps Research and Assessment
● The DevOps Cookbook, John Allspaw, Patrick Debois, Damon Edwards, Jez Humble, Gene Kim, Mike Orzen, and John Willis
● The Phoenix Project, Gene Kim, Kevin Behr, George Spafford
● Data-Driven DevOps: Use Metrics to Help Guide Your Journey, Gartner Inc. 2014, Cameron Haight and Tapati Bandopadhyay
● Metrics that Matter, Mark Michaelis, IntelliTect
● DevOps and the Cost of Downtime: Fortune 1000, IDC
● DevOps Best Practice Metrics: Fortune 1000 Survey, IDC, 2014
● The Seven Habits Of Highly Effective DevOps, Forrester Research, Inc., October 2, 2014
49

49. Thank You