IT organizations are increasingly using machine data - including in DevOps practices - to get away from 'vanity metrics' and instead to generate 'metrics that matter'. These metrics provide visibility into the delivery of new application code and the business value of DevOps, to both IT and business stakeholders. Machine data provides DevOps teams and others - including QA, secops, CxOs and LOB leaders - with meaningful and actionable metrics. This allows stakeholders to monitor, measure, and continuously improve the velocity and quality of code throughout the software lifecycle, from dev/test to customer-facing outcomes and business impact. In this session Andi Mann, chief technology advocate at Splunk, will share core methodologies, interesting case studies, key success factors and 'gotcha' moments from real-world experience with mining machine data to produce 'metrics that matter' in a DevOps context.

1. Copyright © 2016 Splunk Inc.
Mining Machine Data
for ‘Metrics that Matter’
in a DevOps Workflow

2. Abstract (Hidden)
IT organizations are increasingly using machine data – including in DevOps practices – to
get away from ‘vanity metrics’ and instead to generate ‘metrics that matter’. These metrics
provide visibility into the delivery of new application code and the business value of
DevOps, to both IT and business stakeholders.
Machine data provides DevOps teams and others – including QA, secops, CxOs and LOB
leaders – with meaningful and actionable metrics. This allows stakeholders to monitor,
measure, manage, and continuously improve the velocity and quality of code throughout
the software lifecycle, from dev/test to customer-facing outcomes and business impact.
In this session Andi Mann, chief technology advocate at Splunk, will share core
methodologies, interesting case studies, key success factors and ‘gotcha’ moments from
real-world experiences with mining machine data to produce ‘metrics that matter’ in a
DevOps context.

3. DevOps is a Culture of Empathy & Sharing
INTEGRATION
COLLABORATION
COMMUNICATION
BETWEEN DEV AND OPS
TO DELIVER BETTER SOFTWARE, FASTER
METHODS FOR IMPROVING

4. Shared Feedback Enables ‘The Three Ways’
Gene Kim, “DevOps Cookbook” and “The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win.”

5. Empowered DevOps Teams
Empathy - more than
understanding
• Feel your teammates’
pain
• Understand their
work and your impact
Empowerment - more
than making decisions
• Be responsible in
decisions, activities
• Be accountable to
your team of teams

6. DevOps Workflow is Becoming Complex and Opaque
6
Build
(Jenkins,
Bamboo)
Code
(Git,
MS-TFS)
Plan
(Jira,
Rally)
Test/QA
(Cucumber,
SonarQube)
Stage
(Pivotal,
AWS)
Release
(Jenkins,
Octopus)
Data Center
Device
Data
Engagement
Data
Config
(Puppet,
Ansible)
Monitor
(NewRelic,
Dynatrace)
Cloud Services Network Services
www/HTTP
Data
Social
Sentiment
Wire
Data
Application
Data
Continuous Integration (CI) / Continuous Delivery (CD)
Site Reliability Engineering
Business Impact Monitoring
API ServicesSecurity/Compliance

7. DevOps complexity raises risk of failure
● Slower Speed
● Longer MTTR
● Lower Quality
● Reduced Agility
● Poor Visibility
● Hard to Scale
● Increased Waste
● Impaired Collaboration
7
DevOps
From Hype Cycle for Application Services 2015, Gartner Group, July 2015, Betsy Burton, Philip Allega,
http://www.gartner.com/document/3096018

8. From every tool, every process, every component, on-prem or off
The One Constant:
Machine Data

9. Common Data Fabric
9
API
SDKs UI
Other Tools
Escalation/
Collaboration
Visibility Across the Whole Dev Lifecycle
Plan Code Build Test/QA Stage Release Config Monitor

10. Common Data Fabric
10
API
SDKs UI
Server, Storage.
N/W
Server
Virtualization
Operating
Systems
Infrastructure
Applications
Mobile
Applications
Cloud Services
Other Tools
Ticketing/Help
Desk
Custom
Applications
Visibility Across the Whole Ops Environment
API Services

11. Machine Data From DevOps Tools
11

12. Provisioning and Config Metrics
12

13. Machine Data from QA/Pre-Prod/Staging
13

14. Machine Data from Release Servers
14

15. Machine Data from Infrastructure Systems
15

16. Machine Data from Database Servers
16

17. Machine Data from Customer Systems

18. What do you measure
and why?

19. I’m working
super hard!!
That’s my
stapler.

20. 20
Yeah, but …
… what are you
achieving?
I’m gonna need you
to come in Sunday.

21. 21
Daily Active
Users?
Installs?
Downloads?
Sales?

22. DevOps Metrics that Matter
Culture
e.g.
• Retention
• Satisfaction
• Callouts
Process
e.g.
• Idea-to-cash
• MTTR
• Deliver time
Quality
e.g.
• Tests passed
• Tests failed
• Best/worst
Systems
e.g.
• Throughput
• Uptime
• Build times
Activity
e.g.
• Commits
• Tests run
• Releases
Impact
e.g.
• Signups
• Checkouts
• Revenue

23. Gartner’s DevOps ‘Metrics that Matter’
Gartner Inc., Data-Driven DevOps: Use Metrics to Help Guide Your Journey, 29 May 2014 G00264319, Analyst(s): Cameron Haight | Tapati Bandopadhyay

24. IDC’s DevOps ‘Metrics that Matter’

25. What Are Your ‘Metrics That Matter’?

26. Finding Your Metrics That Matter
Work from business backwards
Mine realtime machine data
Close the feedback loops
26

27. Outcomes

28. Measurement drives Feedback loops
Velocity
Deliver on time
& on budget
IT is delivering on
time, on budget
IT and Business
Leaders
Impact
Deliver code for
business needs
IT is achieving
business goals
IT and Business Leaders,
Customers, Staff
Show you when you deliver. And when you don’t.
Quality
Deliver the
quality you
promised
We deliver a quality
experience for users
Dev and Ops
Organizations

29. Measurement identifies ‘Waste’
Plan
Develop
(UI)
Develop
(Db)
Develop
(M’ware)
Develop
(Backend)
Security
Test
Monitor
Build
(Prod)
Architect
Secure/
Comply
DeployAccept
Unit
Test
Document
Cap Plan
Train
Feedback
Integration
Test
Configure
System
Test
Launch
CAB
Develop
(APIs)
Budget
Build
(Dev)
Mgmt/
Tooling
W
W
W
W
W
W
W
W
W
16 40 52 35 96 40 48 24 --8 2 5 6 8 2 12

30. Measurement Ensures Transparency
• Release when
ready, not a date!
• Best / worst
developers
• Best / worst
providers
• Impact of new
code on ops
• Impact of new
code on biz

31. Measurement Enables Continuous Improvement
Defect
Information
Capacity
Planning
Quality
Standards
Enhancement
Requests
Integration
Requirements
Acceptance
Metrics
Service Levels
and KPIs
Application Development Test and Acceptance Production
BuildCodePlan Test/QA Stage Release Config Monitor
Infrastructure
Dependencies

32. Measurement Improves Quality
Code quality scans Static security scans
White BoxDeveloper
checks in code
Automated
Acceptance Tests
Dynamic Security
Scans
Black Box
“Chaos Monkey”
tests
Test Fail:
Return
Test Fail:
Return
X
X
Production
QA Prod Pattern
QA Pattern Library
Test Pass:
Promote
Test Pass:
Promote to
Production
Pattern
library used
for test and
QA

33. Measurement Accelerates Velocity
Pivot & improve with
Continuous Insights
Product Managers
identify new
opportunities
Continuously delivered to market
… and Auditors are “happy”

34. Measurement Aligns Business Impact

35. Fast-feedback loop for actionable commercial insights
So You Can Innovate at Market Speed
BUSINESS DEV/OPS CUSTOMERS
HOW IS OUR:
• Security?
• Quality?
• Stability?
• Performance?
• Compliance?
HOW IS OUR:
• Market Launch?
• Feature Usage?
• Marketing Changes?
• Prioritization?
• Customer Sat?

36. Summary

37. Metrics that Matter Drive Better Feedback Loops
Improve
Application Velocity
Visibility across silos,
tools, and processes
exposes bugs and
bottlenecks so you
can remediate,
iterate, and innovate
faster.
Improve
Application Quality
Track quality across
multiple teams,
tools, systems, and
service providers, so
you can find and fix
more issues before
production
Improve
Application Impact
Real-time analytics
correlates
application delivery
with business goals,
so you can drive
better experience
and iterate faster

38. Sources/Additional Reading
● splunk.com/DevOps - Resources on Splunk for DevOps incl. case studies, customer stories, partners, products, videos, etc.
● dev.splunk.com – Resources for developing with or on ther Splunk platform, incl. SDKs, API Docs, guides, etc.
● blogs.splunk.com – Check the ‘DevOps’ and ‘Ansible’ tags for specifics, including how to deploy Spunk w/ Ansible
● splunkbase.splunk.com – Splunk add-ons and applications incl. Ansible Tower App for Splunk and 1000+ more
● DevOps Review 2016: Accelerating Innovation, Computing Research UK, July 2016
● 2016 State of DevOps Report, DevOps Research and Assessment
● The DevOps Cookbook, John Allspaw, Patrick Debois, Damon Edwards, Jez Humble, Gene Kim, Mike Orzen, and John Willis
● The Phoenix Project, Gene Kim, Kevin Behr, George Spafford
● Data-Driven DevOps: Use Metrics to Help Guide Your Journey, Gartner Inc. 2014, Cameron Haight and Tapati Bandopadhyay
● Metrics that Matter, Mark Michaelis, IntelliTect
● DevOps and the Cost of Downtime: Fortune 1000, IDC
● DevOps Best Practice Metrics: Fortune 1000 Survey, IDC, 2014
38

39. Copyright © 2016 Splunk Inc.
Thank You!
Andi Mann
Chief Technology Advocate, Splunk
@andimann