No matter how often you deploy your application, or how sophisticated your delivery pipeline is, you always need to know the quality status of the software you are building. This can only be done if you measure it. But measure what exactly? Andreas Grabner and Brett Hofer, app performance evangelists, explain five key metrics to increase your confidence in securing a safe build for production. • Learn why metrics can be huge quality gateways • Identify key metrics to take back to your team (Dev, Test, Ops and Business) • Understand how to use, measure and report these metrics • 3 short use cases and how using metrics can help you avoid them

1. 1 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #APMLive
5 Key Metrics to Release
Better Software Faster
Senior Solutions Architect
Brett Hofer
@brett_solarch
Performance Advocate
Andi Grabner
@grabnerandi

2. 2 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #APMLive
• Velocity insights – interesting to share for today
• Why do we care about metrics anyway?
• 3 interesting examples for learning
• Real life – let’s walk through a metric together
• Summary
• Q&A
Welcome! Today’s agenda

4. Waterfall Agile: 3 Years
1 Deployment per Month
“EVERYONE can do Continuous Delivery”
“EVERY MANUAL TESTER DOES AUTOMATION!”

7. So, why
did they do
it?

13. Uber: 1 million
Shared taxis per day
Didi: 5 million
Sources: KPCB, Uber. Date: May 2014 (Didi), Feb 2015 (Uber)

14. Rapidly changing markets,
requirements & user
expectations

16. Goal

17. Utmost goal: minimize cycle time
feature cycle time time
Customer / Market Users

18. Utmost goal: minimize cycle time
feature cycle time time
minimize Users

19. Utmost goal: minimize cycle time
feature cycle time time
This is where you
create value!
minimize
You

20. ROI?

21. High performers are more agile
30x 8,000x
more frequent
deployments
faster lead times
than their peers
Source: Puppet Labs 2013 State Of DevOps: http://puppetlabs.com/2013-state-of-devops-infographic

22. High performers are more reliable
Source: Puppet Labs 2013 State Of DevOps: http://puppetlabs.com/2013-state-of-devops-infographic
2x 12x
the change
success rate
faster mean time
to recover (MTTR)

23. Challenges

25. Fail
Faster!?

26. Right Focus?!

27. Unless you work for
Google or Microsoft 

29. I
learning from
others

30. 3 use cases
 WHY did it happen?
 HOW to avoid it!
 METRICS to guide you.
 TIPS along the way

32. Tip 1
don't push
without a plan

33. Mobile landing page of Super Bowl ad
434 Resources in total on that page:
230 JPEGs, 75 PNGs, 50 GIFs, …
Total size of ~
20MB

34. Fifa.com during Worldcup
Source: http://apmblog.compuware.com/2014/05/21/is-the-fifa-world-cup-website-ready-for-the-tournament/

35. Leverage Key
Metrics
1. # Resources
2. Size of Resources
3. Page Size

37. Tip 2
don't ASSUME you
know the environment

38. Distance calculation issues
480km biking
in 1 hour!
Solution: Unit Test in
Live App reports Geo
Calc Problems
Finding: Only
happens on certain
Android versions

39. 3rd party issues
Impact of bad
3rd party calls

40. Leverage Key
Metrics
4. # of functional errors
5. 3rd party calls

42. Tip 3
don't "blindly" (re)use
existing components

43. Requirement: we need a report

44. Using Hibernate results in 4k+ SQL Statements to
display 3 items!
Hibernate
Executes 4k+
Statements
Individual
Execution VERY
FAST
But Total SUM
takes 6s

45. 2 Additional Metrics
6. # SQL executions
7. # of SAME SQL’s

46. What have we
learned today?

47. 1. # Resources
2. Size of Resources
3. Page Size
4. # Functional Errors
5. 3rd Party calls
6. # SQL Executions
7. # of SAME SQLs
Metric
Based
Decisions
Are Cool

48. 8. Time Spent in API
9. # Calls into API
10. # of Domains
11. Total Size
12. # Items per Page
13. # AJAX per Page
14. etc.
And there
are other
great
metrics,
too!

49. We want to get from here …

50. To here!

51. How does
this stuff
actually
work?
Looking closely, what does it
actually look like in action?

52. Stage
Ops
Dev
Life of a metric in a
continuous delivery
real world example

53. Example Technologies Continuous DeliveryContinuous Delivery
IntelliJ IDEA
Ant
Silk Performer
Selenium
- Evaluate local
processes
- Code Linking
- Architectural
Evaluations
- Metric
evaluation prior
to check-ins
- Agent
injections to
monitor and
record tests
- Monitor for
metric
degradation
- Link builds to
test run
monitoring
- Report build
health based
on metric
evaluation

55. In this example the
Eclipse IDE is used.
A Maven project
structure has been
chosen
JAVA Class with
a few methods.
JAVA Method
begins using some
JDBC calls.
Dynatrace will see
all of the JDBC
executions made
in the JVM.
Developer authors the code

56. In this example the
JUnit Test Case
methods are built
The JUnit Testing
Method now makes
whichever calls the
test case should
cover. In this case
it’s a 1:1 call to our
sample DB call
Developer authors unit test

57. The MAVEN project
files are key to
structuring the build
and executions
throughout
the delivery chain.
Ultimately this file
is checked in and
then picked up
by the build server
linking metric
results to each
build.
Developer integrates Maven & Dynatrace

58. The Maven
Build/Test is executed
on the Developers
Workstation and local
Project, injecting the
Dynatrace agent and
Supplying metadata for
tracking.
The Maven POM file
has the Dynatrace
plug-in integrated.
Developer runs a Maven test

59. Developers might
evaluate their
own runs.
Developer uses Dynatrace to analyze a local build

60. Evaluate the actual
PurePaths of the
test to visualize
the SQL executions
per test method.
Developer uses Dynatrace to analyze a local build

61. Evaluate all of the
DB statistics around
the test or entire
local execution.
Developer uses Dynatrace to analyze a local build

62. Architecturally
validate the structure
of the calls that have
the suspected metric
prior to check-in
Developer uses Dynatrace to analyze a local build

63. CI Build Server Looks
For Updates. Pulls them
down and executes
the Maven goals or any
other necessary steps.
Developer finalizes the analysis and checks-in the
change

64. • The CI Build Server is
configured to poll the
SCM at a specific interval.
• Maven or any Integration
Testing Steps with
Dynatrace Integrations
are executed. Dynatrace
auto alerts may be fired.
• Dyntrace Reporting and
results are configured
into Jenkins
• Results are displayed and
deploy decisions can be
made automatically or
manually.
The CI build server is configured to poll for changes

65. 65 @Dynatrace
12 0 120ms
3 1 68ms
Build 15 testDBFunctions OK
testWebServiceCalls OK
Build 12 testDBFunctions OK
testWebServiceCalls OK
Build 13 testDBFunctions FAILED
testWebServiceCalls OK
Build 14 testDBFunctions OK
testWebServiceCalls OK
Build # Test Case Status # SQL # Excep CPU
12 0 120ms
3 1 68ms
12 5 60ms
3 1 68ms
75 0 230ms
3 1 68ms
Test & Monitoring Framework Results Architectural Data
We identified a regresesion
Problem solved
Exceptions probably reason for
failed tests
Problem fixed but now we have an
architectural regression
Problem fixed but now we have an
architectural regression
Now we have the functional and
architectural confidence
Let’s look behind the
scenes
Deployment decisions are made
a “go” or “no go”

68. Thank-you!
Time for Q & A
Andi Grabner
@grabnerandi
http://blog.dynatrace.com
Brett Hofer
@brett_solarch
http://blog.dynatrace.com

69. Participate in our Forum ::
community.dynatrace.com
Like us on Facebook ::
facebook.com/dynatrace
Follow us on LinkedIn ::
linkedin.com/company/dynatrace
Connect with us!
Follow us on Twitter ::
twitter.com/dynatrace
Watch our Videos & Demos ::
youtube.com/dynatrace
Read our Blog ::
application-performance-blog.com