Your product doesn't matter if you can't get it into the hands of your users. And once in their hands, it does't matter if you can't quickly detect and respond to feedback and usage patterns to realize the value of these opportunities. Product organizations need to be able to Continuously Deliver their product - shipping small valuable increments to users, gathering feedback, and iterating on opportunities. In recent years, there have been many silver bullets to enable Continuous Delivery - practices such as Lean Startup, Agile, LeanUx, ChatOps, and DevOps have promised to help ship better products faster while responding more quickly to your users. And tools, frameworks, programming languages, containers, and microservices have promised to reduce the effort and complexity to do so. So do you really need all of these things? And how to they all fit together? To be an effective Product Manager, it's essential to understand the role technical practices and tools to enable the Continuous Delivery of your product. As the keeper of value and priority, Product Managers often decide between product and technical investments. This session is for Product Managers and leadership who want to gain empathy and examples of why balancing product, process, and technical investments are essential to creating a great product that users love! Shared through the perspective and stories of a Product Manager on the CA Agile Central release train, this session explores how technical practices and tools are essential to enabling Continuous Delivery - shipping value daily, tighten feedback cycles, and more quickly reacting to opportunities.

2. TITLE SPONSORS
TRACK SPONSORS

3. HEADLINE SPONSORS
PARTNER SPONSORS MEMBER SPONSORS
Baker Hostetler
Bradford, LTD
Capital One
Coastal Cloud
Colorado Impact Fund
Connect For Health Colorado
Cooley
Corus360
EKS & H
Event Integrity
Full Contact
Gary Community Investments
General Assembly
Groundfloor Media / CenterTable
Guiceworks
ImageSeller
Ink Monstr
Intelivideo
Luna Gourmet Coffee & Tea
Nanno
Office of Economic Development
Pass Gas Denver
Slalom
Wazee Digital
Accenture
Bridgepoint Education
Butler Snow
Delta Tables
Hogan Lovells
Meyer Law
Name.com
Slifer Smith & Frampton
SoGnar
Sounddown
Swiftpage
The Denver Foundation
Zipcar

4. CONTINUOUS DELIVERY
WILL MAKE OR BREAK
YOUR PRODUCT
Adam Zolyak
Product, Agile Central (Rally) @ CA Technologies
@azolyak #DENStartupWeek

5. Agile Central
WHO AM I?
•Member of the Product team
•Focus on the rapid, easy delivery of
value to our customers
•Work with 16x other teams on CA
Agile Central

6. This is NOT a talk about
DELIVERING THE RIGHT THING
@azolyak #DENStartupWeek

7. This is a talk about
DELIVERING THINGS OFTEN
…and WHY THAT MATTERS
@azolyak #DENStartupWeek

9. How OFTEN can you deliver to your users?
@azolyak #DENStartupWeek

10. HOW OFTEN CAN YOU DELIVER ?
“The only way to win is to learn faster then anyone else.”
“We must learn what customers really want, not what they say they
want or what we think they want.”
Eric Ries, The Lean Startup
@azolyak #DENStartupWeek

11. HOW OFTEN CAN YOU DELIVER ?
1x Every 2 Weeks
=
2x Feedback Cycles Per Month
@azolyak #DENStartupWeek

12. HOW OFTEN CAN YOU DELIVER ?
1x A Week
=
4x Feedback Cycles Per Month
@azolyak #DENStartupWeek

13. HOW OFTEN CAN YOU DELIVER ?
1x A Day
=
20x Feedback Cycles Per Month
@azolyak #DENStartupWeek

14. HOW OFTEN CAN YOU DELIVER ?
5x A Day
=
100x Feedback Cycles Per Month
@azolyak #DENStartupWeek

15. HOW OFTEN SHOULD YOU DELIVER ?
On Demand
=
???x Feedback Cycles Per Month
@azolyak #DENStartupWeek

16. How OFTEN can you deliver to your users?
@azolyak #DENStartupWeek

17. HOW OFTEN CAN YOU DELIVER ?
https://blog.newrelic.com/2016/02/04/data-culture-survey-results-faster-deployment/ @azolyak #DENStartupWeek

18. But it’s not just about how OFTEN you deliver…
@azolyak #DENStartupWeek

19. How QUICKLY can you deliver to your users?
@azolyak #DENStartupWeek

20. HOW QUICKLY CAN YOU DELIVER?
@azolyak #DENStartupWeek

21. HOW QUICKLY CAN YOU DELIVER?
Step 1
Step 2
Step 3
Step 5
Step 4
@azolyak #DENStartupWeek
Your
Process

22. HOW QUICKLY CAN YOU DELIVER?
Step 1
Step 2
Step 3 Step 5
Step 4
@azolyak #DENStartupWeek
Your Process

23. How QUICKLY can you recover
when 💩 GOES 🔥?
@azolyak #DENStartupWeek

25. WHAT ARE THESE METRICS FOR YOUR PRODUCT?
12x
changes a day
2-3 day median
cycle time
As short as 50
minutes
Agile Central August 2017 @azolyak #DENStartupWeek

26. That’s great, but SO WHAT?
@azolyak #DENStartupWeek

27. YOU!
… make investment and prioritization decisions
… decide between product, process, and tool investments
… determine how often and quickly you can deliver
@azolyak #DENStartupWeek

28. How often and quickly do YOU want to deliver
to our users?
@azolyak #DENStartupWeek

29. How would YOU ACT if you wanted these
things?
@azolyak #DENStartupWeek

30. So let’s learn the
BASICS OF CONTINUOUS DELIVERY
… to build empathy and understanding
…to make good decisions
@azolyak #DENStartupWeek

31. ShapeandPrioritize
Design/UX
Develop
Test
Deploy
Measure
Operate
Continuous Delivery
@azolyak #DENStartupWeek

32. BASICS OF CONTINUOUS DELIVERY
@azolyak #DENStartupWeek

33. Let’s talk about PRINCIPLES…
… your PRACTICES & TOOLS may vary
@azolyak #DENStartupWeek

34. Principles of Continuous Delivery
MAKE SMALL, FREQUENT CHANGES
@azolyak #DENStartupWeek

35. PRINCIPLES OF CONTINUOUS DELIVERY
MAKE SMALL, FREQUENT CHANGES
Shape and Size Work into Small Increments. Iterate.
Value
Time
@azolyak #DENStartupWeek

36. PRINCIPLES OF CONTINUOUS DELIVERY
MAKE SMALL, FREQUENT CHANGES
Shape and Size Work into Small Increments. Iterate.
Value
Time
Value
Time
Value
Time
@azolyak #DENStartupWeek

37. PRINCIPLES OF CONTINUOUS DELIVERY
MAKE SMALL, FREQUENT CHANGES
Shape and Size Work into Small Increments. Iterate.
http://agileforall.com/patterns-for-splitting-user-stories/ @azolyak #DENStartupWeek

38. PRINCIPLES OF CONTINUOUS DELIVERY
MAKE SMALL, FREQUENT CHANGES
Shape and Size Work into Small Increments. Iterate.
Value
Time
Value
Time
Value
Time
Pivot
or
Persevere
Pivot
or
Persevere
@azolyak #DENStartupWeek

39. PRINCIPLES OF CONTINUOUS DELIVERY
MAKE SMALL, FREQUENT CHANGES
Multidisciplinary Teams That Can Deliver Without Cross Team Dependencies
•Shape
•UX
•Develop
•Test
•Deploy
•Measure
•Operate
Delivery
@azolyak #DENStartupWeek

40. Delivery
Multidisciplinary Teams That Can Deliver Without Cross Team Dependencies
PRINCIPLES OF CONTINUOUS DELIVERY
MAKE SMALL, FREQUENT CHANGES
@azolyak #DENStartupWeek

41. Delivery
Multidisciplinary Teams That Can Deliver Without Cross Team Dependencies
PRINCIPLES OF CONTINUOUS DELIVERY
MAKE SMALL, FREQUENT CHANGES
@azolyak #DENStartupWeek

42. Delivery
Multidisciplinary Teams That Can Deliver Without Cross Team Dependencies
PRINCIPLES OF CONTINUOUS DELIVERY
MAKE SMALL, FREQUENT CHANGES
@azolyak #DENStartupWeek

43. Delivery
Multidisciplinary Teams That Can Deliver Without Cross Team Dependencies
PRINCIPLES OF CONTINUOUS DELIVERY
MAKE SMALL, FREQUENT CHANGES
@azolyak #DENStartupWeek

44. Delivery
Multidisciplinary Teams That Can Deliver Without Cross Team Dependencies
PRINCIPLES OF CONTINUOUS DELIVERY
MAKE SMALL, FREQUENT CHANGES
@azolyak #DENStartupWeek

45. Delivery
Multidisciplinary Teams That Can Deliver Without Cross Team Dependencies
PRINCIPLES OF CONTINUOUS DELIVERY
MAKE SMALL, FREQUENT CHANGES
@azolyak #DENStartupWeek

46. PRINCIPLES OF CONTINUOUS DELIVERY
MAKE SMALL, FREQUENT CHANGES
Make Small Changes to Small Things
Code Tests Delivery
@azolyak #DENStartupWeek

47. Code Tests Delivery
Code Tests
Code Tests
Delivery
Delivery
PRINCIPLES OF CONTINUOUS DELIVERY
MAKE SMALL, FREQUENT CHANGES
Make Small Changes to Small Things
@azolyak #DENStartupWeek

48. Principles of Continuous Delivery
BAKE QUALITY INTO THE PROCESS
@azolyak #DENStartupWeek

49. PRINCIPLES OF CONTINUOUS DELIVERY
BAKE QUALITY INTO THE PROCESS
Automated Tests Are Real
@azolyak #DENStartupWeek

50. The Life of A Change - Idea to Outcome
PRINCIPLES OF CONTINUOUS DELIVERY
BAKE QUALITY INTO THE PROCESS
Delivery Process
@azolyak #DENStartupWeek

51. The Life of A Change - Branch and Start Working
Branch
PRINCIPLES OF CONTINUOUS DELIVERY
BAKE QUALITY INTO THE PROCESS
@azolyak #DENStartupWeek

52. The Life of A Change - Pair Programming
Pair Programming
PRINCIPLES OF CONTINUOUS DELIVERY
BAKE QUALITY INTO THE PROCESS
@azolyak #DENStartupWeek
•Sharing Knowledge
•Different Perspectives
•Optional (not required)

53. The Life of A Change - Pair Programming
PRINCIPLES OF CONTINUOUS DELIVERY
BAKE QUALITY INTO THE PROCESS
@azolyak #DENStartupWeek

54. The Life of A Change - Commit, Compose Test Envs, Run Tests
Commit
Automatically Triggers
•Unit Tests
•Integration Tests
•End to End Tests
Code
Code
Code
Code
Code
Code
Ephemeral
Test
Infrastructure
Branch
PRINCIPLES OF CONTINUOUS DELIVERY
BAKE QUALITY INTO THE PROCESS
@azolyak #DENStartupWeek

55. The Life of A Change - Commit, Compose Test Envs, Run Tests
x 50 a Day
PRINCIPLES OF CONTINUOUS DELIVERY
BAKE QUALITY INTO THE PROCESS
@azolyak #DENStartupWeek

56. The Life of A Change - More Commits, Compose Test Envs, Run Tests
Commit Commits
Branch
Automatically Triggers
•Unit Tests
•Integration Tests
•End to End Tests
PRINCIPLES OF CONTINUOUS DELIVERY
BAKE QUALITY INTO THE PROCESS
@azolyak #DENStartupWeek

57. The Life of A Change - Pull Request and Get Feedback
Commit Commits
Branch
Pull
Request
PRINCIPLES OF CONTINUOUS DELIVERY
BAKE QUALITY INTO THE PROCESS
@azolyak #DENStartupWeek

58. The Life of A Change - Pull Request and Get Feedback
PRINCIPLES OF CONTINUOUS DELIVERY
BAKE QUALITY INTO THE PROCESS
@azolyak #DENStartupWeek

59. The Life of A Change - Pull Request and Get Feedback
PRINCIPLES OF CONTINUOUS DELIVERY
BAKE QUALITY INTO THE PROCESS
@azolyak #DENStartupWeek

60. The Life of A Change - Pull Request and Get Feedback
PRINCIPLES OF CONTINUOUS DELIVERY
BAKE QUALITY INTO THE PROCESS
@azolyak #DENStartupWeek

61. The Life of A Change - More Commits, Compose Test Envs, Run Tests
Commit Commits
Branch
Commit
Pull
Request
Automatically Triggers
•Unit Tests
•Integration Tests
•End to End Tests
PRINCIPLES OF CONTINUOUS DELIVERY
BAKE QUALITY INTO THE PROCESS
@azolyak #DENStartupWeek

62. The Life of A Change - Manual Exploratory Testing
Commit Commits
Branch
Commit
Pull
Request
•Manual Exploratory Testing
•Docs and Screenshots
PRINCIPLES OF CONTINUOUS DELIVERY
BAKE QUALITY INTO THE PROCESS
@azolyak #DENStartupWeek

63. The Life of A Change - Manual Exploratory Testing
x 60 a Day
PRINCIPLES OF CONTINUOUS DELIVERY
BAKE QUALITY INTO THE PROCESS
@azolyak #DENStartupWeek

64. Team Reviews / Demos
Team Reviews / Demos
PRINCIPLES OF CONTINUOUS DELIVERY
BAKE QUALITY INTO THE PROCESS
@azolyak #DENStartupWeek

65. The Life of A Change - Merge, Test, and Deploy
Commit Commits
Branch
Commit
Pull
Request
Merge
Automatically Triggers
•Unit Tests
•Integration Tests
•End to End Tests
•Deploy to Prod
PRINCIPLES OF CONTINUOUS DELIVERY
BAKE QUALITY INTO THE PROCESS
@azolyak #DENStartupWeek

66. The Life of A Change - Measure Impact of Change
Commit Commits
Branch
Commit
Pull
Request
Merge
Measure
•User Metrics
•User Feedback
•System Metrics
•User Interviews
PRINCIPLES OF CONTINUOUS DELIVERY
BAKE QUALITY INTO THE PROCESS
@azolyak #DENStartupWeek

67. The Life of A Change - Cycle Time of A Change
Commit Commits
Branch
Commit
Pull
Request
Merge
2-3 Days
Measure
•User Metrics
•User Feedback
•System Metrics
•User Interviews
Minutes to Weeks
PRINCIPLES OF CONTINUOUS DELIVERY
BAKE QUALITY INTO THE PROCESS
@azolyak #DENStartupWeek

68. Principles of Continuous Delivery
LIMIT THE BLAST RADIUS OF CHANGES
@azolyak #DENStartupWeek

69. PRINCIPLES OF CONTINUOUS DELIVERY
LIMIT THE BLAST RADIUS OF CHANGES
Continuously Deploy, Release When Ready
@azolyak #DENStartupWeek

70. Feature Toggles
Enable / Disable Features
•By User
•By Customer
•For Everyone
PRINCIPLES OF CONTINUOUS DELIVERY
LIMIT THE BLAST RADIUS OF CHANGES
@azolyak #DENStartupWeek

71. A/B Testing
99% of Users
Implementation A Implementation B
1% of Users
vs.
Measure
•User Metrics
•User Feedback
•System Metrics
•User Interviews
PRINCIPLES OF CONTINUOUS DELIVERY
LIMIT THE BLAST RADIUS OF CHANGES
@azolyak #DENStartupWeek

72. Principles of Continuous Delivery
MEASURE THE IMPACT OF CHANGES
@azolyak #DENStartupWeek

73. PRINCIPLES OF CONTINUOUS DELIVERY
MEASURE THE IMPACT OF CHANGES
How Long Does It Take You to Achieve Customer Value
Measure Measure
…
Pivot
or
Persevere
Pivot
or
Persevere
Measure
???
@azolyak #DENStartupWeek
Agile Central

74. Principles of Continuous Delivery
INSPECT AND ADAPT
@azolyak #DENStartupWeek

75. PRINCIPLES OF CONTINUOUS DELIVERY
INSPECT AND ADAPT
Understand What’s Slowing You Down and Eliminate It
@azolyak #DENStartupWeek

76. PRINCIPLES OF CONTINUOUS DELIVERY
INSPECT AND ADAPT
Understand What’s Slowing You Down and Eliminate It
@azolyak #DENStartupWeek

77. PRINCIPLES OF CONTINUOUS DELIVERY
•Make Small, Frequent Changes
•Bake Quality Into the Process
•Limit Blast Radius of Changes
•Measure Impact of Changes
•Inspect and Adapt
@azolyak #DENStartupWeek

78. So how will you INVEST and PRIORITIZE?
@azolyak #DENStartupWeek

79. HOW WOULD YOU ACT if you wanted these
things?
@azolyak #DENStartupWeek

80. Customer
Outcome A
Deploying More Frequentlyvs.
HOW WILL YOU INVEST AND PRIORITIZE?
@azolyak #DENStartupWeek

81. Automating Tests Customer Outcome Bvs.
HOW WILL YOU INVEST AND PRIORITIZE?
@azolyak #DENStartupWeek

82. Customer Outcome A
Customer Outcome BAutomate Tests
Customer Outcome C
Deploy More Frequently
It’s A Balance
HOW WILL YOU INVEST AND PRIORITIZE?
@azolyak #DENStartupWeek

83. ASKs
@azolyak #DENStartupWeek

84. ASKS
@azolyak #DENStartupWeek
1. Know how frequently and how quickly you can delivery to
your users.
2. Know the longest step in your delivery process. What’s
the next step to make it shorter?
3. Never stop getting better! There are too many big
challenging problems to solve! The world needs you!

85. QUESTIONS / DISCUSSION
@azolyak #DENStartupWeek