Treating operational aspects of software as 'non-functional requirements' and 'an Ops problem' rather than a core part of the software product leads to poor live service and unexplained errors in Production. Traceability, deployability, recoverability, diagnosability, monitorability, and high quality logging are key features of a software system, along with user-visible features surfaced via the UI, or a capability of an API endpoint. However, many Product Owners understandably feel uneasy about taking on the (necessary) responsibility for prioritising operational features alongside user-visible and API features. This session brings Scrum Masters and Product Owners up to speed on operational features and covers proven practices for improving operability in an Agile context, empowering Product Owners to make effective prioritisation choices about all kinds of product features, whether user-visible or operational.

1. How to address operational aspects
effectively with Agile practices
Agile in the City – 20th November 2015
#agileinthecity
Matthew Skelton
Skelton Thatcher Consulting
@matthewpskelton

2. “Operational Features”
how to develop and test
prioritisation techniques
collaboration approaches

3. availability is the best feature

4. transforming technology
and teams
Cloud, Agile, DevOps
high impact expertise

5. transaction reporting
credit reference
FOREX
online payments

6. Operational Features

7. “the properties of a system
which make it work well in
Production”

8. Not PIMP MY RIDE
MORE
Greasy Mechanic

9. Not PIMP MY RIDE
MORE
Greasy Mechanic

10. Terminology

11. what happened to NFRs?
(non-functional requirements)

12. Non-Functional Functional

13. language impact

14. non-starter
non compos mentis
non-compete

15. nonsense !

17. holistic product view

18. How did we get to this?

19. admission: IT folk have been
guilty of making operational
features quite scary & mysterious

21. long lists of requirements
crazy test plans

22. poor explanation of needs
failure to engage stakeholders
gold-plating

24. de-mystify operational features

25. better approach
pragmatic and effective
rapid, safe, valuable

26. “the properties of a system
which make it work well in
Production”

27. Why value
Operational Features?

29. downtime:
$$$
reputation
($$)

31. non-linear increase in
complexity and problems

32. Internet of Things

33. we can no longer deal
manually with the scale/volume
of potential problems

35. agility and response to
incidents

37. remote car hacking:
security as an operational feature

38. “We have ‘cloud’ now”
(HA + DR + Backup + Metrics + Diagnostics + …)

39. think:
"when it fails, how will we recover?“
it will fail

40. How do we develop and test
Operational Features?

41. defined features
testable and measurable

42. ahead lie the ‘ilities’...

44. 1. What
2. How to test

45. Operational Hooks

47. Deployment Pipeline

49. Configurability

51. re-read config (SIGHUP)
text files in version control
inject settings – no ‘black boxes’

53. toggle features via config
“Postcode lookup unavailable”
 better UX

54. Deployability

56. immutable artefacts
concurrent releases (SxS)
symlinks

57. rapid
scriptable
simple failure modes

58. Maintainability

60. holding page as MVP!

61. live system component diagrams

62. modularity
ability to upgrade
version numbering (SemVer?)

63. BasketItemAdded
grep BasketItem

64. logging for insights

66. Testability

68. every component has a
/health endpoint

69. stubbed/mocked/faked
endpoints
test things individually

70. Recoverability

71. asynchronous service start
expect services to be erroring
logs are not wiped (rotated: okay)
avoid flooding logs

73. no nasty zombies after failures
MTTR more important than MTBF*
* for most kinds of F

74. Performance

76. run key 'hotspot' areas early
use a deployment pipeline
‘critical path’

78. early pipeline tests act as a
barometer for later
performance problems

80. derive transit time metrics

81. Monitorability

83. stream of metrics
transaction tracing

84. Resilience

86. assume missing or failing
Chaos Monkey
don’t crash on HTTP 503

87. Saboteur
+
deployment pipeline

88. Scalability

90. concurrent workers
queues and bottlenecks
throttling is your friend

92. Security and ‘securability’

94. securability by practice
SSL certs & HEARTBLEED

95. Gauntlt
+
deployment pipeline

96. Availability

98. “available but unusable"
synthetic transactions

99. special HTTP header: trigger
additional metrics/reporting

100. How the organisation affects
Operational Features

101. Budgets

102. bonuses:
story points delivered
tickets closed

103. Capex vs Opex
tax breaks

104. avoiding the Capex/Opex evil

105. Developers seen as more
valuable than Ops people
3x hiring bonus for Devs (!)

106. improved awareness in product
teams

108. share ownership and decision
making

109. features
end-user
operational
end-user

110. single product backlog

112. Product Owner on call for
incidents

113. tricky!
high degree of maturity
honesty about the product

114. Product Owner and Tech Lead
are both on the hook for
outages

116. 15-30% ‘tax’ on product
budget for operational aspects

117. AVOID
‘user features’ always taking
precedence over
‘operational features’

118. How to evaluate Operational
Features vs User Features

119. treat Ops team folk as another
user persona

121. alternatives to User Stories?

122. NOT:
"as a logging subsystem, I want..."

123. Metrics

124. Live: downtime, A/B for
operational aspects (speed)
Pre-live: time spent re-deploying

125. Metrics for better
conversations

126. metric-ify your delivery and test
infrastructure
 99.99% uptime, but 20
redeployments every time

128. Heuristics for operational features
30% of total product budget
30% of dev team time

129. Improving operational
awareness

130. Run Book Collaboration

131. Run Book
•Detailed description of
how the system operates
•Maintenance
•Repair
•Error recovery

132. Run Book / Ops Manual
• 1 Table of Contents
• 2 System Overview
• 2.1 Service Overview
• 2.2 Contributing Applications, Daemons, and Windows Services
• 2.3 Hours of Operation
• 2.4 Execution Design
• 2.5 Infrastructure and Network Design
• 2.6 Resilience, Fault Tolerance and High-Availability
• 2.7 Throttling and Partial Shutdown
• 2.8 Required Resources
• 2.9 Expected Traffic and Load
• 2.9.1 Hot or Peak Periods
• 2.9.2 Warm Periods
• 2.9.3 Cool or Quiet Periods
• 2.10 Environmental Differences
• 2.11 Tools
• 3 Security and Access Control
• 4 System Configuration
• 4.1 Configuration Management
• 5 System Backup and Restore
• 5.1 Backup Requirements
• 5.1.1 Special Files
• 5.2 Backup Procedures
• 5.3 Restore Procedures
• 6 Monitoring and Alerting
• 6.1 Error Messages
• 6.2 Events
• 6.3 Health Checks
• 6.4 Other Messages
• 7 Operational Tasks
• 7.1 Deployment
• 7.2 Batch Processing
• 7.3 Power Procedures
• 7.4 Routine Checks
• 7.4.1 System Rebuilds
• 7.5 Troubleshooting
• 8 Maintenance Tasks
• 8.1 Maintenance Procedures
• 8.1.1 Patching
• 8.1.1.1 Normal Cycle
• 8.1.1.2 Zero-Day Vulnerabilities
• 8.1.2 GMT/BST time changes
• 8.1.3 Cleardown Activities
• 8.1.3.1 Log Rotation
• 8.2 Testing
• 8.2.1 Technical Testing
• 8.2.2 Post-Deployment
• 9 Failure and Recovery Procedures
• 9.1 Failover
• 9.2 Recovery
• 9.3 Troubleshooting Failover and Recovery
• 10 Contact Details

133. Run Book / Ops Manual
2.1 Service Overview
2.2 Contributing
Applications,
Daemons, and
Windows Services
2.3 Hours of
Operation
2.4 Execution Design
2.5 Infrastructure and
Network Design
2.6 Resilience, Fault
Tolerance and High-
Availability
2.7 Throttling and
Partial Shutdown
2.8 Required
Resources
2.9 Expected Traffic
and Load

134. Run Book collaboration
Dev team is responsible for the first draft
“But I know nothing about Production!”
Encourages collaboration with Ops team

135. Will Gray

136. not documentation
build trust and understanding
automate more over time
http://runbookcollab.info/

137. choose tools that encourage
collaboration

140. http://rashidkpc.github.io/Kibana/images/screenshots/searchss.png

141. “How does [the use of] this tool
help people to collaborate*?”
* Work together, at the same
keyboard/screen

143. ‘How to choose tools for DevOps
and Continuous Delivery’
http://bit.ly/ChooseDevOpsTools

144. test early and often for
operational readiness

145. operational readiness
network testing
security testing
performance testing
auxiliary infrastructure testing:
monitoring
log aggregation
…

146. small set of rapid ‘weathervane’
tests for early warning

147. Network testing
iTrinegy network emulators
•Scripted setup and automated test runs
•http://www.itrinegy.com/
Saboteur:
•Network fault injection tool
•https://github.com/tomakehurst/saboteur

148. Security testing
Gauntlt: http://gauntlt.org/
SSL certs
HTTP
SQL injection
…
# nmap-simple.attack
Feature: simple nmap attack to check for open ports
Background:
Given "nmap" is installed
And the following profile:
| name | value |
| hostname | example.com |
Scenario: Check standard web ports
When I launch an "nmap" attack with:
"""
nmap -F <hostname>
"""
Then the output should match /80.tcps+open/
Then the output should not match:
"""
25/tcps+open
"""

149. When I launch an "nmap" attack with:
"""
nmap -F <hostname>
"""
Then the output should match
/80.tcps+open/

150. Deployment pipeline
Make operational testing visible

151. holistic product view

152. MVP: ‘service unavailable’ page

153. test early for operational features
using a deployment pipeline

154. single product backlog:
(user) features +
(operational) features

155. availability is the best feature

156. further reading
operabilitybook.comoperationalfeatures.com

157. thank you
http://skeltonthatcher.com/
enquiries@skeltonthatcher.com
@SkeltonThatcher
+44 (0)20 8242 4103