In this talk, we explore five practical, tried-and-tested, real world techniques for improving operability with many kinds of software systems, including cloud, Serverless, Microservices, on-premise, and IoT. Based on our work in many industry sectors, we will share our experience of helping teams to improve the operability of their software systems through these straightforward, team-friendly techniques. From a talk given at DevOpsCon Munich 2017 https://devopsconference.de/microservices/practical-team-focused-operability-techniques-for-distributed-systems/

1. Practical, Team-focused Operability
Techniques for Distributed Systems
Matthew Skelton, Skelton Thatcher Consulting
skeltonthatcher.com / @SkeltonThatcher
DevOpsCon 2017, MUC, DE – 20 Nov 2017

2. Today
Run Book dialogue sheets
User Personas for dashboards
Endpoint healthchecks
Modern logging
Correlation IDs

3. You ?
Software Developer
Tester / QA
DevOps Engineer
Team Leader
Head of Department

4. Operability:
use these
team collaboration techniques
to make software work better

6. About me
Matthew Skelton
@matthewpskelton
Co-founder at
Skelton Thatcher Consulting
skeltonthatcher.com

7. Team Guide to
Software Operability
Matthew Skelton & Rob Thatcher
skeltonthatcher.com/publications
Download a free sample chapter

8. Team Guide series
#Operability #BusinessMetrics #Testability #Releasability
skeltonthatcher.com/publications

9. Practical Operability
Techniques for Teams

10. What is operability?

11. Operability
Security, Scaling, Reliability,
Failover, Backups, Diagnostics,
Error Handling, Fault Tolerance,
High Availability, SLAs, Load,
Storage, Hours of Operation, …

13. Operability is CRUCIAL
for 24x7 cloud services!

14. Operability
making software work well
in Production

16. Run Book dialogue sheets

18. runbooktemplate.infoRun Book dialogue sheets

19. Run Book dialogue sheets
Checklists for typical operational
considerations
Team-friendly exploration

20. System characteristics
Hours of operation
During what hours does the service or system actually need to operate? Can portions or features of the
system be unavailable at times if needed?
Hours of operation - core features
(e.g. 03:00-01:00 GMT+0)
Hours of operation - secondary features
(e.g. 07:00-23:00 GMT+0)
Data and processing flows
How and where does data flow through the system? What controls or triggers data flows?
(e.g. mobile requests / scheduled batch jobs / inbound IoT sensor data )
…

21. http://runbooktemplate.info/

22. runbooktemplate.infoRun Book dialogue sheets

23. Run Book dialogue sheets
Collaborate on runtime characteristics
of the software system.
"Early warning" of
forgotten operational aspects!

24. Today
Run Book dialogue sheets
User Personas for dashboards
Endpoint healthchecks
Modern logging
Correlation IDs

25. Lightweight user personas

27. Lightweight user personas:
Ops Engineer
Test Engineer
Build & Deployment Engineer
Service Owner

28. Lightweight user personas:
Consider the User Experience (UX) of
engineers and team members using
and working with the software

29. http://www.keepitusable.com/blog/?tag=alan-cooper

30. https://www.geckoboard.com/blog/visualisation-upgrades-
progressing-towards-a-more-useful-and-beautiful-dashboard/

31. Lightweight user personas:
What data does the User Persona
need visible on a dashboard in order
to make decisions rapidly & safely?

32. Today
Run Book dialogue sheets
User Personas for dashboards
Endpoint healthchecks
Modern logging
Correlation IDs

33. Endpoint healthchecks

35. endpoint healthchecks
Every runnable app/service/daemon
exposes /status/health
An HTTP GET to the endpoint returns:
200 – "I am healthy"
500 – "I am sick"

36. endpoint healthchecks
Each component is responsible for
determining its own health and
viability – this is very contextual

37. endpoint healthchecks
Use JSON as a response type –
parsable by both
machines and humans!

39. endpoint healthchecks
For databases and other non-HTTP
components, run a lightweight HTTP
service in front of the component
200 / 500 responses

40. Helper service

41. https://github.com/Lugribossk/simple-dashboard

42. Today
Run Book dialogue sheets
User Personas for dashboards
Endpoint healthchecks
Modern logging
Correlation IDs

43. Logging with Event IDs

45. logging with Event IDs
reduce time to detect problems
increase team engagement
increase configurability
enhance collaboration
#operability

46. search by
event
Event ID
{Delivered,
InTransit,
Arrived}

47. transaction
trace
Correlation ID
612999958…

48. How many distinct event
types (state transitions)
in your application?

50. represent distinct states

51. enum
Human-readable sets:
unique values, sparse,
immutable
C#, Java, Python, node
(Ruby, PHP, …)

52. Technical
Domain
public enum EventID
{
// Badly-initialised logging data
NotSet = 0,
// An unrecognised event has occurred
UnexpectedError = 10000,
ApplicationStarted = 20000,
ApplicationShutdownNoticeReceived = 20001,
MessageQueued = 40000,
MessagePeeked = 40001,
BasketItemAdded = 60001,
BasketItemRemoved = 60002,
CreditCardDetailsSubmitted = 70001,
// ...
}

53. BasketItemAdded = 60001
BasketItemRemoved = 60002

54. log using Event IDs
with a modern
‘structured logging’ library

55. example:
https://github.com/EqualExperts/opslogger
Sean Reilly
@seanjreilly

59. Example: video processing
On-demand processing of TV and
mobile streaming adverts
Ad-agency  TV broadcaster
High throughput
Glitch-free video & audio

60. Storage I/O
Worker Job
Queue
Upload

63. Example: video processing
Discover processing bottlenecks
Trigger alerts via LogEntries /
HostedGraphite
Report on KPIs
Target areas for improvement

64. http://honeycomb.tv/

65. Today
Run Book dialogue sheets
User Personas for dashboards
Endpoint healthchecks
Modern logging
Correlation IDs

66. Correlation IDs

68. ‘Unique-ish’ identifier for each request
Passed through downstream layers

69. Unique-ish ID

70. Synchronous HTTP:
X-HEADER e.g. X-trace-id
X-trace-id: 348e1cf8
If header is present, pass it on
(Yes, RFC6648, but this is internal only)

71. Asynchonous (queues, etc.):
Message Attributes, name:value pair
e.g. "trace-id":"348e1cf8"
AWS SQS: SendMessage() / ReceiveMessage()
Log the Correlation ID if present

72. Example: OpenTracing / PCF
3 tracing elements:
TraceID, SpanID, ParentSpan
"X-B3-TraceId" "X-B3-SpanId"
"X-B3-ParentSpan"

73. Example: OpenTracing / PCF
Always log the TraceID as-is
Log calling SpanID as ParentSpan
Log new SpanID

74. Trace
Span
ParentSpan

75. Summary

76. Operability
making software work well
in Production

77. Run Book dialogue sheets
explore and establish
operational requirements as
a team, around a physical
table, together

78. lightweight user personas
explore the UX and needs of
different roles for rapid
decisions via dashboards

79. endpoint healthchecks
HTTP 200 / 500 responses to
/status/health call with
JSON details – good for
tools and humans

80. logging with Event IDs
use enum-based Event IDs to
explore runtime behaviour
and fault conditions

81. Correlation IDs
trace execution using
correlation IDs:
synchronous (HTTP X-trace-id)
async (SQS MessageAttribute)

82. Operability:
use these
team collaboration techniques
to make software work better

83. Team Guide to
Software Operability
Matthew Skelton & Rob Thatcher
skeltonthatcher.com/publications
Download a free sample chapter

84. Team Guide series
#Operability #BusinessMetrics #Testability #Releasability
skeltonthatcher.com/publications

85. Resources
• Team Guide to Software Operability by Matthew Skelton and Rob
Thatcher (Skelton Thatcher Publications, 2016)
http://operabilitybook.com/
• Run Book template & Run Book dialogue sheets
http://runbooktemplate.info/

86. Questions?
Twitter: @SkeltonThatcher | #operability
email: questions@skeltonthatcher.com

87. thank you
@SkeltonThatcher
skeltonthatcher.com