Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Practical Operability
Techniques for Teams
Matthew Skelton
Skelton Thatcher Consulting
skeltonthatcher.com / @SkeltonThatc...
Today
What is operability?
Modern logging
Run Book dialogue sheets
Endpoint healthchecks
Correlation IDs
User Personas for...
You
Software Developer
Tester / QA
DevOps Engineer
Team Leader
Head of Department
Operability:
use modern logging, Run Book
dialogue sheets, endpoint
healthchecks, correlation IDs,
and user personas as
te...
About me
Matthew Skelton
@matthewpskelton
Co-founder at
Skelton Thatcher Consulting
skeltonthatcher.com
Team Guide to
Software Operability
Matthew Skelton & Rob Thatcher
skeltonthatcher.com/publications
Download a free sample ...
Team Guide series
#Operability #BusinessMetrics #Testability #Releasability
skeltonthatcher.com/publications
Practical Operability
Techniques for Teams
What is operability?
Operability
making software work well
in Production
Logging with Event IDs
logging with Event IDs
reduce time to detect problems
increase team engagement
increase configurability
enhance collaborat...
search by
event
Event ID
{Delivered,
InTransit,
Arrived}
transaction
trace
Correlation ID
612999958…
How many distinct event
types (state transitions)
in your application?
represent distinct states
enum
Human-readable sets:
unique values, sparse,
immutable
C#, Java, Python, node
(Ruby, PHP, …)
Technical
Domain
public enum EventID
{
// Badly-initialised logging data
NotSet = 0,
// An unrecognised event has occurred...
BasketItemAdded = 60001
BasketItemRemoved = 60002
log using Event IDs
with a modern
‘structured logging’ library
example:
https://github.com/EqualExperts/opslogger
Sean Reilly
@seanjreilly
Example: video processing
On-demand processing of TV and
mobile streaming adverts
Ad-agency  TV broadcaster
High throughp...
Storage I/O
Worker Job
Queue
Upload
Example: video processing
Discover processing bottlenecks
Trigger alerts via LogEntries /
HostedGraphite
Report on KPIs
Ta...
http://honeycomb.tv/
Run Book dialogue sheets
Run Book dialogue sheets
Checklists for typical operational
considerations
Team-friendly exploration
runbooktemplate.infoRun Book dialogue sheets
System characteristics
Hours of operation
During what hours does the service or system actually need to operate? Can porti...
http://runbooktemplate.info/
runbooktemplate.infoRun Book dialogue sheets
Endpoint healthchecks
endpoint healthchecks
Every runnable app/service/daemon
exposes /status/health
An HTTP GET to the endpoint returns:
200 – ...
endpoint healthchecks
Each component is responsible for
determining its own health and
viability – this is very contextual
endpoint healthchecks
Use JSON as a response type –
parsable by both
machines and humans!
endpoint healthchecks
For databases and other non-HTTP
components, run a lightweight HTTP
service in front of the componen...
Helper service
https://github.com/Lugribossk/simple-dashboard
Correlation IDs
‘Unique-ish’ identifier for each request
Passed through downstream layers
Unique-ish ID
Synchronous HTTP:
X-HEADER e.g. X-trace-id
X-trace-id: 348e1cf8
If header is present, pass it on
(Yes, RFC6648, but this i...
Asynchonous (queues, etc.):
Message Attributes, name:value pair
e.g. "trace-id":"348e1cf8"
AWS SQS: SendMessage() / Receiv...
Example: electronic trading
High speed, low latency
Trading options & derivatives
Connected to stock exchanges
Sub-millise...
Correlations IDs for trading
Evidence for timely operation
Help identify bottlenecks
Target areas for perf tuning
Identify...
Example: OpenTracing / PCF
3 tracing elements:
TraceID, SpanID, ParentSpan
"X-B3-TraceId" "X-B3-SpanId"
"X-B3-ParentSpan"
Example: OpenTracing / PCF
Always log the TraceID as-is
Log calling SpanID as ParentSpan
Log new SpanID
Trace
Span
ParentSpan
Lightweight user personas
Lightweight user personas:
Ops Engineer
Test Engineer
Build & Deployment Engineer
Service Owner
Lightweight user personas:
Consider the User Experience (UX) of
engineers and team members using
and working with the soft...
http://www.keepitusable.com/blog/?tag=alan-cooper
https://www.geckoboard.com/blog/visualisation-upgrades-
progressing-towards-a-more-useful-and-beautiful-dashboard/
Lightweight user personas:
What data does the User Persona
need visible on a dashboard in order
to make decisions rapidly ...
Summary
Operability
making software work well
in Production
logging with Event IDs
use enum-based Event IDs to
explore runtime behaviour
and fault conditions
Run Book dialogue sheets
explore and establish
operational requirements as
a team, around a physical
table, together
endpoint healthchecks
HTTP 200 / 500 responses to
/status/health call with
JSON details – good for
tools and humans
Correlation IDs
trace execution using
correlation IDs:
synchronous (HTTP X-trace-id)
async (SQS MessageAttribute)
lightweight user personas
explore the UX and needs of
different roles for rapid
decisions via dashboards
Operability
use modern logging, Run Book
dialogue sheets, endpoint
healthchecks, correlation IDs,
and user personas as
tea...
Team Guide to
Software Operability
Matthew Skelton & Rob Thatcher
skeltonthatcher.com/publications
Download a free sample ...
Team Guide series
#Operability #BusinessMetrics #Testability #Releasability
skeltonthatcher.com/publications
Resources
• Team Guide to Software Operability by Matthew Skelton and Rob
Thatcher (Skelton Thatcher Publications, 2016)
h...
Questions?
Twitter: @SkeltonThatcher | #operability
email: questions@skeltonthatcher.com
thank you
@SkeltonThatcher
skeltonthatcher.com
Practical operability techniques for teams - Matthew Skelton - Agile in the City Bristol 2017
Practical operability techniques for teams - Matthew Skelton - Agile in the City Bristol 2017
Practical operability techniques for teams - Matthew Skelton - Agile in the City Bristol 2017
Practical operability techniques for teams - Matthew Skelton - Agile in the City Bristol 2017
Practical operability techniques for teams - Matthew Skelton - Agile in the City Bristol 2017
Practical operability techniques for teams - Matthew Skelton - Agile in the City Bristol 2017
Practical operability techniques for teams - Matthew Skelton - Agile in the City Bristol 2017
Practical operability techniques for teams - Matthew Skelton - Agile in the City Bristol 2017
Practical operability techniques for teams - Matthew Skelton - Agile in the City Bristol 2017
Practical operability techniques for teams - Matthew Skelton - Agile in the City Bristol 2017
Practical operability techniques for teams - Matthew Skelton - Agile in the City Bristol 2017
Practical operability techniques for teams - Matthew Skelton - Agile in the City Bristol 2017
Practical operability techniques for teams - Matthew Skelton - Agile in the City Bristol 2017
Practical operability techniques for teams - Matthew Skelton - Agile in the City Bristol 2017
Practical operability techniques for teams - Matthew Skelton - Agile in the City Bristol 2017
Practical operability techniques for teams - Matthew Skelton - Agile in the City Bristol 2017
Upcoming SlideShare
Loading in …5
×

Practical operability techniques for teams - Matthew Skelton - Agile in the City Bristol 2017

414 views

Published on

In this talk, Matthew Skelton (Skelton Thatcher Consulting) explores five practical, tried-and-tested, real-world techniques for improving operability with many kinds of software systems, including cloud, Serverless, on-premise, and IoT.

Logging as a live diagnostics vector with sparse event IDs
Operational checklists and 'run book dialogue sheets' as a discovery mechanism for teams
Endpoint healthchecks as a way to assess runtime dependencies and complexity
Correlation IDs beyond simple HTTP calls
Lightweight 'User Personas' as drivers for operational dashboards
These techniques work very differently with different technologies. For instance, an IoT device has limited storage, processing, and I/O, so generation and shipping of logs and metrics looks very different from the cloud or 'serverless' case. However, the principles - logging as a live diagnostics vector, event IDs for discovery, etc - work remarkably well across very different technologies.

From a talk at Agile in the City Bristol 2017 http://agileinthecity.net/2017/bristol/sessions/index.php?session=44

Published in: Software
  • Be the first to comment

Practical operability techniques for teams - Matthew Skelton - Agile in the City Bristol 2017

  1. 1. Practical Operability Techniques for Teams Matthew Skelton Skelton Thatcher Consulting skeltonthatcher.com / @SkeltonThatcher Agile in the City, Bristol 2017, London – 02 Nov 2017
  2. 2. Today What is operability? Modern logging Run Book dialogue sheets Endpoint healthchecks Correlation IDs User Personas for dashboards
  3. 3. You Software Developer Tester / QA DevOps Engineer Team Leader Head of Department
  4. 4. Operability: use modern logging, Run Book dialogue sheets, endpoint healthchecks, correlation IDs, and user personas as team collaboration techniques
  5. 5. About me Matthew Skelton @matthewpskelton Co-founder at Skelton Thatcher Consulting skeltonthatcher.com
  6. 6. Team Guide to Software Operability Matthew Skelton & Rob Thatcher skeltonthatcher.com/publications Download a free sample chapter
  7. 7. Team Guide series #Operability #BusinessMetrics #Testability #Releasability skeltonthatcher.com/publications
  8. 8. Practical Operability Techniques for Teams
  9. 9. What is operability?
  10. 10. Operability making software work well in Production
  11. 11. Logging with Event IDs
  12. 12. logging with Event IDs reduce time to detect problems increase team engagement increase configurability enhance collaboration #operability
  13. 13. search by event Event ID {Delivered, InTransit, Arrived}
  14. 14. transaction trace Correlation ID 612999958…
  15. 15. How many distinct event types (state transitions) in your application?
  16. 16. represent distinct states
  17. 17. enum Human-readable sets: unique values, sparse, immutable C#, Java, Python, node (Ruby, PHP, …)
  18. 18. Technical Domain public enum EventID { // Badly-initialised logging data NotSet = 0, // An unrecognised event has occurred UnexpectedError = 10000, ApplicationStarted = 20000, ApplicationShutdownNoticeReceived = 20001, MessageQueued = 40000, MessagePeeked = 40001, BasketItemAdded = 60001, BasketItemRemoved = 60002, CreditCardDetailsSubmitted = 70001, // ... }
  19. 19. BasketItemAdded = 60001 BasketItemRemoved = 60002
  20. 20. log using Event IDs with a modern ‘structured logging’ library
  21. 21. example: https://github.com/EqualExperts/opslogger Sean Reilly @seanjreilly
  22. 22. Example: video processing On-demand processing of TV and mobile streaming adverts Ad-agency  TV broadcaster High throughput Glitch-free video & audio
  23. 23. Storage I/O Worker Job Queue Upload
  24. 24. Example: video processing Discover processing bottlenecks Trigger alerts via LogEntries / HostedGraphite Report on KPIs Target areas for improvement
  25. 25. http://honeycomb.tv/
  26. 26. Run Book dialogue sheets
  27. 27. Run Book dialogue sheets Checklists for typical operational considerations Team-friendly exploration
  28. 28. runbooktemplate.infoRun Book dialogue sheets
  29. 29. System characteristics Hours of operation During what hours does the service or system actually need to operate? Can portions or features of the system be unavailable at times if needed? Hours of operation - core features (e.g. 03:00-01:00 GMT+0) Hours of operation - secondary features (e.g. 07:00-23:00 GMT+0) Data and processing flows How and where does data flow through the system? What controls or triggers data flows? (e.g. mobile requests / scheduled batch jobs / inbound IoT sensor data ) …
  30. 30. http://runbooktemplate.info/
  31. 31. runbooktemplate.infoRun Book dialogue sheets
  32. 32. Endpoint healthchecks
  33. 33. endpoint healthchecks Every runnable app/service/daemon exposes /status/health An HTTP GET to the endpoint returns: 200 – "I am healthy" 500 – "I am sick"
  34. 34. endpoint healthchecks Each component is responsible for determining its own health and viability – this is very contextual
  35. 35. endpoint healthchecks Use JSON as a response type – parsable by both machines and humans!
  36. 36. endpoint healthchecks For databases and other non-HTTP components, run a lightweight HTTP service in front of the component 200 / 500 responses
  37. 37. Helper service
  38. 38. https://github.com/Lugribossk/simple-dashboard
  39. 39. Correlation IDs
  40. 40. ‘Unique-ish’ identifier for each request Passed through downstream layers
  41. 41. Unique-ish ID
  42. 42. Synchronous HTTP: X-HEADER e.g. X-trace-id X-trace-id: 348e1cf8 If header is present, pass it on (Yes, RFC6648, but this is internal only)
  43. 43. Asynchonous (queues, etc.): Message Attributes, name:value pair e.g. "trace-id":"348e1cf8" AWS SQS: SendMessage() / ReceiveMessage() Log the Correlation ID if present
  44. 44. Example: electronic trading High speed, low latency Trading options & derivatives Connected to stock exchanges Sub-millisecond timings > £1 million per day traded
  45. 45. Correlations IDs for trading Evidence for timely operation Help identify bottlenecks Target areas for perf tuning Identify race conditions Increase operability
  46. 46. Example: OpenTracing / PCF 3 tracing elements: TraceID, SpanID, ParentSpan "X-B3-TraceId" "X-B3-SpanId" "X-B3-ParentSpan"
  47. 47. Example: OpenTracing / PCF Always log the TraceID as-is Log calling SpanID as ParentSpan Log new SpanID
  48. 48. Trace Span ParentSpan
  49. 49. Lightweight user personas
  50. 50. Lightweight user personas: Ops Engineer Test Engineer Build & Deployment Engineer Service Owner
  51. 51. Lightweight user personas: Consider the User Experience (UX) of engineers and team members using and working with the software
  52. 52. http://www.keepitusable.com/blog/?tag=alan-cooper
  53. 53. https://www.geckoboard.com/blog/visualisation-upgrades- progressing-towards-a-more-useful-and-beautiful-dashboard/
  54. 54. Lightweight user personas: What data does the User Persona need visible on a dashboard in order to make decisions rapidly & safely?
  55. 55. Summary
  56. 56. Operability making software work well in Production
  57. 57. logging with Event IDs use enum-based Event IDs to explore runtime behaviour and fault conditions
  58. 58. Run Book dialogue sheets explore and establish operational requirements as a team, around a physical table, together
  59. 59. endpoint healthchecks HTTP 200 / 500 responses to /status/health call with JSON details – good for tools and humans
  60. 60. Correlation IDs trace execution using correlation IDs: synchronous (HTTP X-trace-id) async (SQS MessageAttribute)
  61. 61. lightweight user personas explore the UX and needs of different roles for rapid decisions via dashboards
  62. 62. Operability use modern logging, Run Book dialogue sheets, endpoint healthchecks, correlation IDs, and user personas as team collaboration techniques
  63. 63. Team Guide to Software Operability Matthew Skelton & Rob Thatcher skeltonthatcher.com/publications Download a free sample chapter
  64. 64. Team Guide series #Operability #BusinessMetrics #Testability #Releasability skeltonthatcher.com/publications
  65. 65. Resources • Team Guide to Software Operability by Matthew Skelton and Rob Thatcher (Skelton Thatcher Publications, 2016) http://operabilitybook.com/ • Run Book template & Run Book dialogue sheets http://runbooktemplate.info/
  66. 66. Questions? Twitter: @SkeltonThatcher | #operability email: questions@skeltonthatcher.com
  67. 67. thank you @SkeltonThatcher skeltonthatcher.com

×