The document discusses DevOps practices and culture, highlighting Jason Hand's experience and insights along with various speakers from the VictorOps Roadtrip, such as Guillaume Binet and Sean Fitzgerald, who share their perspectives on incident management, continuous improvement, and the importance of collaboration in IT organizations. It references the 2014 State of DevOps report, which emphasizes that effective information processing is crucial for job satisfaction and, in turn, profitability. The document also outlines challenges in current practices and calls for better integration, automation, and communication to enhance overall operational efficiency.

2. JASON HAND |
DevOps Evangelist
• Holds over 15 years of experience as
a developer, system administrator,
and support specialist
• Fully emerged into the world of agile
development and the DevOps
movement with Colorado tech
startups
#DevOpsRoadTrip

3. #DevOpsRoadtrip
#DevOpsRoadTrip

6. A little about VictorOps…
VictorOps is the real-time incident
management platform that combines the
power of people and data to embolden
DevOps pros to handle incidents as they
occur.
#DevOpsRoadTrip

8. Why Are
We Here?

14. Culture

19. Culture

20. “How Organizations Process Information”
Roy Westrum: A Typology of Organizational Cultures
2014 State of DevOps Report shows that in the context of IT, job satisfaction is the biggest predictor of
profitability, market share, and productivity. The biggest predictor of job satisfaction, in turn, is how
effectively organizations process information, as determined by a model created by sociologist Ron
Westrum, shown below. 1
1: https://continuousdelivery.com/implementing/culture/

23. Words are how we think – stories are how we link.
- Christina Baldwin
Oral narrative is and for a long time has been the
chief basis of culture itself.
Stories from the road

26. Stable Systems

27. Cynefin

28. Un-ordered Ordered
Complicated
Obvious
Complex
Chaotic
Cause Effect Obvious
From Experience
Cause Effect Requires
Analysis
Cause Effect Only
Apparent in Hindsight
Cause & Effect Cannot
Be Related
Sense – Categorize -
Respond
Sense – Analyze -
Respond
Probe – Sense -
Respond
Act – Sense -
Respond

30. The systems we engineer, maintain, and improve are
Complicated
.. or ..
Known unknowns

31. The systems we engineer, maintain, and improve are
ComplexUnknown unknowns

33. What is the
Root
Cause?

34. What are the..
Contributing
Factors?

35. Identifying a “root cause” helps us to …
Put it back
how it was

36. What we really want is to..
Continuously
Improve

37. TimeToRepair(TTR)
Continuous Improvement Efforts
Reactive
(chaotic)
Tactical
(obvious)
Integrated
(complicated)
Strategic
(complex)
✓ No automation
✓ No operational stack
awareness
✓ Poor collaboration between
teams (Dev & Ops)
✓ Documentation not available
✓ No standardized
communication
✓ High focus on consistent
continuous learning ✓ Uses a NOC
✓ Some monitoring & alerting
instrumentation
✓ Collaboration in crisis
✓ "Mission critical" processes are
available
✓ Understood crisis
communication protocols
✓ Remediation data available to
IT Operations ✓ Team rotations, paging
policies, role hunting
✓ Continuous improvement of
key health indicators
✓ Technical collaboration across
all incidents
✓ Docs up to date and easily
accessible
✓ Consistent real-time
communication practices
✓ Automated docs and remediation
✓ Actionable Alerts with full context
✓ High collaboration among all teams
✓ Documentation part of remediation
✓ Targeted, proactive crisis comms
✓ High focus on continuous learning
Incident Management
Maturity

38. Reactive
(chaotic)
✓No automation
✓No operational stack awareness
✓Poor collaboration between teams (Dev & Ops)
✓Documentation not available
✓No standardized communication
✓High focus on consistent continuous learning

39. Tactical
(obvious)
✓Uses a NOC
✓Some monitoring & alerting instrumentation
✓Collaboration in crisis
✓"Mission critical" processes are available
✓Understood crisis communication protocols
✓Remediation data available to IT Operations

40. Integrated
(complicated)
✓Team rotations, paging policies, role hunting
✓Continuous improvement of key health indicators
✓Technical collaboration across all incidents
✓Docs up to date and easily accessible
✓Consistent real-time communication practices

41. Strategic
(complex)
✓Automated docs and remediation
✓Actionable Alerts with full context
✓High collaboration among all teams
✓Documentation part of remediation
✓Targeted, proactive crisis comms
✓High focus on continuous learning

42. “Six Trends Shape DevOps Adoption, Q1 2015”
Forrester report
• The Foundation For Success Is In Place . . . Mostly
• Fear Of Failure Will Hamper Advancement
• Monitoring And Analytics Strategies Must Make A Big Leap Forward
• The Focus On Customer Experience Is Not Second Nature . . . Yet
• Change And Release Processes Are Not Delivering Business Needs
• You Must Prioritize And Focus Sourcing Strategies

43. Automation
Awareness
Collaboration
Documentation User Empathy
Learning

44. Learning

45. Failure not seen as opportunity to learn
Source: “Six Trends Shape DevOps Adoption, Q1 2015”, Forrester report

46. Awareness
http://blog.vmware.com

47. © 2015 Forrester Research, Inc. Reproduction Prohibited 47
Single Source Of Truth Lacking In Many Orgs
– 95% only most of the time or less
Source: April 15, 2015 “Six Trends That Will Shape DevOps Adoption”, Forrester report

48. Collaboration

49. Teams siloed throughout life cycle
Source: “Six Trends Shape DevOps Adoption, Q1 2015”, Forrester report

50. User Empathy
https://open.buffer.com/wp-content/uploads/2015/12/empathy3.jpg

51. © 2015 Forrester Research, Inc. Reproduction Prohibited 51
IT teams aren’t measured on customer
experience goals.

52. Automation
http://thelifedesignproject.com/wp-content/uploads/2009/09/373881476_217d24ef6d.jpg

54. Delays in notifications Leads To Customers
Finding the Problem First
Source: “Six Trends Shape DevOps Adoption, Q1 2015”, Forrester report

55. Documentation
http://blog.vmware.com

56. Reduce MTTR
State of DevOps Report (2015)
– by Puppet Labs

57. Automation
Awareness
Collaboration
Documentation User Empathy
Learning

58. jhand.co/DRT_SF

65. GUILLAUME BINET |
TECH LEAD, SENIOR SOFTWARE ENGINEER
• At Google since 2013, and is working on the Signals team for
Google Cloud.
• Previously held responsibilities at startups in Europe and the
US, including an API Tech Lead at Twilio
• Author of Errbot (http://errbot.io)
• When note working or hacking on Errbot, you can find him car
racing, scuba diving or tinkering on his latest electronic project
#DevOpsRoadTrip

66. Chatops ?
Guillaume Binet
2016 VictorOps Devops Roadtrip SFO
Hosted
by

67. Chat + DevOps
67

68. w00t! let’s roll !
Build #423 started...
This is getting exciting ! The press release is ready.
Build #423 finished. Triggering tests on image avengers-v2-
423.
avengers-v2-423: all tests are green.
!build avengers-v2
ok, I just pushed the latest touches on branch avengers-v2
68

69. $ nohup python r2d2.py &

70. Sprinting

71. Dependency graph
>!issue deps 1284
1284 - User Story
App skins
Assignee: gbin
1285 - Feature
Skin dls
Assignee: gbin
1289 - Feature
Android skins
Assignee: stevo
1287 - Feature
Skin console
Assignee: gbin

72. Building &
Deploying

73. Continuous integration
Mars-4.4.123 passed all tests.

74. Releases
>!qa 4.4.123
Deploying Mars-4.4.123 on qa...

75. Monitoring

76. Alerts
CRITICAL: On prod, trebuchet-3
is at 80% CPU.

77. End to end test
Test call failed 3 times in a
row on prod.

78. Fun !

79. >!facepalm
>!tourney
>!devops borat
>!ask gbin
[...]

80. Non stop
breakages

81. "This thing is
critical!"
O. Dauby, Ops Manager

82. http://errbot.io

83. Easy to start with ...
83
Hot changes
resilient
Chatops
on
Chatops
Accessible to non-
programmers

84. … but very powerful !
84
Conversations
Automation
Filter /
security
Persistence
Templates Provisioning

85. Make it your
own !

86. http://errbot.io
Thanks !

88. SEAN FITZGERALD |
SOFTWARE ENGINEER, SNAPCHAT
• (seanfitz) is a Software Engineer focused on infrastructure at
Snapchat.
• Career ranging from startups as small 5 to a tour of duty at
Amazon.
#DevOpsRoadTrip

89. Devops at Snapchat
VictorOps Roadtrip SF, 2016

90. Ownership

91. Writing Code

92. Deploying Code

93. Monitoring Code

94. Finding Root Cause

95. AARON MERRILL |
NOC MANAGER, SONY INTERACTIVE ENTERTAINMENT – PLAYSTATION
• Process leader and evangelist with 20 years worth of
experience leveraging technical expertise to improve
operations efficiency and reliability, while reducing operational
costs
• Working with Fortune 500 companies including NewsCorp,
Fidelity, Sempra Energy and Sony
• Effective in reducing time-to-resolution of priority incidents by
double digit percentages through enhanced technologies and
improved process design.
#DevOpsRoadTrip

96. 96
24X7 Monitoring
Around the clock monitoring
ensuring the availability of
IT services supporting the
advancement of Sony
Interactive Entertainment -
WWS
Service Restoration
Expert outage triage and
high impacting incident
identification leading to
rapid action and resolution
Notification
Timely notification to our
WWS clients and
assembling the appropriate
resources to drive
resolutions as quickly as
possible
Problem Management
Driving the documentation
and root cause analysis on
all high impacting incidents -
ensuring resolution and
proactive prevention of
future events
Maintenance
Providing skilled, flexible,
client focused first-level
game and IT service
maintenance 24 hours a day
Reporting
Consistent accurate
reporting on incident and
performance metrics
Detailed post-mortem
documentation on all high
impacting outages

97. ❖ Increasing cost of toolsets
❖ Time consuming activities that lend themselves to automation
❖ Need for better integration with current monitoring systems
❖ Better context around alerts from disparate systems
❖ Better reporting - MTTA, MTTR, and incident event chronologies
❖ Managing incident notification subscriptions
❖ Faster means of communicating during incident
CHALLENGES

98. ❖ Re-routing incidents to one or more teams
❖ Automated ticket creation
❖ Ability to quiet un-actionable or false alarms
❖ IT communication during incident (chat, automated conference
calling, Slack integration)
❖ Providing better contextual information (triage docs &
runbooks) along with alerts to quickly solve the incidents
❖ Real-time on-call handoffs and schedule overrides
❖ Continuous incident documentation for post-mortem reporting
for RCA creation
❖ Reporting on MTTA, MTTR, and incident event chronologies
SOLUTION

99. 99
SOLUTION - INTEGRATION

100. ❖ Integrated with VictorOps
❖ Self service subscription based communications (SMS /
Email)
❖ Scheduled maintenance with reminders
❖ Automatically displays the status of Infrastructure providers
(Box, Amazon, etc.)
❖ Provide users incident post mortems reports
❖ Integrated translation services
SOLUTION

101. 101
SOLUTION

102. ❖ Enhanced real-time communications to our internal clients
❖ Further tool integrations and reporting
❖ Providing enhanced contextual information along with
alerts to quickly solve the incidents at hand
❖ Correlation of change, incidents and problems
❖ Enhanced knowledge articles and runbooks
❖ Translation in Slack channels for real-time communication
Ongoing improvements

103. Questions

105. Q&A

106. BREAK TIME
#DevOpsRoadTrip

107. Breakout Sessions
◻ ChatOps - Guillaume Binet, Tech Lead/Senior Software Engineer, Google
◻ Security & Compliance in a DevOps World - J. Paul Reed, DevOps Consultant
◻ Finding Signal in the Noise – Aneel Lakhani, SignalFX
◻ DevOps Culture and Burnout – Ken Mugrage, Tech Evangelist, Thoughtworks
◻ Straddling Deployment and Operations – Aaron Merrill, NOC Manager
◻ DevOps Unlocked – Social Contracts and Code – Sean Fitzgerald, Software Engineer at
Snapchat
#DevOpsRoadTrip

108. KEN MUGRAGE |
TECHNICAL EVANGELIST, THOUGHTWORKS
• 25 years experience in IT
• Focused on Continuous Delivery and DevOps for much of the
past decade.
• Worked with organizations all over the world, ranging from
startups to Fortune 500 companies.
• Passionate about helping others get better at building, testing
and deploying software and using technology to increase
business effectiveness, as opposed to using the “latest cool
thing”
#DevOpsRoadTrip

109. ANEEL LAKHANI |
Signal FX
• In technology full-time since high school – from startups to
consulting to teaching to big tech companies to analyst-ing and
back to startups.
• Former research director at Gartner
#DevOpsRoadTrip

110. J. PAUL REED |
DEVOPS CONSULTANT
• Over a decade of experience in the trenches as a build/release and tools
engineer, working with such organizations as VMware, Mozilla, and
Symantec.
• In 2012, he founded Release Engineering Approaches, a consultancy
incorporating a host of tools and techniques to help organizations “Simply
Ship. Every time.”
• Worked with organizations across a number of industries, from financial
services to cloud-based infrastructure, with teams from 2 to 200.
• Paul is also a founding host of The Ship Show, a twice-monthly podcast
tackling topics related to build engineering, DevOps, and release
management.
#DevOpsRoadTrip

111. Breakout Sessions
◻ ChatOps - Guillaume Binet, Tech Lead/Senior Software Engineer, Google
◻ Security & Compliance in a DevOps World - J. Paul Reed, DevOps Consultant
◻ Finding Signal in the Noise – Aneel Lakhani, SignalFX
◻ DevOps Culture and Burnout – Ken Mugrage, Tech Evangelist, Thoughtworks
◻ Straddling Deployment and Operations – Aaron Merrill, NOC Manager
◻ DevOps Unlocked – Social Contracts and Code – Sean Fitzgerald, Software Engineer at
Snapchat
#DevOpsRoadTrip

112. BREAK TIME
#DevOpsRoadTrip

113. J. PAUL REED |
DEVOPS CONSULTANT
• Over a decade of experience in the trenches as a build/release and tools
engineer, working with such organizations as VMware, Mozilla, and
Symantec.
• In 2012, he founded Release Engineering Approaches, a consultancy
incorporating a host of tools and techniques to help organizations “Simply
Ship. Every time.”
• Worked with organizations across a number of industries, from financial
services to cloud-based infrastructure, with teams from 2 to 200.
• Paul is also a founding host of The Ship Show, a twice-monthly podcast
tackling topics related to build engineering, DevOps, and release
management.
#DevOpsRoadTrip

114. Q&A

116. How did you
Score?

117. How Organizations Process Information
Roy Westrum: A Typology of Organizational Cultures
2014 State of DevOps Report shows that in the context of IT, job satisfaction is the biggest predictor of
profitability, market share, and productivity. The biggest predictor of job satisfaction, in turn, is how
effectively organizations process information, as determined by a model created by sociologist Ron
Westrum, shown below. 1
1: https://continuousdelivery.com/implementing/culture/

118. TimeToRepair(TTR)
Continuous Improvement Efforts
Reactive (0 – 4)
(chaotic)
Tactical (5 – 9)
(obvious)
Integrated (10 -14)
(complicated)
Strategic (15 – 18)
(complex)
✓ No automation
✓ No operational stack
awareness
✓ Poor collaboration between
teams (Dev & Ops)
✓ Documentation not available
✓ No standardized
communication
✓ High focus on consistent
continuous learning ✓ Uses a NOC
✓ Some monitoring & alerting
instrumentation
✓ Collaboration in crisis
✓ "Mission critical" processes are
available
✓ Understood crisis
communication protocols
✓ Remediation data available to
IT Operations ✓ Team rotations, paging
policies, role hunting
✓ Continuous improvement of
key health indicators
✓ Technical collaboration across
all incidents
✓ Docs up to date and easily
accessible
✓ Consistent real-time
communication practices
✓ Automated docs and remediation
✓ Actionable Alerts with full context
✓ High collaboration among all teams
✓ Documentation part of remediation
✓ Targeted, proactive crisis comms
✓ High focus on continuous learning
Incident Management Maturity

119. TO POPULATE WITH REAL-TIME RESULTS

120. RAFFLE TIME
#DevOpsRoadTrip

121. Join us for Happy Hour
Co-Sponsored by
• Your nametag is your ticket
• Down the elevator, turn towards Spear Street, and you’ll see it
on your right.
• One Market Street Restaurant
#DevOpsRoadTrip

122. DENVER - SEATTLE - SAN FRANCISCO - MINNEAPOLIS - NEW YORK CITY