The document summarizes lessons learned from implementing DevOps practices at Google and applying them to modernize practices in the U.S. federal government. It discusses how Google used transparency, autonomy, collaboration, and other principles to change culture and empower developers. It then outlines similar principles that could be applied in government to overcome inertia, ignorance, and other challenges through initiatives like a shared knowledge base, collaboration groups, and emphasizing learning over process. The overall message is that DevOps success depends on empowering individuals and embracing principles that have worked for open source and other organizations.

1. #B20CON
ITSM DEVOPS CONFERENCE
The Convergence of Wills
Mike Bland
Instigator
#B20CON

2. October 2013

3. November 2013

4. April 2014

5. So now what?

6. How I Learned To Stop Worrying and
Love the Bomb...Again

7. Google 2005

8. Inexperience

9. Code gets added.
Tools get slower.
Builds take longer.
Tests take forever.
Code goes untested.
Dependency cruft builds.

10. Large, infrequent changes frequently conflict.
Builds break overnight.
Emergency pushes common.
Fear is the mind killer.

12. InertiaEnormous early success
Overconfidence, arrogance,
Impostor Syndrome
Insecurity
Inexperience,
“My code is too hard to test”
Ignorance
Old tools,
“I don’t have time to test.”
Friction

13. (After-the-fact: goto fail; and Heartbleed)
Impact of testing is impossible to
measure a priori

14. Priority Structure
If it can’t be measured,
(e.g. more clicks)
it doesn’t matter.
(i.e. won’t get me promoted)
Ignorance/Communication Breakdown

15. How does culture change?

16. Not like this…

17. Or like this…
Beware of heroes, echo chambers

18. Cultivate mythology as a useful
model
The organization should
empower every individual

19. “If people believe they lack the power…”
If people believe they lack the power to
solve a problem,
Saul Alinsky, paraphrased from
Rules for Radicals
they won’t even think of
trying to solve it.
Knowledge and Power

20. What did we have to work with?

21. Transparency
Employee directory, project
database, wiki/Sites
Freedom to experiment,
20% time
Autonomy
Grouplet system,
startup ethos
Collaboration

22. Crossing the Chasm

23. GWS tech lead Bharat Mediratta believed
automated testing would help…
…and it did.

24. Started by Bharat
Mediratta and Nick
Lesiecki
Volunteers pooling
20% time
to drive adoption of
automated testing
Testing Grouplet

25. Testing on the Toilet (TotT)
Test Certified (TC)
Test Mercenaries
Ubiquitous,
incremental exposure
Clear, tangible path via
measurement, policy, goals
Hands-on help, tool
adoption and advocacy

26. Company-wide events, usually
one day long
Address “important but not
urgent” backlog
Focus, motivation, concrete
goals, free stuff
Fixits

27. Five years later…

28. Rainbow of Death: Testing Grouplet
Intervene Validate Inform Inspire EmpowerMentor
Dependent Independent
Fixits
Test
Certified
Build Orbs
Lectures
TotT
CodelabsTool development
(w/ Testing Tech,
Build Tools)
Test Mercenaries
Tech Talks Testing Grouplet
All projects
Test Certified
Level 3
Revolution Fixit
(build tools)
Test Certified
Mentors
TAP Fixit
(CI platform)

29. Google Stats 2015 via Rachel Potvin
15 million LoC in 250K files changed by humans
per week
15K commits by humans per day
30K commits by automated systems per day
800K/second peak file requests

30. Power and knowledge to do the right thing
Thorough automated testing now the norm
Most breakages fixed before clients notice
Less fear, more confidence, flow, and joy
The Value to Developers

31. David and Golaith

32. How could we be so sure
we were doing the right thing?

33. Business Value
Value is a hypothesis that an investment will
contribute to desired outcomes,
Mark Schwartz, paraphrased from The
Art of Business Value
discovered via experimentation,
by a team empowered to create
value.

34. Complex adaptive systems
Self-organizing under leadership’s influence
Elements, interconnections, purpose
Behavior > information flow >
interconnections
Changing the purpose changes the
system
and its behavior

35. Systems devised to shape human
behavior without accounting for human
nature are destined to fail.

36. What is innovation?
Ford: “...they’d’ve asked for a faster horse.”
Jobs: “1000 songs in your pocket!”
Jefferson: “...someone, with whom no one of
these ideas was original, combines all
together...”

37. U.S. federal government 2016

38. limiting perceived risk
meeting regulatory requirements
job security
Internalization: Don’t rock the boat
Priority Structure

39. Inertia
No quality incentives, PCSRA,
“successful company” people
Avoid risk/“accountability”,
“gov’t can’t attract talent”
Insecurity
Waterfall is familiar,
testing is someone else’s job
Ignorance
Outdated tools/procedures,
vendor lock-in of code, data
Friction

40. Policy often mandated by nontechnical people
Development teams disconnected from end users
They don’t know what they don’t know
Ignorance/Communication Breakdown

41. Employee directory
Code browser
Project data base
Wiki
EngEDU
Codelabs
First day at Google, August 29, 2005
Tech Talks
Snippets
Objectives and Key
Results
20% time
Grouplets

42. Where are the docs?
Who do I ask?
What do I need to know?
How do I get access to everything?
Who’s on my team?
Who’s working on what?
How can I contribute?
First day in government, November 3, 2014

43. Building a learning organization
The team that became great
didn’t start off great—it learned
how to produce extraordinary
results.
Peter Senge, from The Fifth Discipline

44. The Hub
Team API
.about.yml

45. Grouplets

46. Pages
Guides
Edu

47. Rainbow of Death: federal gov’t 2016
Intervene Validate Inform Inspire EmpowerMentor
Dependent Independent
Consulting
Success
stories on blog
Hub
Delivery
Discovery
sprints
Guides
Edu
Workshops
Useful
Mythology
Positive user
experiences
Digital Coalition
Working
Groups/Guilds
Onboarding
Revamp
Pages
Gov’t-wide
Hub
Cross-agency
collaboration
Team API

48. TransparencyThe Hub, Team API, .about.yml
Pages, Guides, Edu Autonomy
Grouplets, Open Source Collaboration

49. Nothing new under the sun
The Mythical Man-Month, 1975:
adding people increases complexity,
data speaks louder than flow charts
“No silver bullet”, 1986:
accidental vs. essential complexity;
foreshadowed the Agile movement by a decade, and
DevOps by two

50. Meet the new boss...
Old organizational habits die hard—even in new
organizations!
“Why Employees Stay”, Harvard Business Review,
July 1973!!!

51. The U.S.: the ultimate system
Life, liberty, and the pursuit of happiness
Checks and balances
Stable, yet amendable constitution
Bill of Rights
Protection from “tyrrany of the majority”

52. Lessons from Open Source
Eric S. Raymond,
“The Cathedral and the Bazaar”
Kropotkin:
“Severe effort of many converging wills”

54. America is a proven framework for the
convergence of wills

55. So is DevOps
Transparency
Autonomy
Collaboration
Systems thinking
Experimentation
Emergence

56. Don’t just check
the right boxes
Empower everyone to
do their best work
The organization isn’t as important as
the people in it

57. None More Black
https://mike-bland.com/
https://github.com/mbland/
Slides: https://goo.gl/DhVxis