The document is a transcript from a presentation given by Joshua Corman and Gene Kim at a security conference in San Francisco in September 2012. The presentation discusses the problems with current security practices, introduces the concepts of DevOps and Rugged DevOps, and provides three ways ("systems thinking", "amplifying feedback loops", and "culture of continual experimentation") to implement Rugged DevOps practices to improve security. The overall message is that cultural and process changes are needed, not just technical fixes, to build more secure software.

1. SECURITY IS DEAD.
LONG LIVE RUGGED DEVOPS:
SEPTEMBER 12 – 14, 2012
IT AT LUDICROUS SPEED…
GRAND HYATT, SAN FRANCISCO
Joshua Corman
TRUTH, LIES Gene Kim
AND DECISIONS
Moving Forward in an Insecure World September 2012
Organized by

2. Gene Kim: Two Truths and a Lie
Please fill out the table below with two statements that are true and one lie
about yourself. I will put the information into the polling system to go live
before your presentation.
Statement Truth or lie?
I didn't know that Purdue University was in Indiana, Truth
otherwise I wouldn't have gone there
I still carry around a J. R. R. Tolkien book in my Lie
briefcase everywhere I go
I have an outrageous man-crush on my co-presenter, Truth
Josh Corman

3. About Joshua Corman
• Director of Security Intelligence for Akamai Technologies
- Former Research Director, Enterprise Security [The 451 Group]
- Former Principal Security Strategist [IBM ISS]
• Industry:
- Expert Faculty: The Institute for Applied Network Security (IANS)
- 2009 NetworkWorld Top 10 Tech People to Know
- Co-Founder of “Rugged Software” www.ruggedsoftware.org
- BLOG: www.cognitivedissidents.com
• Things I’ve been researching:
- Compliance vs Security
- Disruptive Security for Disruptive Innovations
- Chaotic Actors
- Espionage
- Security Metrics
3

4. Josh Corman: Two Truths and a Lie
Please fill out the table below with two statements that are true and one lie
about yourself. I will put the information into the polling system to go live
before your presentation.
Statement Truth or lie?
My philosophy thesis was entitled "Schizophrenic Truth
Alienated Tennis Pros in Love"
I'm the president of my local zombie survivalist Lie
chapter
I have a life sized statue of Spider-Man in my foyer Truth

5. About Gene Kim
• Researcher, Author
• Industry:
- Invented and founded Tripwire, CTO (1997-2010)
- Co-author: “Visible Ops Handbook”(2006), “Visible Ops Security” (2008)
- Co-author: “When IT Fails: The Novel,” “The DevOps Cookbook” (Coming May 2012)
• Things I’ve been researching:
- Benchmarked 1300+ IT organizations to test effectiveness of IT controls vs. IT
performance
- DevOps, Rugged DevOps
- Scoping PCI Cardholder Data Environment
5

6. PART 1: THE PROBLEM
SEPTEMBER 12 – 14, 2012
GRAND HYATT, SAN FRANCISCO
Joshua Corman
TRUTH, LIES Gene Kim
AND DECISIONS
Moving Forward in an Insecure World September 2012
Organized by

8. Consequences: Value & Replaceability
http://blog.cognitivedissidents.com/2011/10/24/a-replaceability-continuum/
8

9. You Don’t Need To Be Faster Than the Bear…
9

10. How will we rise?

12. DEPENDENCE
SEPTEMBER 12 – 14, 2012
Grand Hyatt, San Francisco Organized by

16. SOFTWARE
AS
VULNERABILITY
SEPTEMBER 12 – 14, 2012
Grand Hyatt, San Francisco Organized by

17. CONNECTED
AS
EXPOSED
SEPTEMBER 12 – 14, 2012
Grand Hyatt, San Francisco Organized by

18. OUR CHALLENGES ARE NOT
TECHNICAL,
BUT CULTURAL
SEPTEMBER 12 – 14, 2012
Grand Hyatt, San Francisco Organized by

19. WE CAN DO
BETTER
SEPTEMBER 12 – 14, 2012
Grand Hyatt, San Francisco Organized by

20. PART 2: DEVOPS
SEPTEMBER 12 – 14, 2012
GRAND HYATT, SAN FRANCISCO
Joshua Corman
TRUTH, LIES Gene Kim
AND DECISIONS
Moving Forward in an Insecure World September 2012
Organized by

21. Source: John Allspaw

22. Source: John Allspaw

25. Source: John Allspaw

26. Source: John Allspaw

27. Source: Theo Schlossnagle

28. Source: Theo Schlossnagle

29. Source: Theo Schlossnagle

30. Source: John Jenkins, Amazon.com

31. Ludicrous Speed?
31

32. Ludicrous Speed
32

34. Ludicrous Speed!
34

35. PART 3: RUGGED
SEPTEMBER 12 – 14, 2012
GRAND HYATT, SAN FRANCISCO
Joshua Corman
TRUTH, LIES Gene Kim
AND DECISIONS
Moving Forward in an Insecure World September 2012
Organized by

36. WHAT IS RUGGED?
SEPTEMBER 12 – 14, 2012
GRAND HYATT, SAN FRANCISCO
36 Organized by

37. WHAT IS RUGGED?
SEPTEMBER 12 – 14, 2012
GRAND HYATT, SAN FRANCISCO
37 Organized by

38. SEPTEMBER 12 – 14, 2012
GRAND HYATT, SAN FRANCISCO
TRUTH, LIES
AND DECISIONS
Moving Forward in an Insecure World
RUGGED SOFTWARE DEVELOPMENT
Joshua Corman, David Rice, Jeff Williams
2010
Organized by

41. RUGGED SOFTWARE

42. …so software not only needs to be…

43. FAST

44. AGILE

45. Are You Rugged?

46. HARSH

47. UNFRIENDLY

48. THE MANIFESTO
SEPTEMBER 12 – 14, 2012
GRAND HYATT, SAN FRANCISCO
Organized by

50. I recognize that my code will be used in ways I
cannot anticipate, in ways it was not designed,
and for longer than it was ever intended.

52. www.ruggedsoftware.org
https://www.ruggedsoftware.org/documents/
CrossTalk
http://www.crosstalkonline.org/issues/marchapril-2011.html

54. From the Rugged Handbook StrawMan

55. WHAT IS RUGGED DEVOPS?
SEPTEMBER 12 – 14, 2012
GRAND HYATT, SAN FRANCISCO
55 Organized by

56. Source: James Wickett

57. http://www.youtube.com/watch?v=JQEBYxp_vKs

59. Survival Guide/Pyramid
www.ruggedsoftware.org
Defensible Infrastructure

60. Survival Guide/Pyramid
Operational Discipline
Defensible Infrastructure

61. Survival Guide/Pyramid
Situational Awareness
Operational Discipline
Defensible Infrastructure

62. Survival Guide/Pyramid
Countermeasures
Situational Awareness
Operational Discipline
Defensible Infrastructure

63. Source: James Wickett

64. PART 4: ROCKING INFOSEC WITH
SEPTEMBER 12 – 14, 2012
RUGGED DEVOPS
GRAND HYATT, SAN FRANCISCO
Joshua Corman
TRUTH, LIES Gene Kim
AND DECISIONS
Moving Forward in an Insecure World September 2012
Organized by

65. The First Way:
Systems Thinking

66. The First Way:
Systems Thinking
(Business) (Customer)

67. The First Way:
Systems Thinking (Left To Right)
 Understand the flow of work
 Always seek to increase flow
 Never unconsciously pass defects downstream
 Never allow local optimization to cause global degradation
 Achieve profound understanding of the system

68. Create One Step Environment Creation Process
 Make environments available early in the Development
process
 Make sure Dev builds the code and environment at the same
time
 Create a common Dev, QA and Production environment
creation process

69. Embed Into Automated Infrastructure Team
 Get educated on open source tools like puppet and chef
 Provide them your hardening guidance
 Add your monitoring tools

70. Break Things Early And Often
 “Do painful things more frequently, so you can make it less
painful… We don’t get pushback from Dev, because they
know it makes rollouts smoother.”
-- Adrian Cockcroft, Architect, Netflix

71. Break Things Early And Often
 Enforce consistency in code, environments and configurations
across the environments
 Add your ASSERTs to find misconfigurations, enforce https,
etc.
 Add static code analysis to automated continuous integration
and testing process

72. The First Way:
Systems Thinking: Infosec Insurgency
 Have someone attend the daily Agile standups
• Gain awareness of what the team is working on
 Define what changes/deploys cannot be made without
triggering full retest

73. Definition: Kanban Board
 Signaling tool to reduce WIP and increase flow
73

74. The First Way:
Outcomes
 Determinism in the release process
 Creating single repository for code and environments
 Consistent Dev, QA, Int, and Staging environments, all
properly built before deployment begins
 Decreased cycle time
• Reduce deployment times from 6 hours to 45 minutes
• Refactor deployment process that had 1300+ steps spanning 4 weeks
 Faster release cadence

75. The Second Way:
Amplify Feedback Loops

76. The Second Way:
Amplify Feedback Loops (Right to Left)
 Understand and respond to the needs of all customers,
internal and external
 Shorten and amplify all feedback loops: stop the line when
necessary
 Create quality at the source
 Create and embed knowledge where we need it

77. “We found that when we woke up developers at 2am, defects
got fixed faster than ever”
-Patrick Lightbody,
CEO, BrowserMob

78. Phase 2: Extend Release Process And Create Right ->
Left Feedback Loops
 Invite Dev to post-mortems/root cause analysis meeting
 Have Dev and Infosec cross-train IT Operations
 Ensure application monitoring/metrics to aid in Ops and
Infosec work (e.g., incident/problem management)

79. The Second Way:
Amplify Feedback Loops: Infosec Insurgency
 Give production feedback to developers: being attacked is a gift
• Capture all instances of “UNION ALL” in user input and graph it, show it to
developers
• Show all instances of segfaults
 Create reusable Infosec use and abuse stories that can be added to every
project
• “Handle peak traffic of 4MM users and constant 4-6 Gb/sec Anonymous DDoS
attacks”
 Pre-enable, shield streamline successful audits
• Document separation of duty and compensating controls
• Don’t let them disrupt the work

80. The Second Way:
Outcomes
 Defects and security issues getting fixed faster than ever
 Reusable Ops and Infosec user stories now part of the Agile
process
 All groups communicating and coordinating better
 Everybody is getting more work done

81. The Third Way:
Culture Of Continual Experimentation And Learning

82. The Third Way:
Culture Of Continual Experimentation And Learning
 Foster a culture that rewards:
• Experimentation (taking risks) and learning from failure
• Repetition is the prerequisite to mastery
 Why?
• You need a culture that keeps pushing into the danger zone
• And have the habits that enable you to survive in the danger zone

83. “The best way to avoid failure is to fail constantly”

84. An Innovation Culture
“By installing a rampant innovation culture, they now do 165
experiments in the three months of tax season.
Our business result? Conversion rate of the website is up 50 percent.
Employee result? Everyone loves it, because now their ideas can make
it to market.”
--Scott Cook, Intuit Founder
85

85. You Don’t Choose Chaos Monkey…
Chaos Monkey Chooses You

86. Help Product Management…
Lesson: Allocate 20% of Dev cycles to paying down technical
debt

87. Phase 3: Organize Dev and Ops To Achieve
Organizational Goals
 Allocate 20% of Dev cycles to non-functional requirements
 Integrate fault injection and resilience into design,
development and production (e.g., Chaos Monkey)

88. The Third Way:
Culture Of Continual Experimentation And Learning:
Infosec
 Infosec remediation projects in the Agile backlog
• Make technical debt visible
• Help prioritize work against features and other non-functional
requirements
 Release your Chaos Monkey
• Evil/Fuzzy/Chaotic Monkey
• Eridicate SQLi and XSS defects in our lifetime
 Find processes that waste everyone’s time
 Eliminate needless complexity

89. The Third Way:
Outcomes
 Technical debt is being paid off
 Exploitable attack surface area decreases
 Continual reduction of unplanned work
 More cycles for planned work
 More resilient code and environments
 Balancing nimbleness and practiced repetition
 Enabling wider range of risk/reward balance

90. PART 5: WHY?
SEPTEMBER 12 – 14, 2012
GRAND HYATT, SAN FRANCISCO
Joshua Corman
TRUTH, LIES Gene Kim
AND DECISIONS
Moving Forward in an Insecure World September 2012
Organized by

94. When IT Fails: The Novel and The DevOps Cookbook
 Coming in July 2012
 “In the tradition of the best MBA case studies, this
book should be mandatory reading for business and
IT graduates alike.” -Paul Muller, VP Software
Marketing, Hewlett-Packard
 “The greatest IT management book of our
generation.” –Branden Williams, CTO Marketing, RSA

95. When IT Fails: The Novel and The DevOps Cookbook
 If you would like these slides, the “Top 10
Things You Need To Know About DevOps,”
Rugged DevOps resources, and updates on
the book:
Sign up at http://itrevolution.com
Email genek@realgenekim.me
Give me your business card

96. END
SEPTEMBER 12 – 14, 2012
GRAND HYATT, SAN FRANCISCO
Joshua Corman
TRUTH, LIES Gene Kim
AND DECISIONS
Moving Forward in an Insecure World September 2012
Organized by