Molly Crowther presents on how Pivotal Software implemented a security vulnerability reporting process modeled after the Toyota Production System's "andon cord". The process involved establishing an email address for security reports, triaging incoming reports, discussing issues with engineering teams to understand severity and fixes, and releasing CVEs once issues were resolved. This allowed security issues to be addressed without disrupting ongoing work. Over time, the process was refined through iterations and lessons learned.

1. The Red Button:
Adventures in Security Leadership
Molly Crowther
October 7–10, 2019
Austin Convention Center

2. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
About Me
Molly Crowther
2
Staff Technical Program Manager /
TPM + Docs Manager
● Cloud Foundry vulnerability
management
● Customer security assessments
● Security leadership team
● Security education

3. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
What I'm going to talk about
vs.
3

4. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Basics

5. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
What is an andon cord?
● Toyota Production System
● Lights up
● Figure out what's wrong
● Stops the line if the problem can't be fixed
quickly
● Line starts when the problem is resolved
5

6. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Well… it's actually more complicated
● Series of steps designed to increase psychological safety of calling out
defects
6

7. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
What is psychological safety
● Safety needed to take interpersonal risks (i.e. learn!)
● Google found it was the most important factor for high-performing teams (at
Google)
7

8. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
How do you increase psychological safety
● Demonstrate engagement
● Show understanding
● Be inclusive in interpersonal settings (includes
expressing gratitude)
● Be inclusive in decision making
● Show confidence and conviction without inflexibility
Source:
https://docs.google.com/document/d/1PsnDMS2emcPLgM
LFAQCXZjO7C4j2hJ7znOq_g2Zkjgk/export?format=pdf
8

9. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
What really happens
● Manager physically goes to the workstation
○
● Thanks the person who pulled the cord
○
● Talks to them to try to understand the problem ("How can I help you?")
○
● Line can start again when the problem is resolved, hopefully by the team
○
9

10. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
What really happens
● Manager physically goes to the workstation
○ Demonstrates engagement!
● Thanks the person who pulled the cord
○ Inclusive in interpersonal settings!
● Talks to them to try to understand the problem ("How can I help you?")
○ Shows understanding!
● Line can start again when the problem is resolved, hopefully by the team
○ Inclusive decision-making! Showing conviction!
1
0

11. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Another example of this? Concourse!
● Automated!
● Everyone around can see it
● Build turns red, testing stops
● Have to fix it before you can start
again
● Normalizes failure!
11

12. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Continuous improvement!
● Toyota actually started getting rid of andon cords in 2014!
● They replaced them with big yellow buttons
● Wanted to reduce overhead clutter
● Buttons are wireless so they can move them around and reconfigure the
workspace
● No cords to trip on
Source:
https://www.autonews.com/article/20140805/OEM01/140809892/toyota-cutting-th
e-fabled-andon-cord-symbol-of-toyota-way
1
2

13. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
What did we do?

14. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
What did we do in CF?
● We knew that we didn't know what all the issues were OR how we could find
out (super scary to leadership!)
● Needed the people closest to the information to tell us what was wrong
● Help everyone understand that we needed a psychologically safe security
environment
● We made our own andon cord (kind of)
1
4

15. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
History
● security@pivotal.io!
● First emails in March 2016
● First mention in slack was from June 2016
● First ticket in October 2016
● Triage team formed in December 2016 (shoutout to Rash and Anthony)
● Top 40 vendor on CVEdetails.com by 2017
● Still going strong after several PM changes and lots of engineering rotations
1
5

16. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Flow
1
6
Someone
emails
security@

17. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Flow
1
7
Someone
emails
security@
Automated
thank you

18. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Flow
1
8
Someone
emails
security@
Automated
thank you
Triage team
takes a look

19. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Flow
1
9
Someone
emails
security@
Automated
thank you
Triage team
takes a look
Might be irrelevant,
they respond and
close the ticket

20. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Flow
2
0
Someone
emails
security@
Automated
thank you
Triage team
takes a look
Might be irrelevant,
they respond and
close the ticket
Start a slack channel,
talk to the team

21. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Flow
2
1
Someone
emails
security@
Automated
thank you
Triage team
takes a look
Might be irrelevant,
they respond and
close the ticket
Start a slack channel,
talk to the team
Team helps
understand severity,
fixes the issue

22. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Flow
2
2
Someone
emails
security@
Automated
thank you
Triage team
takes a look
Might be irrelevant,
they respond and
close the ticket
Start a slack channel,
talk to the team
Team helps
understand severity,
fixes the issue
CVE goes out,
issue is closed

23. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
In hindsight...
● I didn't know it at the time but we were implementing the same practices as
Toyota.
○ Email to the security team could interrupt flow of what team was working
on
○ Automated thank you
○ Triage team would come to the team to understand the issue and
severity
○ Team would make the necessary changes
○ Work could continue on other features
2
3

24. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
What did we learn?

25. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
It's tough!
● Psychological safety doesn't come automatically or overnight
○ Baggage - other companies or experiences
○ Intentional - had to explain who we were and why we were asking
questions
○ Fear - people would reach out to me before sending
○ Overreactions - making people work on the weekend, upsetting the
community
2
5

26. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Iterations
● Engagement from leadership
● Started with an email + a few people, eventually moved to a ticketing system
and a dedicated team
● Few different types of classifications for security issues over time
● Timing of releases
● Team time zones
● Support embeds
● Closer partnership with infosec
● Davos!
2
6

27. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Shoutouts
LL - Alex Jackson
R&D Leadership - Onsi Fakhouri, Mike Dalessio
OG Triage pair - Rasheed Abdul-Aziz and Anthony Emengo
PMs - Steven Levine, Dan Jahner, Jake Knostman
Engineers
Justin Sampson
Margaret Crawford
Ramkumar Vengadakrishnan
Steven Locke
Tim Hausler
Adam Berkovec
Andrew Edstrom
Anna Thornton
Annie Sing
Anoop Gopalakrishnan
Aram Price
Ben Moss
2
7
Brian Vullo
Can Berk Güder
Charles Vieth
Christopher Brown
Clement Ng
Dan Jahner
Dhruv Sheth
Jesse Malone
Jim Myers
Joe Kirwin
Julien Cherry
Kalai Wei
Kris Hicks
Mark Fussell
Merric de Launey
Michelle He
Natasha Hull-Richter
Nayan Chandni
Phil Goodwin
Rob Dimsdale-Zucker
Slawek Ligus
Tony Wong
Weyman Fung
William Brandon

28. Thanks!
#springone@s1p

29. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Resources
● https://itrevolution.com/kata/
● https://www.leanvets.com/2018/05/andon-cords-and-practice.html
● https://rework.withgoogle.com/guides/understanding-team-effectiveness/ste
ps/foster-psychological-safety/
2
9

30. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Safe Harbor Statement
The following is intended to outline the general direction of Pivotal's offerings. It is intended for information
purposes only and may not be incorporated into any contract. Any information regarding pre-release of
Pivotal offerings, future updates or other planned modifications is subject to ongoing evaluation by Pivotal
and is subject to change. This information is provided without warranty or any kind, express or implied, and
is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making
purchasing decisions regarding Pivotal's offerings. These purchasing decisions should only be based on
features currently available. The development, release, and timing of any features or functionality described
for Pivotal's offerings in this presentation remain at the sole discretion of Pivotal. Pivotal has no obligation to
update forward looking information in this presentation.