Aaron Rinehart and DJ Schleen trade war stories about the implementation of DevSecOps practices at Global Healthcare Giants. Culture, Technique, and Technology.

2. 2@aaronrinehart @verica_io #chaosengineering
WASHINGTON, DC, JUNE 25th, 2020 -- This
morning while vacationing in Washington, City,
W. B. Earp, a Marshal from Tombstone, A. T.,
arrested a cowboy named Aaron “Mad Dog”
Rinehart. Rinehart, a former Chief Security
Architect at United Healthcare, and formerly a
renegade at DoD, and NASA is a frequent
speaker and author on Chaos Engineering. As a
pioneer behind Security Chaos Engineering
he’s authored Chaos Engineering, and Security
Chaos Engineering books for O’Reilly. After
being fined twenty five dollars and disarmed in
the Justice’s Court, Rinehart left, swearing
vengeance with his Chaos Slingr.
CHAOS LEADER APPREHENDED BY
VACATIONING MARSHALL

3. 3@djschleen @rally_health #deadpool
GOLDEN, CO, JUNE 25th, 2020 -- While on
his way to Glenwood Springs Bat Masterson
subdued DJ Schleen, while he was panning for
gold in Clear Creek. Schleen, a DevSecOps
pioneer and Application Security Manager at
Rally Health tried to evade capture with crafty
automation tactics. While on the run for the last
10 years, he’s been involved with the “DevOps
Crew” and has been preaching about automated
security at gatherings across the world and in
the books that he has written. After refusing to
pay the $10 fine, Schleen escaped using
automated security tools through engineering
pipelines.
ALLEGEDLY INSANE AUTOMATION
INNOVATOR CAPTURED IN COLORADO

7. A Grass Roots Beginning
• Teams across Silos & Disciplines w/ No Funding
• 60 Developers, Operations Engineers, and
Security Leaders from across the entire
company.• Began with Six Core DevOps Security Problem Sets
• Security Baseline + Configuration Validation w/
Chef & Inspec• Gauntlt Rugged Attack Framework
• Static Code Analysis (SAST): Automating
Fortify with Jenkins via API
• Application Vulnerability Scans(DAST):
Automating WebInspect with Jenkins via API
• DevOps Self-Governance & Operationalization
Framework: How does this world look from an
operational support perspective?
• Clair Container Image Scanning: Building Image
Scanning into Jenkins
Warm Regards, Aaron
The Journey
It all started when I was part of a startup
recently acquired by a massive healthcare
organization.
We needed to rapidly address problems with
new technologies and built our own
orchestration when no other tools were
available
We started with nothing and needed to steer
the boat towards AppSec as a practice.
Security had become more than GRC,
TPRG, IAM, SOC.
There were pockets of knowledge, no centers
of excellence in software security.
D.J. Schleen, ESQ

8. Eureka! Gold!
Successfully delivered an open source DAST tool
into CI/CD pipeline to drive instrumentation of
runtime security left in the delivery pipeline
Drove down delivery times of highly regulated
workloads by automating the verification of security
hardening configuration using InSpec
Delivered the ability for teams to initiate their own
DAST/SAST scanning via API
Adopted Commercial IAST solution. Took a very
long time to procure but saved the company millions
of dollars in efficiency per month.
Built empathy within the Security Organization by
adopting a Everyone Must Learn to Code learning
development
Warm Regards, Aaron
The Good Parts
Reduced overall vulnerabilities in our
code base
Educational programs (mentoring,
champions, etc) helped both developers
and security engineers understand the
challenges facing each other
Codifying automation improved
efficiency
Developers could react to vulnerabilities
and zero days faster than they could
without security in the mindset
D.J. Schleen, ESQ

9. Fools Gold!
Dagnabbit. We started with a
SAST program first - should
have started with OSS
Started with tools but should
have started with
relationships
We foolishly looked at
integration first before
knowing where the highest
risk application code was!
D.J. Schleen, ESQ
Can we have a do-over?
Focus on top down transformation
more. Bottom-Up was more
successful until we hit a point of
needing funding to go further
Spend more time helping to
transform flagship company
products. This sets the proper tone
for the rest of the enterprise.
Spend more time educating
security counterparts on the
business value of what
transforming
•
Warm Regards, Aaron

11. Image courtesy of Warner Bros.

12. A need for a Compass
Initially automating our existing
SAST/DAST scanning tools via API
caused the scanning infrastructure to
crash. The servers that supported it could
not withstand the volume.
Initially implementing Secrets
Management was difficult. Security
teams did not understand what software
secrets were. There was confusion
between Secrets and Privileged
Accounts.
Docker Container Image Scanning with
Clair didn’t meet needs
Warm Regards, Aaron
Hold yer Horses
We failed builds based on security
vulnerabilities before we helped
burn down vulnerabilities.
Doing this blocked production
deployments.
Blocked deployments meant
controls were taken out without
security knowledge.
We tried to move too quickly and
didn’t plan as much as we should
have
D.J. Schleen, ESQ

13. How to ride a Horse
Important Skills are Listening and Mutual Empathy
Show something Built is Better than an Idea
Fail small, fail fast
Its a culture shift, not just about automation
Continuous Learning is more important than
Continuous Fixing
Don't try to reduce complexity, learn to navigate it.
Avoid Analysis Paralysis: DevOps is a culture and
a living organism
DevOps is not a fad, it is the future
Automation is Important but “Don’t be Distracted
by it
Warm Regards, Aaron
Colorado Territory
If people aren’t on board, nobody cares.
You’re dealing with traditional security
organizations being assholes. That
shouldn’t be surprising.
Don’t invite people to your party if you
aren’t ready yet.
Know (or at least have a good idea)
where the highest risks are.
Look before you automate (look before
you cross the road)
Its the human fear of not being in control
that hinders automation.
D.J. Schleen, ESQ

15. Witchcraft
Tools haven’t caught up yet. We areusing flashlights for high mountedbrake lights and feathers when we
need airbags
Current Security Tooling SUCKS
We’ll see more innovation in thedetection of security issues
Tighter feedback loops for securityissues - fixing security issues withconfidence of break risk
DevSecOps becomes known as
“Engineering”
D.J. Schleen, ESQ
Snake Oil
The Next Generation of Security Professionals
will be Chosen from DevOps Teams
Shared Responsibility becomes more of a
reality.
Security continues the move towards value
stream
Security becomes a recognized skill within
Site Reliability Engineering (SRE)
Chaos Engineering becomes a core discipline
within DevSecOps
Compliance in DevSecOps becomes a
byproduct of good engineering practices
•
•
•
Warm Regards, Aaron

17. @djschleen @aaronrinehart
cutt.ly/verica-book
Free copy mailed to you complements of Verica