tom@cloudzero.com @tmclaughbos
Iterative Security:โ€จ
Secrets Management
When Youโ€™re Not Ready For Vault
tom@cloudzero.com @tmclaughbos
tom@cloudzero.com @tmclaughbos
Who is this guy up here?
His headshot is lasers and a cat
tom@cloudzero.com @tmclaughbos
Iโ€™m Tom!
โ€ข Community Engineer at CloudZero
โ€ข Previously infrastructure engineer with
a focus on automation.
tom@cloudzero.com @tmclaughbos
Background & Biases
Where Iโ€™m coming from
tom@cloudzero.com @tmclaughbos
Background & Biases: I like startups
tom@cloudzero.com @tmclaughbos
Background & Biases: I like startups
tom@cloudzero.com @tmclaughbos
Background & Biases: Engineering is just a title
tom@cloudzero.com @tmclaughbos
Does this look like you?
tom@cloudzero.com @tmclaughbos
There is a lot of work to be done each day
tom@cloudzero.com @tmclaughbos
For many of us, this ends up as realityโ€ฆ
tom@cloudzero.com @tmclaughbos
We have all this technology!
So why does these problems still exist?
tom@cloudzero.com @tmclaughbos
How we present security
tom@cloudzero.com @tmclaughbos
Security Paralysis
I donโ€™t know what to do
tom@cloudzero.com @tmclaughbos
Security things you in ops might end up
responsible for
โ€ข Access controls
โ€ข How much is too much access?
โ€ข Password policies
โ€ข How often should I force password rotation?
โ€ข Wait, NIST has changed their recommendation? Donโ€™t force
rotation?
โ€ข Patching
โ€ข Do I patch immediately on every vendor release or test first?
โ€ข and moreโ€ฆ
tom@cloudzero.com @tmclaughbos
tom@cloudzero.com @tmclaughbos
I am not a โ€œsecurity personโ€
tom@cloudzero.com @tmclaughbos
Iterative Security
Starting and progressively improving your security stance
tom@cloudzero.com @tmclaughbos
Why is security hard for us?
tom@cloudzero.com @tmclaughbos
OMG SECURITY!!!
tom@cloudzero.com @tmclaughbos
Things we get excited
about with security
tom@cloudzero.com @tmclaughbos
Things we get excited about with security
0-Days!!!
tom@cloudzero.com @tmclaughbos
Things we get excited about with security
Hash collisions
tom@cloudzero.com @tmclaughbos
Things we get excited about with security
What the CIA/NSA/DoD has
tom@cloudzero.com @tmclaughbos
Things we get excited about in security
Logos
tom@cloudzero.com @tmclaughbos
Things we donโ€™t get excited
about with security
tom@cloudzero.com @tmclaughbos
Patching!
tom@cloudzero.com @tmclaughbos
Not leaving MongoDB exposed to the
internet with weak credentials
tom@cloudzero.com @tmclaughbos
Not leaving Elasticsearch exposed to
the internet with weak credentials
tom@cloudzero.com @tmclaughbos
Learning from our mistakes
tom@cloudzero.com @tmclaughbos
Many of us focus on the wrong things
http://www.littlebobbycomic.com/projects/week-115/
tom@cloudzero.com @tmclaughbos
How we present security
tom@cloudzero.com @tmclaughbos
What we should be
teaching
โ€ข What are you trying to do?
โ€ข Where do you start?
โ€ข How do you progress
tom@cloudzero.com @tmclaughbos
Thereโ€™s a lot of info at the extremes
tom@cloudzero.com @tmclaughbos
But this is where many of us are
tom@cloudzero.com @tmclaughbos
We know not toโ€ฆ
Put passwords, API keys, tokens,
etc. in code.
tom@cloudzero.com @tmclaughbos
But it still happensโ€ฆ
tom@cloudzero.com @tmclaughbos
Where do you go from here?
tom@cloudzero.com @tmclaughbos
Developing a threat model!
tom@cloudzero.com @tmclaughbos
Be Realistic
tom@cloudzero.com @tmclaughbos
USB sticks are a bigger threat than the man
in the ceiling
tom@cloudzero.com @tmclaughbos
A breach probably wonโ€™t end business
tom@cloudzero.com @tmclaughbos
What are we trying to protect?
โ€ข Intellectual property
โ€ข Customer data (data about who our customers are)
โ€ข Customerโ€™s data (data from our customers)
โ€ข etc.
tom@cloudzero.com @tmclaughbos
What does our architecture look like?
tom@cloudzero.com @tmclaughbos
Decompose the system
tom@cloudzero.com @tmclaughbos
Decompose system: Perimeters
tom@cloudzero.com @tmclaughbos
Decompose system: Data pipeline
tom@cloudzero.com @tmclaughbos
Decompose system: Data pipeline
tom@cloudzero.com @tmclaughbos
Identify threats
โ€ข Exposed network ports (network)
โ€ข Unpatched EC2 instances (host)
โ€ข Weak secrets management(application)
โ€ข User submitted data (application)
โ€ข etc.
tom@cloudzero.com @tmclaughbos
Document Threats
โ€ข Weak password management
โ€ข At two points in our infrastructure weโ€™re not managing
passwords
โ€ข Both points involve highly valuable assets
โ€ข Breach would be bad
โ€ข Reputation loss -> customer loss
โ€ข Data could be leveraged against our customers
tom@cloudzero.com @tmclaughbos
Rate Threats
Risk = Probability * Damage Potential
tom@cloudzero.com @tmclaughbos
Rate Threats
โ€ข D: Damage potential
โ€ข R: Reproducibility
โ€ข E: Exploitability
โ€ข A: Affected users
โ€ข D: Discoverability
https://en.wikipedia.org/wiki/DREAD_(risk_assessment_model)
tom@cloudzero.com @tmclaughbos
Rate Threats
โ€ข D: Disaster if this is found (high)
โ€ข R: Easy to reproduce (high)
โ€ข E: Easy to exploit; requires existing access (medium)
โ€ข A: Affects all users (high)
โ€ข D: Not easy to find; users are hops away from issue (medium)
tom@cloudzero.com @tmclaughbos
Putting a response into
action
Letโ€™s manage those secrets
tom@cloudzero.com @tmclaughbos
Constraints
Time Complexity Risk
tom@cloudzero.com @tmclaughbos
Constraints: Time
tom@cloudzero.com @tmclaughbos
Constraints: complexity
tom@cloudzero.com @tmclaughbos
Constraints: Risk of failure
โ€œItโ€™s all codeโ€ฆ We monitor it using Nagios.โ€
tom@cloudzero.com @tmclaughbos
Constraints
โ€ข Time: Few days to a few weeks
โ€ข The faster we get this done the more likely we will finish.
โ€ข Complexity: Weโ€™re going to go with what we know.
โ€ข Less surprises.
โ€ข Less to learn and get wrong.
โ€ข Risk: Taking only as much risk as weโ€™re ready for.
โ€ข Weโ€™re moving fast!
โ€ข Letโ€™s limit the failure blast radius
tom@cloudzero.com @tmclaughbos
Approaches to get you
started
Finally we secure some secrets
tom@cloudzero.com @tmclaughbos
git-crypt
โ€ข Encrypts secrets directly in your repository
โ€ข Find secrets, rotate, and store them
https://github.com/AGWA/git-crypt
tom@cloudzero.com @tmclaughbos
git-crypt
โ€ข Pro
โ€ข Youโ€™ve done the exercise of auditing your code base
โ€ข Cons
โ€ข Symmetric encryption
โ€ข Everyone needs the master password
โ€ข TODO
โ€ข Prevent key proliferation
โ€ข Move to new secrets management when ready
tom@cloudzero.com @tmclaughbos
Configuration Management: Puppet
โ€ข Hiera-eyaml: Encrypt values in your Hiera hierarchy
โ€ข Can use public key encryption
โ€ข Multiple backends
https://github.com/voxpupuli/hiera-eyaml
tom@cloudzero.com @tmclaughbos
Configuration Management: Puppet
โ€ข Pros
โ€ข Youโ€™ve centralized your secrets in one repo.
โ€ข Public key encryption support
โ€ข Cons
โ€ข May require manual intervention when rolling Puppetmasters.
โ€ข May need to cleanup your Puppet code if you havenโ€™t already
moved to Hiera.
โ€ข TODO:
โ€ข Figure out master rekey strategy
tom@cloudzero.com @tmclaughbos
Configuration Management: Ansible
โ€ข Ansible Vault: Encrypts entire var files in playbook
http://docs.ansible.com/ansible/playbooks_vault.html
tom@cloudzero.com @tmclaughbos
Configuration Management: Ansible
โ€ข Pros
โ€ข Youโ€™ve done the exercise of auditing your code base
โ€ข Cons
โ€ข Symmetric encryption
โ€ข Everyone needs the shared password
โ€ข key proliferation
โ€ข TODO
โ€ข Preventing the proliferation of the Vault key
โ€ข rekeying and rolling secrets.
tom@cloudzero.com @tmclaughbos
S3 Buckets
โ€ข Sneaker
โ€ข Encrypt, store, and retrieve secrets from S3.
https://github.com/codahale/sneaker
tom@cloudzero.com @tmclaughbos
S3 Buckets
โ€ข Pros
โ€ข Secrets no longer live in repos
โ€ข reduced secret proliferation
โ€ข Secrets encrypted in S3.
โ€ข Cons
โ€ข How are you managing S3 buckets?
โ€ข TODO
โ€ข Manage your S3 buckets with CloudFormation, Terraform, etc.
https://github.com/codahale/sneaker
tom@cloudzero.com @tmclaughbos
What should we have gotten
out of all this?
tom@cloudzero.com @tmclaughbos
Less thisโ€ฆ
tom@cloudzero.com @tmclaughbos
โ€ฆMore this!
tom@cloudzero.com @tmclaughbos
YOU CAN DO THIS!
tom@cloudzero.com @tmclaughbos
Thank You!
http://strayc.at/feedback
tom@cloudzero.com @tmclaughbos
tom@cloudzero.com @tmclaughbos
Threat Modeling: startup edition
https://twitter.com/CommitStrip/status/876830310780071936
tom@cloudzero.com @tmclaughbos
Threat Modeling: startup edition response
https://twitter.com/ErrataRob/status/876963608076439556
tom@cloudzero.com @tmclaughbos
We know what not to do.
We (think) we know where we want to be.
But we donโ€™t know how to get there.

Iterative Security: Secrets when you're not ready for Vault

Editor's Notes

  • #2ย Iterative Security Secrets Management When Youโ€™re Not Ready For Vault (or Conjur) I want to emphasize the Iterative Security part I changed the focus of this presentation from whatโ€™s on the agenda as I wrote tgis, I realized secrets management wasnโ€™t the actual interesting part. This is partially based on past expereinces in my career
  • #3ย No new amazing tech in this presentation The technology isnโ€™t the problem!!! We have good tech for managing secrets! Vault and Conjur exist. theyโ€™re really cool. they work well
  • #4ย I HAVE OTHER WORK THATโ€™S MORE IMPORTANT Whatโ€™s more important? How about keeping the site up? Been there. Secrets management systems can be complicated systems to understand and operate They take time to understand
  • #5ย <SLIDE> laughโ€ฆ But your friend next to you might be dealing with this A lot of us are or have been in situations weโ€™re not proud of. We just donโ€™t like to talk about this or admit it to others. I have been here previously in my career. Database passwords 3rd party service tokens. etc.
  • #7ย This is how we present security to many people There is no in between Thereโ€™s a lot of assumed knowledge from 1 to 2.
  • #8ย We donโ€™t have a clue what to do. We donโ€™t know how to get from point A to point B and given the choice of 8 other priorities we choose what we feel we can accomplish. Weโ€™re rewarded for what we accomplishment Security isnโ€™t always one of our KPIs.
  • #9ย <SLIDE>
  • #10ย With all those questions <SLIDE> * Weโ€™ve put this in your hands and now itโ€™s up to you to figure this out.
  • #11ย <SLIDE>
  • #12ย <SLIDE>
  • #13ย <SLIDE> This is a HUGE part of my role at Threat Stack as our Engineering Advocate Just listen find common problems address them secrets management that wasnโ€™t as complicated and could be solved quick happened to be a topic. Posted this to reddit good response
  • #14ย going to discuss โ€œiterative securityโ€ Just want to get folks started securing their environment How do we get people taking care of the basics and gradually improving? Wide open security groups? What about S3 buckets? we can start with lower hanging fruit and work up
  • #15ย <SLIDE>
  • #16ย OMG security! Letโ€™s get all excited! SHINY OBJECTS!
  • #17ย What do people get excited about when talking security?
  • #18ย 0-days Nobody expects them! I couldnโ€™t find a good 0-day vulnerability logo But nobody expects the Spanish Inquisitionโ€ฆ Really, you just wake up and roll into work and, โ€œWTH is this?โ€
  • #19ย Hash collisions (and other assorted cryptographic weaknesses) SHAttered: two different files that produce identical SHA1s Maybe someone will poison the Linux kernel repo
  • #20ย The government can hack things It considers The Cyber to be a battlefield and is doing what other states are doing.
  • #21ย * things with logos that everyone is talking about on Twitter/Slack/IRC
  • #22ย * What donโ€™t we get excited about?
  • #23ย <SLIDE>
  • #24ย <SLIDE> Same damn thing a week or two later.
  • #25ย <SLIDE> MySQL was next after ES I did my part to warn the PostgreSQL community * This all took place in about 3 months
  • #26ย Some of us in here legitimately need to focus on 0-days the logo of the month Many of us do not because we have not even done the basics weโ€™re worried about APTs (advanced persistent threats) over our open databases.
  • #27ย Back to the owl
  • #28ย * What are you trying to look like? We talking headshot? side view frontal? Where do you start? How do you progress?
  • #29ย You can find something that works on your machine You can find something that works at Big McLarge Co that has an Ops team many times bigger than yours.
  • #30ย * Most of us are in the middle * There isnโ€™t a lot out there on the left but just to the right of having gotten started solving our issues.
  • #31ย <SLIDE>
  • #32ย Weโ€™re going to apply the idea of iterating on security to Secrets Management
  • #33ย <SLIDE>
  • #34ย <SLIDE>
  • #35ย * Letโ€™s solve this like a real problem at work.
  • #36ย * Weโ€™ll look at secrets management and impose some constraints on our decisions. In particular time How long will this work take? complexity How hard is this to do? How well can I understand it? failure risk What happens if the system fails? * These are normal business constraints we all face.
  • #37ย Going to take months You should do the perfect thing! I have other things that need to get done! for most of us, security isnโ€™t the only thing on our plate. I have news systems to deploy, systems to tune, systems to fix, tools to buildโ€ฆ Weโ€™re judged at work by that and not security.
  • #38ย Vault looks simple! How will you handle the master encryption key shards? Where they stored? Who has access? What storage backends meet your needs? How will you auth? * THANK YOU!
  • #39ย * โ€œItโ€™s all codeโ€ฆ We monitor it using Nagios.โ€
  • #41ย Lets apply this! Secrets management if youโ€™re not doing Vault (or Conjur)โ€ฆ
  • #43ย <slide> Build, satisfy needs, and throw away later You were at the worst point possible, youโ€™ve at least taken a stepโ€ฆ
  • #45ย <slide> reddit thread with author
  • #47ย Just like git-crypt <slide>
  • #51ย So, weโ€™ll focus on less of thisโ€ฆ. Sweet car! Iโ€™m sure itโ€™s fast But Itโ€™s also on fire.
  • #52ย We can manage this. roll in tomorrow and start putting out that garbage fiere
  • #53ย <slide> None of the solutions I talked about are Earth shattering. *One of you, go make things better tomorrow. Makes changes to prod on friday and leave