Recommended
Recommended
More Related Content
Similar to Iterative Security: Secrets when you're not ready for Vault
Similar to Iterative Security: Secrets when you're not ready for Vault (20)
Recently uploaded
Recently uploaded (17)
Iterative Security: Secrets when you're not ready for Vault
- 1. tom@cloudzero.com @tmclaughbos Iterative Security: Secrets Management When You’re Not Ready For Vault tom@cloudzero.com @tmclaughbos
- 2. tom@cloudzero.com @tmclaughbos Who is this guy up here? His headshot is lasers and a cat
- 3. tom@cloudzero.com @tmclaughbos I’m Tom! • Community Engineer at CloudZero • Previously infrastructure engineer with a focus on automation.
- 4. tom@cloudzero.com @tmclaughbos Background & Biases Where I’m coming from
- 5. tom@cloudzero.com @tmclaughbos Background & Biases: I like startups
- 6. tom@cloudzero.com @tmclaughbos Background & Biases: I like startups
- 7. tom@cloudzero.com @tmclaughbos Background & Biases: Engineering is just a title
- 8. tom@cloudzero.com @tmclaughbos Does this look like you?
- 9. tom@cloudzero.com @tmclaughbos There is a lot of work to be done each day
- 10. tom@cloudzero.com @tmclaughbos For many of us, this ends up as reality…
- 11. tom@cloudzero.com @tmclaughbos We have all this technology! So why does these problems still exist?
- 12. tom@cloudzero.com @tmclaughbos How we present security
- 13. tom@cloudzero.com @tmclaughbos Security Paralysis I don’t know what to do
- 14. tom@cloudzero.com @tmclaughbos Security things you in ops might end up responsible for • Access controls • How much is too much access? • Password policies • How often should I force password rotation? • Wait, NIST has changed their recommendation? Don’t force rotation? • Patching • Do I patch immediately on every vendor release or test first? • and more…
- 16. tom@cloudzero.com @tmclaughbos I am not a “security person”
- 17. tom@cloudzero.com @tmclaughbos Iterative Security Starting and progressively improving your security stance
- 18. tom@cloudzero.com @tmclaughbos Why is security hard for us?
- 20. tom@cloudzero.com @tmclaughbos Things we get excited about with security
- 21. tom@cloudzero.com @tmclaughbos Things we get excited about with security 0-Days!!!
- 22. tom@cloudzero.com @tmclaughbos Things we get excited about with security Hash collisions
- 23. tom@cloudzero.com @tmclaughbos Things we get excited about with security What the CIA/NSA/DoD has
- 24. tom@cloudzero.com @tmclaughbos Things we get excited about in security Logos
- 25. tom@cloudzero.com @tmclaughbos Things we don’t get excited about with security
- 27. tom@cloudzero.com @tmclaughbos Not leaving MongoDB exposed to the internet with weak credentials
- 28. tom@cloudzero.com @tmclaughbos Not leaving Elasticsearch exposed to the internet with weak credentials
- 29. tom@cloudzero.com @tmclaughbos Learning from our mistakes
- 30. tom@cloudzero.com @tmclaughbos Many of us focus on the wrong things http://www.littlebobbycomic.com/projects/week-115/
- 31. tom@cloudzero.com @tmclaughbos How we present security
- 32. tom@cloudzero.com @tmclaughbos What we should be teaching • What are you trying to do? • Where do you start? • How do you progress
- 33. tom@cloudzero.com @tmclaughbos There’s a lot of info at the extremes
- 34. tom@cloudzero.com @tmclaughbos But this is where many of us are
- 35. tom@cloudzero.com @tmclaughbos We know not to… Put passwords, API keys, tokens, etc. in code.
- 36. tom@cloudzero.com @tmclaughbos But it still happens…
- 37. tom@cloudzero.com @tmclaughbos Where do you go from here?
- 38. tom@cloudzero.com @tmclaughbos Developing a threat model!
- 40. tom@cloudzero.com @tmclaughbos USB sticks are a bigger threat than the man in the ceiling
- 41. tom@cloudzero.com @tmclaughbos A breach probably won’t end business
- 42. tom@cloudzero.com @tmclaughbos What are we trying to protect? • Intellectual property • Customer data (data about who our customers are) • Customer’s data (data from our customers) • etc.
- 43. tom@cloudzero.com @tmclaughbos What does our architecture look like?
- 44. tom@cloudzero.com @tmclaughbos Decompose the system
- 45. tom@cloudzero.com @tmclaughbos Decompose system: Perimeters
- 46. tom@cloudzero.com @tmclaughbos Decompose system: Data pipeline
- 47. tom@cloudzero.com @tmclaughbos Decompose system: Data pipeline
- 48. tom@cloudzero.com @tmclaughbos Identify threats • Exposed network ports (network) • Unpatched EC2 instances (host) • Weak secrets management(application) • User submitted data (application) • etc.
- 49. tom@cloudzero.com @tmclaughbos Document Threats • Weak password management • At two points in our infrastructure we’re not managing passwords • Both points involve highly valuable assets • Breach would be bad • Reputation loss -> customer loss • Data could be leveraged against our customers
- 50. tom@cloudzero.com @tmclaughbos Rate Threats Risk = Probability * Damage Potential
- 51. tom@cloudzero.com @tmclaughbos Rate Threats • D: Damage potential • R: Reproducibility • E: Exploitability • A: Affected users • D: Discoverability https://en.wikipedia.org/wiki/DREAD_(risk_assessment_model)
- 52. tom@cloudzero.com @tmclaughbos Rate Threats • D: Disaster if this is found (high) • R: Easy to reproduce (high) • E: Easy to exploit; requires existing access (medium) • A: Affects all users (high) • D: Not easy to find; users are hops away from issue (medium)
- 53. tom@cloudzero.com @tmclaughbos Putting a response into action Let’s manage those secrets
- 57. tom@cloudzero.com @tmclaughbos Constraints: Risk of failure “It’s all code… We monitor it using Nagios.”
- 58. tom@cloudzero.com @tmclaughbos Constraints • Time: Few days to a few weeks • The faster we get this done the more likely we will finish. • Complexity: We’re going to go with what we know. • Less surprises. • Less to learn and get wrong. • Risk: Taking only as much risk as we’re ready for. • We’re moving fast! • Let’s limit the failure blast radius
- 59. tom@cloudzero.com @tmclaughbos Approaches to get you started Finally we secure some secrets
- 60. tom@cloudzero.com @tmclaughbos git-crypt • Encrypts secrets directly in your repository • Find secrets, rotate, and store them https://github.com/AGWA/git-crypt
- 61. tom@cloudzero.com @tmclaughbos git-crypt • Pro • You’ve done the exercise of auditing your code base • Cons • Symmetric encryption • Everyone needs the master password • TODO • Prevent key proliferation • Move to new secrets management when ready
- 62. tom@cloudzero.com @tmclaughbos Configuration Management: Puppet • Hiera-eyaml: Encrypt values in your Hiera hierarchy • Can use public key encryption • Multiple backends https://github.com/voxpupuli/hiera-eyaml
- 63. tom@cloudzero.com @tmclaughbos Configuration Management: Puppet • Pros • You’ve centralized your secrets in one repo. • Public key encryption support • Cons • May require manual intervention when rolling Puppetmasters. • May need to cleanup your Puppet code if you haven’t already moved to Hiera. • TODO: • Figure out master rekey strategy
- 64. tom@cloudzero.com @tmclaughbos Configuration Management: Ansible • Ansible Vault: Encrypts entire var files in playbook http://docs.ansible.com/ansible/playbooks_vault.html
- 65. tom@cloudzero.com @tmclaughbos Configuration Management: Ansible • Pros • You’ve done the exercise of auditing your code base • Cons • Symmetric encryption • Everyone needs the shared password • key proliferation • TODO • Preventing the proliferation of the Vault key • rekeying and rolling secrets.
- 66. tom@cloudzero.com @tmclaughbos S3 Buckets • Sneaker • Encrypt, store, and retrieve secrets from S3. https://github.com/codahale/sneaker
- 67. tom@cloudzero.com @tmclaughbos S3 Buckets • Pros • Secrets no longer live in repos • reduced secret proliferation • Secrets encrypted in S3. • Cons • How are you managing S3 buckets? • TODO • Manage your S3 buckets with CloudFormation, Terraform, etc. https://github.com/codahale/sneaker
- 68. tom@cloudzero.com @tmclaughbos What should we have gotten out of all this?
- 71. tom@cloudzero.com @tmclaughbos YOU CAN DO THIS!
- 74. tom@cloudzero.com @tmclaughbos Threat Modeling: startup edition https://twitter.com/CommitStrip/status/876830310780071936
- 75. tom@cloudzero.com @tmclaughbos Threat Modeling: startup edition response https://twitter.com/ErrataRob/status/876963608076439556
- 76. tom@cloudzero.com @tmclaughbos We know what not to do. We (think) we know where we want to be. But we don’t know how to get there.
Editor's Notes
- Iterative Security Secrets Management When You’re Not Ready For Vault (or Conjur) I want to emphasize the Iterative Security part I changed the focus of this presentation from what’s on the agenda as I wrote tgis, I realized secrets management wasn’t the actual interesting part. This is partially based on past expereinces in my career
- No new amazing tech in this presentation The technology isn’t the problem!!! We have good tech for managing secrets! Vault and Conjur exist. they’re really cool. they work well
- I HAVE OTHER WORK THAT’S MORE IMPORTANT What’s more important? How about keeping the site up? Been there. Secrets management systems can be complicated systems to understand and operate They take time to understand
- <SLIDE> laugh… But your friend next to you might be dealing with this A lot of us are or have been in situations we’re not proud of. We just don’t like to talk about this or admit it to others. I have been here previously in my career. Database passwords 3rd party service tokens. etc.
- This is how we present security to many people There is no in between There’s a lot of assumed knowledge from 1 to 2.
- We don’t have a clue what to do. We don’t know how to get from point A to point B and given the choice of 8 other priorities we choose what we feel we can accomplish. We’re rewarded for what we accomplishment Security isn’t always one of our KPIs.
- <SLIDE>
- With all those questions <SLIDE> * We’ve put this in your hands and now it’s up to you to figure this out.
- <SLIDE>
- <SLIDE>
- <SLIDE> This is a HUGE part of my role at Threat Stack as our Engineering Advocate Just listen find common problems address them secrets management that wasn’t as complicated and could be solved quick happened to be a topic. Posted this to reddit good response
- going to discuss “iterative security” Just want to get folks started securing their environment How do we get people taking care of the basics and gradually improving? Wide open security groups? What about S3 buckets? we can start with lower hanging fruit and work up
- <SLIDE>
- OMG security! Let’s get all excited! SHINY OBJECTS!
- What do people get excited about when talking security?
- 0-days Nobody expects them! I couldn’t find a good 0-day vulnerability logo But nobody expects the Spanish Inquisition… Really, you just wake up and roll into work and, “WTH is this?”
- Hash collisions (and other assorted cryptographic weaknesses) SHAttered: two different files that produce identical SHA1s Maybe someone will poison the Linux kernel repo
- The government can hack things It considers The Cyber to be a battlefield and is doing what other states are doing.
- * things with logos that everyone is talking about on Twitter/Slack/IRC
- * What don’t we get excited about?
- <SLIDE>
- <SLIDE> Same damn thing a week or two later.
- <SLIDE> MySQL was next after ES I did my part to warn the PostgreSQL community * This all took place in about 3 months
- Some of us in here legitimately need to focus on 0-days the logo of the month Many of us do not because we have not even done the basics we’re worried about APTs (advanced persistent threats) over our open databases.
- Back to the owl
- * What are you trying to look like? We talking headshot? side view frontal? Where do you start? How do you progress?
- You can find something that works on your machine You can find something that works at Big McLarge Co that has an Ops team many times bigger than yours.
- * Most of us are in the middle * There isn’t a lot out there on the left but just to the right of having gotten started solving our issues.
- <SLIDE>
- We’re going to apply the idea of iterating on security to Secrets Management
- <SLIDE>
- <SLIDE>
- * Let’s solve this like a real problem at work.
- * We’ll look at secrets management and impose some constraints on our decisions. In particular time How long will this work take? complexity How hard is this to do? How well can I understand it? failure risk What happens if the system fails? * These are normal business constraints we all face.
- Going to take months You should do the perfect thing! I have other things that need to get done! for most of us, security isn’t the only thing on our plate. I have news systems to deploy, systems to tune, systems to fix, tools to build… We’re judged at work by that and not security.
- Vault looks simple! How will you handle the master encryption key shards? Where they stored? Who has access? What storage backends meet your needs? How will you auth? * THANK YOU!
- * “It’s all code… We monitor it using Nagios.”
- Lets apply this! Secrets management if you’re not doing Vault (or Conjur)…
- <slide> Build, satisfy needs, and throw away later You were at the worst point possible, you’ve at least taken a step…
- <slide> reddit thread with author
- Just like git-crypt <slide>
- So, we’ll focus on less of this…. Sweet car! I’m sure it’s fast But It’s also on fire.
- We can manage this. roll in tomorrow and start putting out that garbage fiere
- <slide> None of the solutions I talked about are Earth shattering. *One of you, go make things better tomorrow. Makes changes to prod on friday and leave