Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

DevSecCon Boston 2018: My rage quit journey: configuring Netflix tools by Sarah Young

661 views

Published on

DevSecCon Boston 2018: My rage quit journey: configuring Netflix tools by Sarah Young

Published in: Technology
  • Positions Available Now! We currently have several openings for social media workers.  https://tinyurl.com/rbrfd6j
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

DevSecCon Boston 2018: My rage quit journey: configuring Netflix tools by Sarah Young

  1. 1. BOSTON 10-11 SEPT 2018 My ragequit journey: configuring Netflix tools SARAH YOUNG BOSTON 10-11 SEPT 2018 My ragequit journey: configuring Netflix tools SARAH YOUNG
  2. 2. BOSTON 10-11 SEPT 2018
  3. 3. BOSTON 10-11 SEPT 2018 whoami • Sarah Young, Security Architect at Versent. • I’m from Melbourne in Australia. • I help customers move their stuff into the cloud securely. • Worked in tech for the past 9ish years. • I’ve worked in Europe, New Zealand and Australia. • I overuse memes and GIFs. • Wannabe crazy bird lady.
  4. 4. BOSTON 10-11 SEPT 2018 If anyone knows Justin Trudeau, please let me know.
  5. 5. BOSTON 10-11 SEPT 2018 I am not a Christian author
  6. 6. BOSTON 10-11 SEPT 2018 Firstly… • This talk is not an attack on Netflix. • I love Netflix as both an end user of their service and a consumer of their SecOps tools. • Alas, I am also not on commission from Netflix. • The aim of this talk is to demonstrate how everyone struggles with tools from time-to-time. • I want to try to reduce “FOFU”, “fear of F!%*ing up”.
  7. 7. BOSTON 10-11 SEPT 2018 Intro to Netflix tools • I don’t have to introduce Netflix… I hope?! • Netflix have been releasing Open Source tools since 2014. • They release numerous types of tools: • Big data • Content encoding • Insight, reliability and performance monitoring • … and much more • I’m going to focus on some of their security tools.
  8. 8. BOSTON 10-11 SEPT 2018 Just one more note… • I’m aware that there are talks at other conferences and meetups where companies and individuals talk about successful implementations of these tools. • This is not one of those talks. • I will link to some of the happier Hollywood stories at the end of talk.
  9. 9. BOSTON 10-11 SEPT 2018 Tools I’m going to look at • BLESS (Bastion's Lambda Ephemeral SSH Service) • Security Monkey • Repokid
  10. 10. BOSTON 10-11 SEPT 2018 The beginning of the journey… • I was equipped with: • Git Readmes. • My work’s sandbox AWS account. • Google. • Slightly rusty Linux skills. • Unlimited cans of fizzy drinks from the fridge. • My patience.
  11. 11. BOSTON 10-11 SEPT 2018 Don’t test the demo gods
  12. 12. BOSTON 10-11 SEPT 2018 BLESS – Qué? • BLESS stands for Bastion's Lambda Ephemeral SSH Service. • It’s an Internal Certificate Authority. • Inside a Lambda function. • Issues short-lived certificates for EC2 access. • Certificates have 120 seconds validity by default.
  13. 13. BOSTON 10-11 SEPT 2018 BLESS – awscli is not my friend • Create an AWS role, easy. • Maybe my Python version is too new for awscli? • Let’s uninstall Python3.
  14. 14. BOSTON 10-11 SEPT 2018 BLESS – saml2aws
  15. 15. BOSTON 10-11 SEPT 2018 Firstly… • Cue lengthy Slack discussion about how Brew/Python/awscli suck. • Let’s just reinstall awscli.
  16. 16. BOSTON 10-11 SEPT 2018 BLESS – Virtual-env is additionally not my friend • False start, let’s go now. • Have to force install virtual-env. • I’m using Docker. • All goes well here.
  17. 17. BOSTON 10-11 SEPT 2018 BLESS – Certificates, KMS and Lambda are dope • Generate certs just fine. • Make keys in KMS just fine. • Make Lambda function just fine. • Things are going too well… surely?! Accurate depiction of me at this point
  18. 18. BOSTON 10-11 SEPT 2018 BLESS – OSX, you make my life hard • BLESS should be finished. • Now to test it. • I don’t have Boto3… • … except I do. • Dammit Python dependencies!
  19. 19. BOSTON 10-11 SEPT 2018 BLESS – Dammit Python Credit: XKCD
  20. 20. BOSTON 10-11 SEPT 2018 BLESS – Dammit again Python
  21. 21. BOSTON 10-11 SEPT 2018 Sidenote
  22. 22. BOSTON 10-11 SEPT 2018 BLESS – What’s the first rule of security…? • I don’t have creds (apparently). • Turns out this is a bug in saml2aws. • I should have updated to 2.7.0 before I started. I deserve Trump shame for this fail.
  23. 23. BOSTON 10-11 SEPT 2018 BLESS – don’t do this
  24. 24. BOSTON 10-11 SEPT 2018 BLESS – do this
  25. 25. BOSTON 10-11 SEPT 2018 BLESS – Real-life issues • Very little guidance on how to scale BLESS. • “Deploy an Amazon Linux AMI” isn’t super helpful. • Re-scaling the application takes downtime. • Debugging BLESS sucks. • When pen testing BLESS, we had to expose Unicreds. • Defeats the object of pen testing somewhat.
  26. 26. BOSTON 10-11 SEPT 2018 BLESS – When devs don’t do what they’re told • Got BLESS running through Jenkins. • Devs still used our manually deployed bastion. • ”Make it easy to do the right thing and hard to do the wrong thing”. • Resources who maintained BLESS rolled off projects. • Nuances introduced by devs could cause problems.
  27. 27. BOSTON 10-11 SEPT 2018 BLESS - scoreboard • Instructions – 6/10 • Accuracy of instructions – 8/10 • Ease of configuration – 5/10 • Ragequit score – 7/10
  28. 28. BOSTON 10-11 SEPT 2018 Security Monkey – Qué? • Security Monkey is a tool that monitors/alerts/reports one or more AWS accounts for anomalies. • Part of a larger suite of tools from Netflix known as the Simian Army.
  29. 29. BOSTON 10-11 SEPT 2018 Security Monkey – Deployment structure
  30. 30. BOSTON 10-11 SEPT 2018 Security Monkey - Hurrah, instructions! • Hey, this one looks like it has a decent walkthrough on Github. • Let’s give it a go.
  31. 31. BOSTON 10-11 SEPT 2018 Security Monkey - Ah, maybe not yay after all. • Oh wait… it’s kind of out of date… M1 instances don’t exist any more. Decide to wing it and pick an M5. This is not free tier.
  32. 32. BOSTON 10-11 SEPT 2018 Security Monkey - When your lab messes things up • Pro tip: never use a lab your colleague has only half configured. • Instance was not accessible from external bastion host. • Bastion host wouldn’t forward SSH keys to the Security Monkey instance. • Cue numerous error messages and troubleshooting of security groups and NACLs.
  33. 33. BOSTON 10-11 SEPT 2018 Security Monkey – Let’s build this • Now for the interesting stuff. • Let’s install this thing. • Pull all the files from Git… • Oops, in my enthusiasm I ran the commands for GCP and Openstack.
  34. 34. BOSTON 10-11 SEPT 2018 Security Monkey - Why doesn’t my instance recognise loopback? • All going well until sudo keeps failing. • My instance does not know it’s own loopback. • Bad Ubuntu! • Change to /etc/hosts fixed this.
  35. 35. BOSTON 10-11 SEPT 2018 Security Monkey - Python isn’t working • When a guide posts something like this, you should probably pay attention to it: • Because when you don’t, you get this:
  36. 36. BOSTON 10-11 SEPT 2018 Security Monkey - Je ne parle pas anglais. • Running in the virtual environment shell now, my bad. • Run the commands to compile the web interface. • Isn’t this installed by default?! • This makes no sense. • Rage level getting critical at this point. • Accurate representation of my face.
  37. 37. BOSTON 10-11 SEPT 2018 Security Monkey - Who doesn’t love a 404? • No idea why, but I had to re-generate the en_US locales. • Then, success!
  38. 38. BOSTON 10-11 SEPT 2018 Security Monkey – I spoke too soon • Now everything should be running, right?
  39. 39. BOSTON 10-11 SEPT 2018 Security Monkey – Mysterious directories • NGNIX can’t find the UI pages to load. • Much searching, much raging. • Transpires that the NGNIX location path was incorrect. • Files had been copied as /usr/local/src/security_monkey/security_monkey/static… There it is
  40. 40. BOSTON 10-11 SEPT 2018 Security Monkey – Damn those SSL certs • Generate self-signed SSL certs. • Getting an error from Chrome, success! • STILL GETTING A 404. • Remove SSL from the config, for now. • I appreciate the irony as a security professional.
  41. 41. BOSTON 10-11 SEPT 2018 Security Monkey – Hello web UI! • …aaaaand:
  42. 42. BOSTON 10-11 SEPT 2018 Security Monkey – Dude, where’s my login server? • Pretty sure I’m supposed to have a login screen? • That red error doesn’t look great. • The Googles reveals that file permissions are a common cause of this issue. • Also need to restart the supervisor service.
  43. 43. BOSTON 10-11 SEPT 2018 Security Monkey – Success!
  44. 44. BOSTON 10-11 SEPT 2018 Security Monkey – Production issues • Issue lists aren’t very detailed. • Dashboard scores for the high score view are not update, but show fine on the summary page. • Daily summary emails don’t get sent out.
  45. 45. BOSTON 10-11 SEPT 2018 Security Monkey - Scoreboard • Instructions – 8/10 • Accuracy of instructions – 7/10 • Ease of configuration – 5/10 • Ragequit score – 8/10
  46. 46. BOSTON 10-11 SEPT 2018 Repokid - Qué? • Repokid uses Access Advisor provided by Aardvark to remove permissions granting access to unused services from the inline policies of IAM roles in an AWS account. • “When used together, Aardvark and Repokid help us get closer to the principle of least privilege without sacrificing speed or introducing heavy process.” - Netflix
  47. 47. BOSTON 10-11 SEPT 2018 Repokid - Wow, these instructions are pretty light. • Even by Netflix standards, these are pretty light… • Pull repo from Git. • Create database. • Create IAM roles.
  48. 48. BOSTON 10-11 SEPT 2018 Repokid – The downsides of using a lab • Run out of elastic IPs. • Reassign one, but now my terminal is angry with me. *sigh*
  49. 49. BOSTON 10-11 SEPT 2018 Repokid – More Python woes… • Instance says virtual env isn’t there (apparently). • Instance also says there is no Git. • Fair enough. • Pull Git package. • Try to pull repo from Github. • There’s already a repokid directory?! Me
  50. 50. BOSTON 10-11 SEPT 2018 Repokid – Only one thing to do TERMINATE
  51. 51. BOSTON 10-11 SEPT 2018 Repokid – Git and SSH key troubles • Wash, rinse, repeat the previous slides. • Accessing Git repo…
  52. 52. BOSTON 10-11 SEPT 2018 Repokid – Let’s try that again • Generate fresh SSH keys. • Add to my agent. • Upload to Github.
  53. 53. BOSTON 10-11 SEPT 2018 Repokid – Never mentioned I needed a database • Apparently I need a Dynamo DB. • Use a small local one for dev purposes. • Pull Java packages, etc. to run it. • This seems to be working fine.
  54. 54. BOSTON 10-11 SEPT 2018 Repokid – Readmes with footnotes • The footnotes for Repokid are important. • Describe what roles need to be set up for the instance to work. • Might have been useful further up the document…
  55. 55. BOSTON 10-11 SEPT 2018 Repokid - Fine tuning JSON • Fine tune the JSON config file. • Point at Aardvark, Dynamo DB and IAM role. • Aaaaand…
  56. 56. BOSTON 10-11 SEPT 2018 Repokid – Production issues • role.policies only checks inline policies. Attached policies are ignored. • Generates heaps of alerts/errors in Lightsail. • The advice for the moment is… just put up and shut up. • (unless you’re going to write your own code to fix)
  57. 57. BOSTON 10-11 SEPT 2018 Repokid - Scoreboard • Instructions – 2/10 • Accuracy of instructions – 2/10 • Ease of configuration – 7/10 • Ragequit score – 7/10
  58. 58. BOSTON 10-11 SEPT 2018 And I’m finished!
  59. 59. BOSTON 10-11 SEPT 2018 Lessons learned • Read what instructions you have carefully… • … but don’t be entirely beholden to them. • Get your base packages and dependencies in order with your code. • Have your supporting tools (terminal, Github, etc.) all in order. • Don’t be afraid to try to run things slightly differently if it works better for your environment. • It’s not failing to ask for help if you’re really stuck.
  60. 60. BOSTON 10-11 SEPT 2018 What’s next? • Diffy! • Diffy is a triage tool to help digital forensics and quickly identify compromised hosts on which to focus their response. • Diffy finds outliers among a group of very similar hosts and highlights those for a human investigator, who can then examine those hosts more closely. • So far… so little instruction.
  61. 61. BOSTON 10-11 SEPT 2018 Documents and links • Netflix Open Source Software Center - https://netflix.github.io/ • Netflix tech blog - https://medium.com/netflix-techblog • Netflix Git repository - https://github.com/Netflix • Lyft’s implementation of BLESS - https://www.youtube.com/watch?v=PMlT1raRMA0 • Versent’s saml2aws repository - https://github.com/Versent/saml2aws • Versent’s unicreds repository - https://github.com/Versent/unicreds • Sethkor’s BLESS repository – https://github.com/sethkor/blesskor • Risky Business #486 Repokid episode - https://risky.biz/RB486/ • Netflix Security’s YouTube Channel - https://www.youtube.com/channel/UCCic- LGj5o892PhU_xrWq-g
  62. 62. BOSTON 10-11 SEPT 2018 Thanks for not ragequitting on my talk and going to happy hour. Questions? @_sarahyo

×