London Devops #9 - Security at a startup
•
0 likes•819 views
This document outlines an approach to security at a startup taken by Beamly. It involves outsourcing commodity services like hosting and monitoring to providers like AWS and Dataloop.io. It advocates setting up a single secure point of truth using LDAP to manage all user accounts and integrate with services. Beamly implemented security "unit" tests as Python scripts to regularly audit third party services and alert the team of any issues. The approach emphasizes enabling two-factor authentication wherever possible and regularly backing up data and configurations.
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to London Devops #9 - Security at a startup
Similar to London Devops #9 - Security at a startup (20)
Recently uploaded
Recently uploaded (20)
London Devops #9 - Security at a startup
- 1. 0 Security at a startup @neiljsaunders neil@beamly.com
- 2. 1
- 3. 2 • At a huge competitive advantage compared to enterprises • No RFPs, Long winded procurement contracts • Outsource commodity services to others: – Most obviously – Hosting – AWS – Monitoring – Dataloop.io – Alerting – Email – Source control and versioning – Messaging Startups
- 4. 3
- 6. 5
- 7. 6
- 8. 7
- 9. 8
- 11. 10 ?
- 12. 11
- 13. 12
- 14. - A security oversight in one system can often be used to compromise another - Are you aware when a new email address gets added? - What’s this IP address in our AWS security group? When was it added? Who added it? If I remove it what will stop working? - Whose Github account is this? - What’s this hook on my github repo? - The list is endless, and is easily work for a dedicated team - That team will never exist in a startup! - I just want to focus on cutting our deploy time in half and implementing Docker! Security Jenga
- 15. 14 One potential solution… - A light weight process that actively audits security and alerts your “Security team” when something doesn’t “smell” right - Needs to be extensible to keep up with all those new third party services being added - Adding services needs to take up as little time as possible
- 16. 15 Single (secure) point of truth - Can be as simple as a text file on github - We chose LDAP - Deserving in its reputation as a PITA to setup: 6 months to learn, build, test, secure, and populate (More on that later) - It integrates with everything (More on that later too) - Created a custom schema extension for a Beamly employees - Github ID - Facebook User UID - Public SSH Key - At first internal only - Now externally accessible
- 17. 16 Security “unit” tests • Now you’ve got a version controlled state • If you can programmatically extract a list of users from a service, you can compare against that • Applies to more than just user accounts: – AWS Security groups – Github hooks • Store exceptions with explanatory comments • Track changes with git history
- 18. 17 Beamly Security Scripts • Implemented as Python “unit” tests (py.unit) • Short & quick to write • Executed every 10 minutes via a scheduled CD pipeline • Failures trigger an email to the platform team • Email can trigger whatever else you want (Pagerduty) • Platform team can login and take appropriate action
- 19. 18
- 20. 19
- 21. 20
- 22. 21
- 23. 22 collected 1280 items test_all_github_hooks_are_known.py ...................................................................................................... ......................................................................................................................................... .................................................................................................................................. test_aws_iam_users_with_passwords_have_2fa.py ..................................................................................... test_aws_security_groups.py ............................................................................................................. ................................................................................. test_github_users_and_in_ldap_and_2fa.py ............................................................. test_jira_users_not_in_ldap.py .......................................................................................................... ........................................................................ test_o365_users.py ........................................................................................................... test_onelogin_users_are_in_ldap.py ........................................................................................... test_pagerduty_users_not_in_ldap.py ........................ test_slack_users_not_in_ldap_and_2fa.py ............................................................F.................................... ..................................................... =================================== FAILURES =================================== _________________________ test_unknown_slack_users[60] _________________________ realname = ’New Joiner', two_factor_enabled = False def two_factor_enabled(realname, two_factor_enabled): > assert two_factor_enabled E assert False test_slack_users_not_in_ldap_and_2fa.py:42: AssertionError =================== 1 failed, 1279 passed in 145.26 seconds ====================
- 24. 23
- 25. 24
- 26. 25
- 27. 26
- 28. 27 Systems Integrated with LDAP - VPN access to VPC - Host access (Using SSH Public keys) - Sudo access (Via group) - Office appliances (NAS) - ….any application that supports it – GO, Grafana (Soon) Couldn’t have done this if we’d have used a text file
- 29. 28
- 30. 29 • Herd all these 3rd party applications in to one place • Password manager on steroids • Centrally managed • Every application used by Beamly added by Platform team • Authenticate with LDAP account and 2FA • Single sign on just to services that users have rights to (LDAP groups) • Users never know the password for shared systems • Can bulk rotate shared passwords for all systems OneLogin
- 31. 30
- 32. 31
- 33. 32 Now what?!
- 34. 33 “Preppers”!
- 38. 37 BE A PREPPER!
- 40. Part I: Stock your stores - Backups • Taking backups isn’t good enough • 3 -2 -1 • 3 Copies of your data • 2 in different formats • 1 Offsite • S3 mutli-region ISN’T OFF SITE • S3 > (Another cloud provider || On site) • March 31st: World backup day
- 41. Part II: Run through restoring from nothing • In the event of the Armageddon, you need to fight to get back up quickly • How long would if take to restore your AWS “Scaffolding” alone – VPC Configuration – Security groups – Routing tables – Subnet configurations – DNS • Ideally you’d have all this in Cloudformation…. • ….We set all this stuff up before Cloudformation was available • AWS Cloudformer
- 42. 41 To sum up: our approach • If a service supports 2FA, it’s mandatory. – Currently Slack, Mailchimp, Github, AWS, and Onelogin • If a service has an API for exporting a security config: try and script an audit for it • All third party services in Onelogin • Bulk rotate shared passwords regularly • Prepare for Armageddon:! – Backup CD config & AWS setup – All data tiers backed up 3-2-1 (One offsite)
- 43. 42 Lessons Learned • Retro-fitting security is hard • Be prepared to break lots of things (temporarily) • Bring everyone along for the ride – Explain why it’s important – Everyone gets the access they need – #security for interested parties • Turn on AWS Cloudtrail and Config everywhere
- 45. LONDON Drury House 34-43 Russell Street London WC2B 5HA NEW YORK CITY 84 Wooster Street Suite 703 New York NY 10012 SYDNEY 22-36 Mountain Street Suite 1.10 Ultimo NSW 2007 © Beamly 2014. All Rights Reserved.