0
Security at a startup
@neiljsaunders
neil@beamly.com
1
2
•  At a huge competitive advantage compared to enterprises
•  No RFPs, Long winded procurement contracts
•  Outsource commodity services to others:
–  Most obviously – Hosting – AWS
–  Monitoring – Dataloop.io
–  Alerting
–  Email
–  Source control and versioning
–  Messaging
Startups
3
5
6
7
8
10
?
11
12
-  A security oversight in one system can often be used to compromise
another
-  Are you aware when a new email address gets added?
-  What’s this IP address in our AWS security group? When was it
added? Who added it? If I remove it what will stop working?
-  Whose Github account is this?
-  What’s this hook on my github repo?
-  The list is endless, and is easily work for a dedicated team
-  That team will never exist in a startup!
-  I just want to focus on cutting our deploy time in half and
implementing Docker!

Security Jenga
14
One potential solution… 

-  A light weight process that actively audits security and alerts your
“Security team” when something doesn’t “smell” right
-  Needs to be extensible to keep up with all those new third party
services being added
-  Adding services needs to take up as little time as possible
15
Single (secure) point of truth
-  Can be as simple as a text file on github
-  We chose LDAP 
-  Deserving in its reputation as a PITA to setup: 6 months to learn, build, test, secure,
and populate (More on that later)
-  It integrates with everything (More on that later too)
-  Created a custom schema extension for a Beamly employees
-  Github ID
-  Facebook User UID
-  Public SSH Key
-  At first internal only
-  Now externally accessible
16
Security “unit” tests 

•  Now you’ve got a version controlled state
•  If you can programmatically extract a list of users from a
service, you can compare against that
•  Applies to more than just user accounts:
–  AWS Security groups
–  Github hooks
•  Store exceptions with explanatory comments
•  Track changes with git history
17
Beamly Security Scripts
•  Implemented as Python “unit” tests (py.unit)
•  Short & quick to write
•  Executed every 10 minutes via a scheduled CD pipeline
•  Failures trigger an email to the platform team
•  Email can trigger whatever else you want (Pagerduty)
•  Platform team can login and take appropriate action
18
19
20
21
22
collected	
  1280	
  items	
  
	
  
test_all_github_hooks_are_known.py	
  ......................................................................................................
.........................................................................................................................................
..................................................................................................................................	
  
test_aws_iam_users_with_passwords_have_2fa.py	
  .....................................................................................	
  
test_aws_security_groups.py	
  .............................................................................................................
.................................................................................	
  
test_github_users_and_in_ldap_and_2fa.py	
  .............................................................	
  
test_jira_users_not_in_ldap.py	
  ..........................................................................................................
........................................................................	
  
test_o365_users.py	
  ...........................................................................................................	
  
test_onelogin_users_are_in_ldap.py	
  ...........................................................................................	
  
test_pagerduty_users_not_in_ldap.py	
  ........................	
  
test_slack_users_not_in_ldap_and_2fa.py	
  ............................................................F....................................
.....................................................	
  
	
  
===================================	
  FAILURES	
  ===================================	
  
_________________________	
  test_unknown_slack_users[60]	
  _________________________	
  
	
  
realname	
  =	
  ’New	
  Joiner',	
  two_factor_enabled	
  =	
  False	
  
	
  
	
  	
  	
  	
  def	
  two_factor_enabled(realname,	
  two_factor_enabled):	
  
	
  	
  	
  	
  	
  
>	
  	
  	
  	
  	
  	
  	
  assert	
  two_factor_enabled	
  
E	
  	
  	
  	
  	
  	
  	
  assert	
  False	
  
	
  
test_slack_users_not_in_ldap_and_2fa.py:42:	
  AssertionError	
  
===================	
  1	
  failed,	
  1279	
  passed	
  in	
  145.26	
  seconds	
  ====================	
  
23
24
25
26
27
Systems Integrated with LDAP
-  VPN access to VPC
-  Host access (Using SSH Public keys)
-  Sudo access (Via group)
-  Office appliances (NAS)
-  ….any application that supports it – GO, Grafana (Soon)

Couldn’t have done this if we’d have used a text file
28
29
•  Herd all these 3rd party applications in to one place
•  Password manager on steroids
•  Centrally managed
•  Every application used by Beamly added by Platform team
•  Authenticate with LDAP account and 2FA
•  Single sign on just to services that users have rights to (LDAP groups)
•  Users never know the password for shared systems
•  Can bulk rotate shared passwords for all systems

OneLogin
30
31
32
Now what?!
33
“Preppers”!
37
BE A PREPPER!
Part I: Stock your stores - Backups
•  Taking backups isn’t good enough
•  3 -2 -1
•  3 Copies of your data
•  2 in different formats
•  1 Offsite
•  S3 mutli-region ISN’T OFF SITE
•  S3 > (Another cloud provider || On site)
•  March 31st: World backup day
Part II: Run through restoring from nothing
•  In the event of the Armageddon, you need to fight to get back up quickly
•  How long would if take to restore your AWS “Scaffolding” alone
–  VPC Configuration
–  Security groups
–  Routing tables
–  Subnet configurations
–  DNS
•  Ideally you’d have all this in Cloudformation….
•  ….We set all this stuff up before Cloudformation was available
•  AWS Cloudformer
41
To sum up: our approach
•  If a service supports 2FA, it’s mandatory.
–  Currently Slack, Mailchimp, Github, AWS, and Onelogin
•  If a service has an API for exporting a security config: try and script an
audit for it
•  All third party services in Onelogin
•  Bulk rotate shared passwords regularly
•  Prepare for Armageddon:!
–  Backup CD config & AWS setup
–  All data tiers backed up 3-2-1 (One offsite)
42
Lessons Learned
•  Retro-fitting security is hard
•  Be prepared to break lots of things (temporarily)
•  Bring everyone along for the ride
–  Explain why it’s important
–  Everyone gets the access they need
–  #security for interested parties
•  Turn on AWS Cloudtrail and Config everywhere
43
Questions?


@neiljsaunders
neil@beamly.com
LONDON

Drury House
34-43 Russell Street
London
WC2B 5HA
NEW YORK CITY

84 Wooster Street
Suite 703
New York
NY 10012
SYDNEY

22-36 Mountain Street
Suite 1.10
Ultimo
NSW 2007
© Beamly 2014. All Rights Reserved.

London Devops #9 - Security at a startup

  • 1.
    0 Security at astartup @neiljsaunders neil@beamly.com
  • 2.
  • 3.
    2 •  At ahuge competitive advantage compared to enterprises •  No RFPs, Long winded procurement contracts •  Outsource commodity services to others: –  Most obviously – Hosting – AWS –  Monitoring – Dataloop.io –  Alerting –  Email –  Source control and versioning –  Messaging Startups
  • 4.
  • 6.
  • 7.
  • 8.
  • 9.
  • 11.
  • 12.
  • 13.
  • 14.
    -  A securityoversight in one system can often be used to compromise another -  Are you aware when a new email address gets added? -  What’s this IP address in our AWS security group? When was it added? Who added it? If I remove it what will stop working? -  Whose Github account is this? -  What’s this hook on my github repo? -  The list is endless, and is easily work for a dedicated team -  That team will never exist in a startup! -  I just want to focus on cutting our deploy time in half and implementing Docker! Security Jenga
  • 15.
    14 One potential solution… -  A light weight process that actively audits security and alerts your “Security team” when something doesn’t “smell” right -  Needs to be extensible to keep up with all those new third party services being added -  Adding services needs to take up as little time as possible
  • 16.
    15 Single (secure) pointof truth -  Can be as simple as a text file on github -  We chose LDAP -  Deserving in its reputation as a PITA to setup: 6 months to learn, build, test, secure, and populate (More on that later) -  It integrates with everything (More on that later too) -  Created a custom schema extension for a Beamly employees -  Github ID -  Facebook User UID -  Public SSH Key -  At first internal only -  Now externally accessible
  • 17.
    16 Security “unit” tests •  Now you’ve got a version controlled state •  If you can programmatically extract a list of users from a service, you can compare against that •  Applies to more than just user accounts: –  AWS Security groups –  Github hooks •  Store exceptions with explanatory comments •  Track changes with git history
  • 18.
    17 Beamly Security Scripts • Implemented as Python “unit” tests (py.unit) •  Short & quick to write •  Executed every 10 minutes via a scheduled CD pipeline •  Failures trigger an email to the platform team •  Email can trigger whatever else you want (Pagerduty) •  Platform team can login and take appropriate action
  • 19.
  • 20.
  • 21.
  • 22.
  • 23.
    22 collected  1280  items     test_all_github_hooks_are_known.py  ...................................................................................................... ......................................................................................................................................... ..................................................................................................................................   test_aws_iam_users_with_passwords_have_2fa.py  .....................................................................................   test_aws_security_groups.py  ............................................................................................................. .................................................................................   test_github_users_and_in_ldap_and_2fa.py  .............................................................   test_jira_users_not_in_ldap.py  .......................................................................................................... ........................................................................   test_o365_users.py  ...........................................................................................................   test_onelogin_users_are_in_ldap.py  ...........................................................................................   test_pagerduty_users_not_in_ldap.py  ........................   test_slack_users_not_in_ldap_and_2fa.py  ............................................................F.................................... .....................................................     ===================================  FAILURES  ===================================   _________________________  test_unknown_slack_users[60]  _________________________     realname  =  ’New  Joiner',  two_factor_enabled  =  False            def  two_factor_enabled(realname,  two_factor_enabled):             >              assert  two_factor_enabled   E              assert  False     test_slack_users_not_in_ldap_and_2fa.py:42:  AssertionError   ===================  1  failed,  1279  passed  in  145.26  seconds  ====================  
  • 24.
  • 25.
  • 26.
  • 27.
  • 28.
    27 Systems Integrated withLDAP -  VPN access to VPC -  Host access (Using SSH Public keys) -  Sudo access (Via group) -  Office appliances (NAS) -  ….any application that supports it – GO, Grafana (Soon) Couldn’t have done this if we’d have used a text file
  • 29.
  • 30.
    29 •  Herd allthese 3rd party applications in to one place •  Password manager on steroids •  Centrally managed •  Every application used by Beamly added by Platform team •  Authenticate with LDAP account and 2FA •  Single sign on just to services that users have rights to (LDAP groups) •  Users never know the password for shared systems •  Can bulk rotate shared passwords for all systems OneLogin
  • 31.
  • 32.
  • 33.
  • 34.
  • 38.
  • 40.
    Part I: Stockyour stores - Backups •  Taking backups isn’t good enough •  3 -2 -1 •  3 Copies of your data •  2 in different formats •  1 Offsite •  S3 mutli-region ISN’T OFF SITE •  S3 > (Another cloud provider || On site) •  March 31st: World backup day
  • 41.
    Part II: Runthrough restoring from nothing •  In the event of the Armageddon, you need to fight to get back up quickly •  How long would if take to restore your AWS “Scaffolding” alone –  VPC Configuration –  Security groups –  Routing tables –  Subnet configurations –  DNS •  Ideally you’d have all this in Cloudformation…. •  ….We set all this stuff up before Cloudformation was available •  AWS Cloudformer
  • 42.
    41 To sum up:our approach •  If a service supports 2FA, it’s mandatory. –  Currently Slack, Mailchimp, Github, AWS, and Onelogin •  If a service has an API for exporting a security config: try and script an audit for it •  All third party services in Onelogin •  Bulk rotate shared passwords regularly •  Prepare for Armageddon:! –  Backup CD config & AWS setup –  All data tiers backed up 3-2-1 (One offsite)
  • 43.
    42 Lessons Learned •  Retro-fittingsecurity is hard •  Be prepared to break lots of things (temporarily) •  Bring everyone along for the ride –  Explain why it’s important –  Everyone gets the access they need –  #security for interested parties •  Turn on AWS Cloudtrail and Config everywhere
  • 44.
  • 45.
    LONDON Drury House 34-43 RussellStreet London WC2B 5HA NEW YORK CITY 84 Wooster Street Suite 703 New York NY 10012 SYDNEY 22-36 Mountain Street Suite 1.10 Ultimo NSW 2007 © Beamly 2014. All Rights Reserved.