Where fans buy & sell tickets™

Security in the Deployment Stack
SF Bay Area Large Scale Production Engineering
October 17...
Disclaimer:
No StubHub! solutions are being discussed in
this presentation, because:
1.  I’m drawing from 20+ years of exp...
Common Security Paradigms:
1)  Analyze every decision for security benefits.
2)  Hard exterior, “soft and gooey interior”
...
These Paradigms are incomplete:
1)  Only small sites can be understood completely.
2)  Applications require differing leve...
Insecure by Protecting One Layer:
Hardened network
prevents external logins
§  Application hacked
to create outbound
cons...
Insecure by Ignorance:
Easy to laugh about, much harder to solve.
1)  Network security is aimed at protecting protocols.
2...
Insecure by Design:
1)  Each web service provides one or more “features”
2)  Security has to identify every approach vecto...
Insecure by Politics:
1)  Draconian security controls frustrate teams and
produce “outside” or “bypass” implementations

2...
Access versus Authorization:
“If we all have root, anyone can fix a problem...”
1)  Most security instruments deal with
au...
“If Can Access” Authorization:
Real e-mail:
Hey guys, this is #####. So I built out our application
stack in the office an...
You can’t lock them out:
1)  Application service providers often must expose
APIs to customers
2)  Mobile customers don’t ...
What the Business Wants: Security
1)  Lots of eyeballs
2)  Careful analysis
3)  Stability is paramount

== Slow
What the Business Wants: Agility
1)  Lean teams
2)  Respond to market quickly
3)  “Let’s break things!”

== Fast
Every person in this room stands
between these points every day:
1)  Keep it up no matter what.
2)  Move faster! Let’s bre...
Security in the Deployment Stack
We’ve discussed the problem.
Now it is time to talk about solutions.
If every component must be secure,
How can you do it better and faster?
1)  You make it smaller.
2)  You make it simpler.
...
Smaller Security Evaluation
1)  Security lead from each team evaluates proposal
2)  Evaluate each intersection -- the smal...
Simpler Design Theory
1)  Make each component as small as possible
2)  Use components that can be reused.
--avoids “what t...
Test More Often
1)  Test for new potentials identified in review
2)  Test previous release concerns, every time
3)  Test p...
Deployment -- devOps
1)  Application security needs and deployment needs
are not the same
2)  Automated is better: manual ...
Security in the Deployment Stack
Keystores
Common Security Keystores
1.  Static files
2.  Database lookups
3.  LDAP
4.  Ja...
Security in the Deployment Stack
Install during System Image
Install security keystores during system build
Advantages:
1....
Security in the Deployment Stack
Install with System Packages
Install security keystores with system packages
Advantages
1...
Security in the Deployment Stack
Install with Config Management
Implement security keystore as policy implemented by
confi...
SUMMARY
1)  Security models don’t match modern expectations.
2)  Security failures are not usually Apps or APIs, but
the i...
Where fans buy & sell tickets™

We are hiring!
Operations Architects
Tools & Automation
devOps
Site Operations
Network Ope...
Upcoming SlideShare
Loading in...5
×

Security in the deployment stack

321

Published on

Jo Rhett's talk from http://www.meetup.com/SF-Bay-Area-Large-Scale-Production-Engineering/events/129859602/

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
321
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Security in the deployment stack

  1. 1. Where fans buy & sell tickets™ Security in the Deployment Stack SF Bay Area Large Scale Production Engineering October 17th, 2013 Jo Rhett Senior Operations Architect
  2. 2. Disclaimer: No StubHub! solutions are being discussed in this presentation, because: 1.  I’m drawing from 20+ years of experience. 2.  I’m an FNG at StubHub! and I haven’t learned all the great things we do yet. 3.  We’ve got even more entertaining ways to break things.
  3. 3. Common Security Paradigms: 1)  Analyze every decision for security benefits. 2)  Hard exterior, “soft and gooey interior” 3)  Strict (or not) host level security 4)  Strict (or not) application level security
  4. 4. These Paradigms are incomplete: 1)  Only small sites can be understood completely. 2)  Applications require differing levels of network access, which creates “gooey spots” in supposedly crunchy exterior. 3)  Doesn’t account for insecure by design: - Wikipedia, Craigslist 4)  Doesn’t account for center-less design: - BitTorrent 5)  Insecure by ignorance: “You could do that?”
  5. 5. Insecure by Protecting One Layer: Hardened network prevents external logins §  Application hacked to create outbound console reverse shell §  Application hacked to change data without login SQL injection, remote file inclusion Hardened application enforces security paradigm – versus – §  Network attacks disable service slowloris, slowpost §  Network attack bypasses authentication session hijacking, replay attacks, authorization sniffer
  6. 6. Insecure by Ignorance: Easy to laugh about, much harder to solve. 1)  Network security is aimed at protecting protocols. 2)  Most things are done on the same protocol today. 3)  Decisions made in different teams create an interwoven mesh of security expectations. - Network only allows HTTPS access - Web API allows DB access via HTTPS
  7. 7. Insecure by Design: 1)  Each web service provides one or more “features” 2)  Security has to identify every approach vector, and the intersections between each feature. 3)  An attacker only has to find one feature that provides data or hook which makes it possible to meet the expectations of a different feature. 4)  Complete security review requires N^N-1 analysis.
  8. 8. Insecure by Politics: 1)  Draconian security controls frustrate teams and produce “outside” or “bypass” implementations 2)  Failure to understand the value of cracking your application. If the attacker can make/save money, they will be very persistent.
  9. 9. Access versus Authorization: “If we all have root, anyone can fix a problem...” 1)  Most security instruments deal with authentication, not authorization. 2)  Static files, DB Lookups, LDAP, Java Keystore… 3)  Many implementations use “if can access” as the authorization scheme. 4)  Projects require more access to enable Mobile and batch processing clients.
  10. 10. “If Can Access” Authorization: Real e-mail: Hey guys, this is #####. So I built out our application stack in the office and took it home on my laptop. Turns out, I can test the entire thing from home if you’ll give me direct database access… This was possible because every dependency was a Tomcat application available on an exposed role. But wait… It gets better! Followup e-mail: Nevermind, I realized I can point at our production gateway for the DB API calls and it works fine…
  11. 11. You can’t lock them out: 1)  Application service providers often must expose APIs to customers 2)  Mobile customers don’t use “trusted networks” 3)  The consumer provides valuable content - Wikipedia - Craigslist - StubHub! to some extent
  12. 12. What the Business Wants: Security 1)  Lots of eyeballs 2)  Careful analysis 3)  Stability is paramount == Slow
  13. 13. What the Business Wants: Agility 1)  Lean teams 2)  Respond to market quickly 3)  “Let’s break things!” == Fast
  14. 14. Every person in this room stands between these points every day: 1)  Keep it up no matter what. 2)  Move faster! Let’s break things! Feels real comfortable, doesn’t it?
  15. 15. Security in the Deployment Stack We’ve discussed the problem. Now it is time to talk about solutions.
  16. 16. If every component must be secure, How can you do it better and faster? 1)  You make it smaller. 2)  You make it simpler. 3)  You do it more often.
  17. 17. Smaller Security Evaluation 1)  Security lead from each team evaluates proposal 2)  Evaluate each intersection -- the small points 3)  Build shared knowledge This collaboration allows each person to focus on the components they know best.
  18. 18. Simpler Design Theory 1)  Make each component as small as possible 2)  Use components that can be reused. --avoids “what the heck is this again?” 3)  Less moving parts with common dependencies
  19. 19. Test More Often 1)  Test for new potentials identified in review 2)  Test previous release concerns, every time 3)  Test production too 4)  Automated testing required humans get bored
  20. 20. Deployment -- devOps 1)  Application security needs and deployment needs are not the same 2)  Automated is better: manual deployments lead to inconsistent deployment human mind: didn’t I do that already? 3)  Operations Developer writes code/policy to deploy application stack a.  Kickstart / Cobbler / Foreman / Razor b.  Cfengine / Puppet / Chef c.  Mcollective / Capistrano d.  …custom built 4)  Commit the deployment code with the release branch
  21. 21. Security in the Deployment Stack Keystores Common Security Keystores 1.  Static files 2.  Database lookups 3.  LDAP 4.  Java Keystores 5.  SSL Certificates All of these can be installed during your application installation, or outside of it (system build or side load)
  22. 22. Security in the Deployment Stack Install during System Image Install security keystores during system build Advantages: 1.  only needs to be tested as a unit Disadvantages: 1.  no updates to an existing role without rebuild
  23. 23. Security in the Deployment Stack Install with System Packages Install security keystores with system packages Advantages 1.  only needs to be tested as a unit 2.  simple redeploy with system packages Disadvantages: 1.  requires package rebuild for simple changes 2.  limited scripting language for install/uninstall
  24. 24. Security in the Deployment Stack Install with Config Management Implement security keystore as policy implemented by configuration management. Advantages: 1.  Configuration change can be pushed 2.  Incremental upgrades are possible 3.  Complex upgrade procedure can be implemented 4.  Failover, failback process can be employed Disadvantages: 1.  must test the deployment as well as the unit
  25. 25. SUMMARY 1)  Security models don’t match modern expectations. 2)  Security failures are not usually Apps or APIs, but the intersection of access between several of them. 3)  evaluate Smaller, build Simpler, test More Often. 4)  Develop deployment code/policy as carefully as you do the application. Commit it with the release branch of the application. 5)  Automate your tests. Automate your deployment. Automate failure analysis.
  26. 26. Where fans buy & sell tickets™ We are hiring! Operations Architects Tools & Automation devOps Site Operations Network Operations

×