SSL* Certificate Reporting                                                         BayLISA                                 ...
SSL* Certificate Reporting                                                         BayLISA                                 ...
Technology Overview                       @royrapoport rsr@netflix.comFriday, March 22, 13
Technology Overview                       • SoA, REST, Mostly Java                       @royrapoport rsr@netflix.comFriday...
Technology Overview                       • SoA, REST, Mostly Java                       • Simple overall architecture:   ...
Technology Overview                       • SoA, REST, Mostly Java                       • Simple overall architecture:   ...
Culture Overview                       @royrapoport rsr@netflix.comFriday, March 22, 13We hire very smart people, give them...
Culture Overview • Freedom and         Responsibility                       @royrapoport rsr@netflix.comFriday, March 22, 1...
Culture Overview • Freedom and         Responsibility • Distributed         Operations                       @royrapoport ...
Culture Overview • Freedom and         Responsibility • Distributed         Operations • Get out of the         way of    ...
So Certificates ...                       @royrapoport rsr@netflix.comFriday, March 22, 13
So Certificates ...                       •   Dozens of Certificates                       @royrapoport rsr@netflix.comFriday...
So Certificates ...                       •   Dozens of Certificates                       •   Different kinds of places    ...
So Certificates ...                       •   Dozens of Certificates                       •   Different kinds of places    ...
So Certificates ...                       •   Dozens of Certificates                       •   Different kinds of places    ...
So Certificates ...                       •   Dozens of Certificates                       •   Different kinds of places    ...
So Certificates ...                       •   Dozens of Certificates                       •   Different kinds of places    ...
So Certificates ...                       •   Dozens of Certificates                       •   Different kinds of places    ...
So Certificates ...                       •   Dozens of Certificates                       •   Different kinds of places    ...
So Certificates ...                       •   Dozens of Certificates                       •   Different kinds of places    ...
So Certificates ...                       • Some Certificates Weren’t[sic]                       @royrapoport rsr@netflix.com...
So Certificates ...                       @royrapoport rsr@netflix.comFriday, March 22, 13(obviously, the ‘standard ways to ...
So Certificates ...            •          SSL Certificates expire                        @royrapoport rsr@netflix.comFriday, ...
So Certificates ...            •          SSL Certificates expire                  •      Millions of people can’t stream   ...
So Certificates ...            •          SSL Certificates expire                  •      Millions of people can’t stream   ...
So Certificates ...            •          SSL Certificates expire                  •      Millions of people can’t stream   ...
So Certificates ...            •          SSL Certificates expire                  •      Millions of people can’t stream   ...
So Certificates ...            •          SSL Certificates expire                  •      Millions of people can’t stream   ...
So Certificates ...            •          SSL Certificates expire                  •      Millions of people can’t stream   ...
Let’s Do This Thing                                                     Cassandra                                         ...
Let’s Do This Thing                        ELB                                                     Cassandra              ...
Let’s Do This Thing                        ELB                                                     Cassandra              ...
Let’s Do This Thing                         ELB                                                      Cassandra            ...
Let’s Do This Thing                          ELB                                                      Cassandra           ...
Let’s Do This Thing                          ELB                                                        Cassandra         ...
Let’s Do This Thing                          ELB                                                        Cassandra         ...
Let’s Do This Thing                          ELB                                                        Cassandra         ...
Since Then                       @royrapoport rsr@netflix.comFriday, March 22, 13We validated the design by continuing to i...
Since Then            • No Production Emergencies due to SSL                       certificate expiration                  ...
Since Then            • No Production Emergencies due to SSL                       certificate expiration            • Vali...
Since Then            • No Production Emergencies due to SSL                       certificate expiration            • Vali...
Soon ...                       @royrapoport rsr@netflix.comFriday, March 22, 13We should be able to figure out who owns a ce...
Soon ...            • Customized, automated alerting                       @royrapoport rsr@netflix.comFriday, March 22, 13...
Soon ...            • Customized, automated alerting            • Automated renewal                       @royrapoport rsr...
Soon ...            • Customized, automated alerting            • Automated renewal             • Telling you a problem is...
Soon ...            • Customized, automated alerting            • Automated renewal             • Telling you a problem is...
Soon ...            • Customized, automated alerting            • Automated renewal             • Telling you a problem is...
Remember ...                       @royrapoport rsr@netflix.comFriday, March 22, 13
Remember ...            • Be Lazy                       @royrapoport rsr@netflix.comFriday, March 22, 13
Remember ...            • Be Lazy            • Help Others Be Lazy                       @royrapoport rsr@netflix.comFriday...
Remember ...            • Be Lazy            • Help Others Be Lazy            • Computers Are Better Than Humans          ...
Remember ...            • Be Lazy            • Help Others Be Lazy            • Computers Are Better Than Humans          ...
Remember ...            • Be Lazy            • Help Others Be Lazy            • Computers Are Better Than Humans          ...
Questions?                       @royrapoport rsr@netflix.comFriday, March 22, 13
Upcoming SlideShare
Loading in...5
×

SSL Certificate Expiration and Howler Monkey's Inception

752

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
752
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
9
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

SSL Certificate Expiration and Howler Monkey's Inception

  1. 1. SSL* Certificate Reporting BayLISA March 21st, 2013 @royrapoport rsr@netflix.comFriday, March 22, 13This is the story of how we went from SSL certificates expiring without notice in production todeploying Security Monkey (later renamed Howler Monkey) and permanently eliminating SSLcertificate expiration as a production-class issue.
  2. 2. SSL* Certificate Reporting BayLISA March 21st, 2013 @royrapoport rsr@netflix.comFriday, March 22, 13This is the story of how we went from SSL certificates expiring without notice in production todeploying Security Monkey (later renamed Howler Monkey) and permanently eliminating SSLcertificate expiration as a production-class issue.
  3. 3. Technology Overview @royrapoport rsr@netflix.comFriday, March 22, 13
  4. 4. Technology Overview • SoA, REST, Mostly Java @royrapoport rsr@netflix.comFriday, March 22, 13
  5. 5. Technology Overview • SoA, REST, Mostly Java • Simple overall architecture: @royrapoport rsr@netflix.comFriday, March 22, 13
  6. 6. Technology Overview • SoA, REST, Mostly Java • Simple overall architecture: @royrapoport rsr@netflix.comFriday, March 22, 13
  7. 7. Culture Overview @royrapoport rsr@netflix.comFriday, March 22, 13We hire very smart people, give them all the context and situational awareness they want, andset them free. We design our environment, our systems, and our teams to be empowered tomake decisions without requiring slow approval processes, cumbersome formalcommunication, or any other unnecessary friction.
  8. 8. Culture Overview • Freedom and Responsibility @royrapoport rsr@netflix.comFriday, March 22, 13We hire very smart people, give them all the context and situational awareness they want, andset them free. We design our environment, our systems, and our teams to be empowered tomake decisions without requiring slow approval processes, cumbersome formalcommunication, or any other unnecessary friction.
  9. 9. Culture Overview • Freedom and Responsibility • Distributed Operations @royrapoport rsr@netflix.comFriday, March 22, 13We hire very smart people, give them all the context and situational awareness they want, andset them free. We design our environment, our systems, and our teams to be empowered tomake decisions without requiring slow approval processes, cumbersome formalcommunication, or any other unnecessary friction.
  10. 10. Culture Overview • Freedom and Responsibility • Distributed Operations • Get out of the way of Developers @royrapoport rsr@netflix.comFriday, March 22, 13We hire very smart people, give them all the context and situational awareness they want, andset them free. We design our environment, our systems, and our teams to be empowered tomake decisions without requiring slow approval processes, cumbersome formalcommunication, or any other unnecessary friction.
  11. 11. So Certificates ... @royrapoport rsr@netflix.comFriday, March 22, 13
  12. 12. So Certificates ... • Dozens of Certificates @royrapoport rsr@netflix.comFriday, March 22, 13
  13. 13. So Certificates ... • Dozens of Certificates • Different kinds of places @royrapoport rsr@netflix.comFriday, March 22, 13
  14. 14. So Certificates ... • Dozens of Certificates • Different kinds of places • Datacenter/private @royrapoport rsr@netflix.comFriday, March 22, 13
  15. 15. So Certificates ... • Dozens of Certificates • Different kinds of places • Datacenter/private • Datacenter/public/LB @royrapoport rsr@netflix.comFriday, March 22, 13
  16. 16. So Certificates ... • Dozens of Certificates • Different kinds of places • Datacenter/private • Datacenter/public/LB • ELBs @royrapoport rsr@netflix.comFriday, March 22, 13
  17. 17. So Certificates ... • Dozens of Certificates • Different kinds of places • Datacenter/private • Datacenter/public/LB • ELBs • EC2 @royrapoport rsr@netflix.comFriday, March 22, 13
  18. 18. So Certificates ... • Dozens of Certificates • Different kinds of places • Datacenter/private • Datacenter/public/LB • ELBs • EC2 • Source Control @royrapoport rsr@netflix.comFriday, March 22, 13
  19. 19. So Certificates ... • Dozens of Certificates • Different kinds of places • Datacenter/private • Datacenter/public/LB • ELBs • EC2 • Source Control • EIPs @royrapoport rsr@netflix.comFriday, March 22, 13
  20. 20. So Certificates ... • Dozens of Certificates • Different kinds of places • Datacenter/private • Datacenter/public/LB • ELBs • EC2 • Source Control • EIPs • Totally Distributed Design @royrapoport rsr@netflix.comFriday, March 22, 13
  21. 21. So Certificates ... • Some Certificates Weren’t[sic] @royrapoport rsr@netflix.comFriday, March 22, 13Some certificates weren’t even SSL certificates -- we have certificates we get from a partnerthat cannot be accessed via SSL, and for which the answer to the question “when does thisexpire?” require scraping a web page.
  22. 22. So Certificates ... @royrapoport rsr@netflix.comFriday, March 22, 13(obviously, the ‘standard ways to solve this’ part here is somewhat facetious, but these are, infact, the standard ways in which most organizations try to deal with keeping up with SSLcertificate expirations)
  23. 23. So Certificates ... • SSL Certificates expire @royrapoport rsr@netflix.comFriday, March 22, 13(obviously, the ‘standard ways to solve this’ part here is somewhat facetious, but these are, infact, the standard ways in which most organizations try to deal with keeping up with SSLcertificate expirations)
  24. 24. So Certificates ... • SSL Certificates expire • Millions of people can’t stream @royrapoport rsr@netflix.comFriday, March 22, 13(obviously, the ‘standard ways to solve this’ part here is somewhat facetious, but these are, infact, the standard ways in which most organizations try to deal with keeping up with SSLcertificate expirations)
  25. 25. So Certificates ... • SSL Certificates expire • Millions of people can’t stream • Hilarity ensues @royrapoport rsr@netflix.comFriday, March 22, 13(obviously, the ‘standard ways to solve this’ part here is somewhat facetious, but these are, infact, the standard ways in which most organizations try to deal with keeping up with SSLcertificate expirations)
  26. 26. So Certificates ... • SSL Certificates expire • Millions of people can’t stream • Hilarity ensues • Standard Ways to Solve This @royrapoport rsr@netflix.comFriday, March 22, 13(obviously, the ‘standard ways to solve this’ part here is somewhat facetious, but these are, infact, the standard ways in which most organizations try to deal with keeping up with SSLcertificate expirations)
  27. 27. So Certificates ... • SSL Certificates expire • Millions of people can’t stream • Hilarity ensues • Standard Ways to Solve This • Excel worksheets @royrapoport rsr@netflix.comFriday, March 22, 13(obviously, the ‘standard ways to solve this’ part here is somewhat facetious, but these are, infact, the standard ways in which most organizations try to deal with keeping up with SSLcertificate expirations)
  28. 28. So Certificates ... • SSL Certificates expire • Millions of people can’t stream • Hilarity ensues • Standard Ways to Solve This • Excel worksheets • Wiki documents @royrapoport rsr@netflix.comFriday, March 22, 13(obviously, the ‘standard ways to solve this’ part here is somewhat facetious, but these are, infact, the standard ways in which most organizations try to deal with keeping up with SSLcertificate expirations)
  29. 29. So Certificates ... • SSL Certificates expire • Millions of people can’t stream • Hilarity ensues • Standard Ways to Solve This • Excel worksheets • Wiki documents • Events on public calendars @royrapoport rsr@netflix.comFriday, March 22, 13(obviously, the ‘standard ways to solve this’ part here is somewhat facetious, but these are, infact, the standard ways in which most organizations try to deal with keeping up with SSLcertificate expirations)
  30. 30. Let’s Do This Thing Cassandra Certificate @royrapoport rsr@netflix.comFriday, March 22, 13Start with a very simple model -- a Certificate entity, which is really just a combination ofname, expiration date, and a series of locations where we can find this. It’d be trivial to feedthis thing from my todo list, if I wanted to (but given the state of my todo list, probably a badidea)
  31. 31. Let’s Do This Thing ELB Cassandra Certificate @royrapoport rsr@netflix.comFriday, March 22, 13Then start building location-aware spiders -- e.g. this spider that knows how to probe all ourELBs to see if they listen on 443 and gets their certificate if they do.
  32. 32. Let’s Do This Thing ELB Cassandra EC2 Instance Certificate @royrapoport rsr@netflix.comFriday, March 22, 13Or this spider that knows how to talk to a specific kind of EC2 instance we have with somecertificates.
  33. 33. Let’s Do This Thing ELB Cassandra EC2 Instance IP Range Certificate @royrapoport rsr@netflix.comFriday, March 22, 13etc ...
  34. 34. Let’s Do This Thing ELB Cassandra EC2 Instance IP Range Certificate Filesystem @royrapoport rsr@netflix.comFriday, March 22, 13
  35. 35. Let’s Do This Thing ELB Cassandra EC2 Instance IP Range Certificate Filesystem DNS @royrapoport rsr@netflix.comFriday, March 22, 13
  36. 36. Let’s Do This Thing ELB Cassandra EC2 Instance IP Range Certificate Filesystem DNS @royrapoport rsr@netflix.comFriday, March 22, 13Once you have all this information, you can easily generate a web page showing certificates,where they are, and when they expire
  37. 37. Let’s Do This Thing ELB Cassandra EC2 Instance IP Range Certificate Filesystem DNS @royrapoport rsr@netflix.comFriday, March 22, 13And send out emails, too -- once we built the capability for teams to subscribe to emails fora given certificate and specify how many days before expiration they should start gettingnotified
  38. 38. Since Then @royrapoport rsr@netflix.comFriday, March 22, 13We validated the design by continuing to iterate on it -- recently, when building the DNSspider component, that work took only about 15 minutes to implement. We also expandedsubscription capabilities so teams could subscribe to certificate expiration warnings based oncertificate name regular expressions.
  39. 39. Since Then • No Production Emergencies due to SSL certificate expiration @royrapoport rsr@netflix.comFriday, March 22, 13We validated the design by continuing to iterate on it -- recently, when building the DNSspider component, that work took only about 15 minutes to implement. We also expandedsubscription capabilities so teams could subscribe to certificate expiration warnings based oncertificate name regular expressions.
  40. 40. Since Then • No Production Emergencies due to SSL certificate expiration • Validated Design @royrapoport rsr@netflix.comFriday, March 22, 13We validated the design by continuing to iterate on it -- recently, when building the DNSspider component, that work took only about 15 minutes to implement. We also expandedsubscription capabilities so teams could subscribe to certificate expiration warnings based oncertificate name regular expressions.
  41. 41. Since Then • No Production Emergencies due to SSL certificate expiration • Validated Design • Better Subscription Capabilities @royrapoport rsr@netflix.comFriday, March 22, 13We validated the design by continuing to iterate on it -- recently, when building the DNSspider component, that work took only about 15 minutes to implement. We also expandedsubscription capabilities so teams could subscribe to certificate expiration warnings based oncertificate name regular expressions.
  42. 42. Soon ... @royrapoport rsr@netflix.comFriday, March 22, 13We should be able to figure out who owns a certificate, most of the time, and alert themdirectly even if they don’t set up a subscription.
  43. 43. Soon ... • Customized, automated alerting @royrapoport rsr@netflix.comFriday, March 22, 13We should be able to figure out who owns a certificate, most of the time, and alert themdirectly even if they don’t set up a subscription.
  44. 44. Soon ... • Customized, automated alerting • Automated renewal @royrapoport rsr@netflix.comFriday, March 22, 13We should be able to figure out who owns a certificate, most of the time, and alert themdirectly even if they don’t set up a subscription.
  45. 45. Soon ... • Customized, automated alerting • Automated renewal • Telling you a problem is about to happen: Good @royrapoport rsr@netflix.comFriday, March 22, 13We should be able to figure out who owns a certificate, most of the time, and alert themdirectly even if they don’t set up a subscription.
  46. 46. Soon ... • Customized, automated alerting • Automated renewal • Telling you a problem is about to happen: Good • Preventing the problem automatically: Priceless @royrapoport rsr@netflix.comFriday, March 22, 13We should be able to figure out who owns a certificate, most of the time, and alert themdirectly even if they don’t set up a subscription.
  47. 47. Soon ... • Customized, automated alerting • Automated renewal • Telling you a problem is about to happen: Good • Preventing the problem automatically: Priceless • Open Source @royrapoport rsr@netflix.comFriday, March 22, 13We should be able to figure out who owns a certificate, most of the time, and alert themdirectly even if they don’t set up a subscription.
  48. 48. Remember ... @royrapoport rsr@netflix.comFriday, March 22, 13
  49. 49. Remember ... • Be Lazy @royrapoport rsr@netflix.comFriday, March 22, 13
  50. 50. Remember ... • Be Lazy • Help Others Be Lazy @royrapoport rsr@netflix.comFriday, March 22, 13
  51. 51. Remember ... • Be Lazy • Help Others Be Lazy • Computers Are Better Than Humans @royrapoport rsr@netflix.comFriday, March 22, 13
  52. 52. Remember ... • Be Lazy • Help Others Be Lazy • Computers Are Better Than Humans • For some things @royrapoport rsr@netflix.comFriday, March 22, 13
  53. 53. Remember ... • Be Lazy • Help Others Be Lazy • Computers Are Better Than Humans • For some things • Don’t compete on their terms @royrapoport rsr@netflix.comFriday, March 22, 13
  54. 54. Questions? @royrapoport rsr@netflix.comFriday, March 22, 13
  1. Gostou de algum slide específico?

    Recortar slides é uma maneira fácil de colecionar informações para acessar mais tarde.

×