Successfully reported this slideshow.

Living on the edge

3

Share

1 of 44
1 of 44

More Related Content

Related Books

Free with a 14 day trial from Scribd

See all

Living on the edge

  1. 1. Living on the Edge Adrian Cole @adrianfcole #netflixoss @denominatorOSS http://www.linkedin.com/in/adrianforrestcole
  2. 2. adrian • engineer at Square • founded apache jclouds • focus on (small) libraries * Worked exclusively on what’s in this deck while at Netflix!
  3. 3. How Netflix Streaming Works Geo DNS at Netflix Denominator
  4. 4. Netflix Member Web Site Home Page Personalization Driven – What goes on to make this?
  5. 5. How Netflix Streaming Works Consumer Electronics User Data Web Site or Discovery API AWS Cloud Services Personalization CDN Edge Locations DRM Customer Device (PC, PS3, TV…) Streaming API QoS Logging OpenConnect CDN Boxes CDN Management and Steering Content Encoding
  6. 6. Content Delivery Service Open Source Hardware Design + FreeBSD, bird, nginx
  7. 7. November 2012 Traffic
  8. 8. Real Web Server Dependencies Flow (Netflix Home page business transaction as seen by AppDynamics) Each icon is three to a few hundred instances across three AWS zones Cassandra memcached Start Here Three Personalization movie group choosers (for US, Canada and Latam) Web service S3 bucket
  9. 9. Netflix entrypoints are Geo DNS Geo == Directional
  10. 10. DNS Things • Nameserver – Server that listens on port 53 for queries • Resolver – Client that makes queries • API – Creates and controls configuration and data on the nameservers.
  11. 11. DNS Lingo • Zone – Name (ex. denominator.io.) – Id (if ambiguous) • Records – Name (ex. www.denominator.io.) – Type (ex. CNAME) – TTL (ex. 300) – RData (ex. myLB-1234567890.us-east1.elb.amazonaws.com)
  12. 12. Record Set • Records visible to the resolver that have the same name and type (also TTL). • Ex. If www.denominator.io has 4 ip addresses, they can be in the same recordset. • Concept is helpful for DNSSEC
  13. 13. Geo (Directional) Record Set Visible to clients in specific territories Group or Qualifier: maybe “US-WEST” Territories: subset directed to this rrset
  14. 14. EU-WEST-1 cbp.nccp.netflix.com Zone A Western US or Canada? Zone C Cassandra Replicas Europe? Zone B Cassandra Replicas Cassandra Replicas Everybody Else US-EAST-1 US-WEST-2 Zone A Zone B Zone C Zone A Zone B Zone C Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas
  15. 15. Changing territories reshapes traffic
  16. 16. DNS for Region Failover aka: why this deck is labeled advanced
  17. 17. Thanks! UltraDNS jonbodner Route53 colmmacc (github ids)
  18. 18. Route53 Alias Reference to an AWS resource such as a load balancer (ELB). Appears to the client as an ddress record set. if IPv4 or for IPv6
  19. 19. Example setup for failover • Point Geo record set to a normal, site-specific . • These s point to a Route53 ELB lias.
  20. 20. cbp.nccp.netflix.com US-WEST-2 US-EAST-1 cbp.nccp.us-west-2.dynprod.netflix.com cbp.nccp.us-east-1.dynprod.netflix.com cbp.nccp.us-west-2.dynprod.netflix.net cbp.nccp.us-east-1.dynprod.netflix.net nccp-cbp-frontend1065034783.us-west2.elb.amazonaws.com. nccp-cbp-frontend512191143.us-east1.elb.amazonaws.com.
  21. 21. (at least) 2 failover options • When Route53 API is available • When Route53 API isn’t available, but your normal DNS is
  22. 22. On Failover When Route53 API is available… Update the route53 alias to point to the healthy region.
  23. 23. cbp.nccp.netflix.com US-WEST-2 US-EAST-1 cbp.nccp.us-west-2.dynprod.netflix.com cbp.nccp.us-east-1.dynprod.netflix.com cbp.nccp.us-west-2.dynprod.netflix.net cbp.nccp.us-east-1.dynprod.netflix.net nccp-cbp-frontend1065034783.us-west2.elb.amazonaws.com. nccp-cbp-frontend512191143.us-east1.elb.amazonaws.com. US-EAST-1 alias indirectly points to the same ELB
  24. 24. On Failover When Route53 API isn’t available, but your normal DNS is… Update normal CNAME for each for each host and geo group pointing to a healthy ELB.
  25. 25. cbp.nccp.netflix.com US-WEST-2 US-EAST-1 cbp.nccp.us-west-2.dynprod.netflix.com cbp.nccp.us-east-1.dynprod.netflix.com cbp.nccp.us-west-2.dynprod.netflix.net cbp.nccp.us-east-1.dynprod.netflix.net dualstack.nccp-cbp-frontend1065034783.us-west2.elb.amazonaws.com. dualstack.nccp-cbpfrontend-512191143.useast-1.elb.amazonaws.com. US-EAST-1 CNAME points to the US-WEST-1 CNAME
  26. 26. Implications • You are pointing to a healthy region, not a specific ELB – No read-lookups needed – Can increase ELBs in healthy region • More setup needed – Must create Geo + region * (normal CNAME + A, AAAA Aliases in Route53)
  27. 27. PORTABLE CONTROL OF DNS CLOUDS
  28. 28. A Cloud Native Open Source Platform
  29. 29. Feature Set • • • • Do stuff in batches Cleanly handle advanced records Play nice with persistence Don’t do too much • … Use cool things like Dagger
  30. 30. Model ResourceRecordSet is the central class Record types (A, CNAME, etc) extend Map<String, Object> mxData.preference() mxData.get("preference”)
  31. 31. Hello Denominator get denominator from bintray or homebrew create ~/.denominatorconfig name: ultradns-prod provider: ultradns credentials: username: your_user password: your_password denominator -n ultradns-prod zone list
  32. 32. Basic list $ denominator -n ultradns-prod zone [UltraDNS#accountId] ---> POST https://ultraapi.ultradns.com:8443/UltraDNS_WS/v01 HTTP/1.1 [UltraDNS#accountId] <--- HTTP/1.1 200 OK (2062ms) [UltraDNS#zonesOfAccount] ---> POST https://ultraapi.ultradns.com:8443/UltraDNS_WS/v01 HTTP/1.1 [UltraDNS#zonesOfAccount] <--- HTTP/1.1 200 OK (2169ms)
  33. 33. Add Record $ denominator -n ultradns-test record -z ultradnstest.denominator.io. replace -n www.ultradnstest.denominator.io. -t A -d 192.0.2.1 [UltraDNS#recordsInZoneByNameAndType] ---> POST https://ultraapi.ultradns.com:8443/UltraDNS_WS/v01 HTTP/1.1 [UltraDNS#recordsInZoneByNameAndType] <--- HTTP/1.1 200 OK (1663ms) [UltraDNS#createRRPoolInZoneForNameAndType] ---> POST https://ultraapi.ultradns.com:8443/UltraDNS_WS/v01 HTTP/1.1 [UltraDNS#createRRPoolInZoneForNameAndType] <--- HTTP/1.1 200 OK (2108ms) [UltraDNS#createRecordInRRPoolInZone] ---> POST https://ultraapi.ultradns.com:8443/UltraDNS_WS/v01 HTTP/1.1 [UltraDNS#createRecordInRRPoolInZone] <--- HTTP/1.1 200 OK (3263ms)
  34. 34. From Java mgr = Denominator.create(”ultradns”, (username, password)) for (Zone zone : mgr.api(). processZone(zone); } mgr. . ()) { (“denominator.io.”) (a("www.denominator.io.", 300, "192.0.2.1"));
  35. 35. Thanks! adrianco jdamick colmmacc everett-toews digitalsanctum quidryan cfieber davidmc24 (github ids)
  36. 36. Takeaway Geo (Directional) DNS helps you manage the flow of traffic based on location. Vendors engagement in OSS >> better place for availability Denominator is a multi-cloud DNS abstraction built as a library and a cli. https://github.com/Netflix/denominator https://groups.google.com/forum/#!forum/denominator-user http://www.linkedin.com/in/adrianforrestcole @adrianfcole #netflixoss @denominatorOSS
  37. 37. Denominator Sidebar Dagger A fast dependency injector for Android and Java.
  38. 38. Dagger • Guice for libraries, particularly android • Speed and Simplicity over features • Extension averse, feature conservative • Friendly forks
  39. 39. Declare Dependencies class DNSAPIManager { } @Inject ZoneApi zoneApi; ... class Route53ZoneApi implements ZoneApi { } @Inject Route53 route53; ...
  40. 40. Satisfy Dependencies @Module(injects = DNSApiManager.class … class Route53Module { @Provides ZoneApi zoneApi(Route53ZoneApi zone) { return zones; } @Provides @Singleton Route53 route53(Feign feign, Route53Target target) { return feign.newInstance(target); } ...
  41. 41. Create the Graph manager = Denominator.create(new Route53Module()); class Denominator { public static DNSApiManager create(Object module) { ObjectGraph objectGraph = ObjectGraph.create(module); return objectGraph.get(DNSApiManager.class); } ... }
  42. 42. javac with compiler dependencies { compile "com.squareup.dagger:dagger” // 52k! provided "com.squareup.dagger:dagger-compiler” ... } Dagger’s compiler writes binding classes instead of reflection binding at runtime.
  43. 43. Takeaway Dagger is a leaner version of Guice, great for android and libraries. http://square.github.io/dagger/ https://groups.google.com/forum/#!forum/dagger-discuss

Editor's Notes

  • Adrian Cockroft championed the project in Netflix, also acting as a design mentor.Jeff Damick was the first pre-oss developer, also helping adjust UltraDNS.Colm helped design data structures and vetted concepts against Amazon Route53. Everett helped implement OpenStack and Rackspace DNS services.Shane helped shine up the commandline tool, particularly yaml config.Justin helped with single-jar syntax for gradleCameron wrote gradle scripts to setup Eclipse and Idea for Dagger processingDavid maintains Feign, the http interface layer used by denominator.
  • ×