Living on the Edge
Adrian Cole
@adrianfcole #netflixoss @denominatorOSS
http://www.linkedin.com/in/adrianforrestcole
adrian

• engineer at Square
• founded apache jclouds
• focus on (small) libraries

* Worked exclusively on what’s in this...
How Netflix Streaming Works
Geo DNS at Netflix
Denominator
Netflix Member Web Site Home Page
Personalization Driven – What goes on to make this?
How Netflix Streaming Works
Consumer
Electronics

User Data
Web Site or
Discovery API

AWS Cloud
Services

Personalization...
Content Delivery Service
Open Source Hardware Design + FreeBSD, bird, nginx
November 2012 Traffic
Real Web Server Dependencies Flow
(Netflix Home page business transaction as seen by AppDynamics)
Each icon is
three to a ...
Netflix entrypoints
are Geo DNS
Geo == Directional
DNS Things
• Nameserver
– Server that listens on port 53 for queries

• Resolver
– Client that makes queries

• API
– Crea...
DNS Lingo
• Zone
– Name (ex. denominator.io.)
– Id (if ambiguous)

• Records
– Name (ex. www.denominator.io.)
– Type (ex. ...
Record Set
• Records visible to the resolver that have the
same name and type (also TTL).
• Ex. If www.denominator.io has ...
Geo (Directional) Record Set
Visible to clients in specific territories
Group or Qualifier: maybe “US-WEST”
Territories: s...
EU-WEST-1

cbp.nccp.netflix.com
Zone A

Western
US or
Canada?

Zone C

Cassandra Replicas

Europe?

Zone B
Cassandra Repli...
Changing territories reshapes traffic
DNS for Region
Failover
aka: why this deck is labeled
advanced
Thanks!
UltraDNS jonbodner
Route53 colmmacc
(github ids)
Route53 Alias
Reference to an AWS resource such as a
load balancer (ELB).
Appears to the client as an ddress record
set.
i...
Example setup for failover

• Point Geo record set to a normal, site-specific
.
• These
s point to a Route53 ELB lias.
cbp.nccp.netflix.com
US-WEST-2

US-EAST-1

cbp.nccp.us-west-2.dynprod.netflix.com

cbp.nccp.us-east-1.dynprod.netflix.com
...
(at least) 2 failover options

• When Route53 API is available
• When Route53 API isn’t available, but your
normal DNS is
On Failover
When Route53 API is available…
Update the route53 alias to point to the healthy
region.
cbp.nccp.netflix.com
US-WEST-2

US-EAST-1

cbp.nccp.us-west-2.dynprod.netflix.com

cbp.nccp.us-east-1.dynprod.netflix.com
...
On Failover
When Route53 API isn’t available, but your
normal DNS is…
Update normal CNAME for each for each host
and geo g...
cbp.nccp.netflix.com
US-WEST-2

US-EAST-1

cbp.nccp.us-west-2.dynprod.netflix.com

cbp.nccp.us-east-1.dynprod.netflix.com
...
Implications
• You are pointing to a healthy region, not a
specific ELB
– No read-lookups needed
– Can increase ELBs in he...
PORTABLE CONTROL OF DNS CLOUDS
A Cloud Native Open Source Platform
Feature Set
•
•
•
•

Do stuff in batches
Cleanly handle advanced records
Play nice with persistence
Don’t do too much

• …...
Model
ResourceRecordSet is the central class
Record types (A, CNAME, etc) extend
Map<String, Object>
mxData.preference()
m...
Hello Denominator
get denominator from bintray or homebrew

create ~/.denominatorconfig
name: ultradns-prod
provider: ultr...
Basic list
$ denominator -n ultradns-prod zone
[UltraDNS#accountId] ---> POST https://ultraapi.ultradns.com:8443/UltraDNS_...
Add Record
$ denominator -n ultradns-test record -z ultradnstest.denominator.io.
replace -n www.ultradnstest.denominator.i...
From Java
mgr = Denominator.create(”ultradns”,
(username, password))
for (Zone zone : mgr.api().
processZone(zone);
}
mgr....
Thanks!
adrianco jdamick colmmacc
everett-toews digitalsanctum
quidryan cfieber davidmc24
(github ids)
Takeaway
Geo (Directional) DNS helps you manage the flow of traffic based on location.
Vendors engagement in OSS >> better...
Denominator Sidebar

Dagger
A fast dependency injector for
Android and Java.
Dagger
• Guice for libraries, particularly android
• Speed and Simplicity over features
• Extension averse, feature conser...
Declare
Dependencies
class DNSAPIManager {

}

@Inject ZoneApi zoneApi;
...

class Route53ZoneApi implements ZoneApi {

}
...
Satisfy Dependencies
@Module(injects = DNSApiManager.class …
class Route53Module {
@Provides
ZoneApi zoneApi(Route53ZoneAp...
Create the Graph
manager = Denominator.create(new Route53Module());

class Denominator {
public static DNSApiManager creat...
javac with compiler
dependencies {
compile "com.squareup.dagger:dagger” // 52k!
provided "com.squareup.dagger:dagger-compi...
Takeaway
Dagger is a leaner version of Guice, great for android
and libraries.
http://square.github.io/dagger/
https://gro...
Living on the edge
Upcoming SlideShare
Loading in …5
×

Living on the edge

2,517 views

Published on

Overview of Netflix edge architecture, the DNS portability problem, and Denominator. Contains a sidebar on Square Dagger.

Some slides lifted from adrianco's dystopian deck.

Published in: Technology, Education
1 Comment
3 Likes
Statistics
Notes
No Downloads
Views
Total views
2,517
On SlideShare
0
From Embeds
0
Number of Embeds
530
Actions
Shares
0
Downloads
0
Comments
1
Likes
3
Embeds 0
No embeds

No notes for slide
  • Adrian Cockroft championed the project in Netflix, also acting as a design mentor.Jeff Damick was the first pre-oss developer, also helping adjust UltraDNS.Colm helped design data structures and vetted concepts against Amazon Route53. Everett helped implement OpenStack and Rackspace DNS services.Shane helped shine up the commandline tool, particularly yaml config.Justin helped with single-jar syntax for gradleCameron wrote gradle scripts to setup Eclipse and Idea for Dagger processingDavid maintains Feign, the http interface layer used by denominator.
  • Living on the edge

    1. 1. Living on the Edge Adrian Cole @adrianfcole #netflixoss @denominatorOSS http://www.linkedin.com/in/adrianforrestcole
    2. 2. adrian • engineer at Square • founded apache jclouds • focus on (small) libraries * Worked exclusively on what’s in this deck while at Netflix!
    3. 3. How Netflix Streaming Works Geo DNS at Netflix Denominator
    4. 4. Netflix Member Web Site Home Page Personalization Driven – What goes on to make this?
    5. 5. How Netflix Streaming Works Consumer Electronics User Data Web Site or Discovery API AWS Cloud Services Personalization CDN Edge Locations DRM Customer Device (PC, PS3, TV…) Streaming API QoS Logging OpenConnect CDN Boxes CDN Management and Steering Content Encoding
    6. 6. Content Delivery Service Open Source Hardware Design + FreeBSD, bird, nginx
    7. 7. November 2012 Traffic
    8. 8. Real Web Server Dependencies Flow (Netflix Home page business transaction as seen by AppDynamics) Each icon is three to a few hundred instances across three AWS zones Cassandra memcached Start Here Three Personalization movie group choosers (for US, Canada and Latam) Web service S3 bucket
    9. 9. Netflix entrypoints are Geo DNS Geo == Directional
    10. 10. DNS Things • Nameserver – Server that listens on port 53 for queries • Resolver – Client that makes queries • API – Creates and controls configuration and data on the nameservers.
    11. 11. DNS Lingo • Zone – Name (ex. denominator.io.) – Id (if ambiguous) • Records – Name (ex. www.denominator.io.) – Type (ex. CNAME) – TTL (ex. 300) – RData (ex. myLB-1234567890.us-east1.elb.amazonaws.com)
    12. 12. Record Set • Records visible to the resolver that have the same name and type (also TTL). • Ex. If www.denominator.io has 4 ip addresses, they can be in the same recordset. • Concept is helpful for DNSSEC
    13. 13. Geo (Directional) Record Set Visible to clients in specific territories Group or Qualifier: maybe “US-WEST” Territories: subset directed to this rrset
    14. 14. EU-WEST-1 cbp.nccp.netflix.com Zone A Western US or Canada? Zone C Cassandra Replicas Europe? Zone B Cassandra Replicas Cassandra Replicas Everybody Else US-EAST-1 US-WEST-2 Zone A Zone B Zone C Zone A Zone B Zone C Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas
    15. 15. Changing territories reshapes traffic
    16. 16. DNS for Region Failover aka: why this deck is labeled advanced
    17. 17. Thanks! UltraDNS jonbodner Route53 colmmacc (github ids)
    18. 18. Route53 Alias Reference to an AWS resource such as a load balancer (ELB). Appears to the client as an ddress record set. if IPv4 or for IPv6
    19. 19. Example setup for failover • Point Geo record set to a normal, site-specific . • These s point to a Route53 ELB lias.
    20. 20. cbp.nccp.netflix.com US-WEST-2 US-EAST-1 cbp.nccp.us-west-2.dynprod.netflix.com cbp.nccp.us-east-1.dynprod.netflix.com cbp.nccp.us-west-2.dynprod.netflix.net cbp.nccp.us-east-1.dynprod.netflix.net nccp-cbp-frontend1065034783.us-west2.elb.amazonaws.com. nccp-cbp-frontend512191143.us-east1.elb.amazonaws.com.
    21. 21. (at least) 2 failover options • When Route53 API is available • When Route53 API isn’t available, but your normal DNS is
    22. 22. On Failover When Route53 API is available… Update the route53 alias to point to the healthy region.
    23. 23. cbp.nccp.netflix.com US-WEST-2 US-EAST-1 cbp.nccp.us-west-2.dynprod.netflix.com cbp.nccp.us-east-1.dynprod.netflix.com cbp.nccp.us-west-2.dynprod.netflix.net cbp.nccp.us-east-1.dynprod.netflix.net nccp-cbp-frontend1065034783.us-west2.elb.amazonaws.com. nccp-cbp-frontend512191143.us-east1.elb.amazonaws.com. US-EAST-1 alias indirectly points to the same ELB
    24. 24. On Failover When Route53 API isn’t available, but your normal DNS is… Update normal CNAME for each for each host and geo group pointing to a healthy ELB.
    25. 25. cbp.nccp.netflix.com US-WEST-2 US-EAST-1 cbp.nccp.us-west-2.dynprod.netflix.com cbp.nccp.us-east-1.dynprod.netflix.com cbp.nccp.us-west-2.dynprod.netflix.net cbp.nccp.us-east-1.dynprod.netflix.net dualstack.nccp-cbp-frontend1065034783.us-west2.elb.amazonaws.com. dualstack.nccp-cbpfrontend-512191143.useast-1.elb.amazonaws.com. US-EAST-1 CNAME points to the US-WEST-1 CNAME
    26. 26. Implications • You are pointing to a healthy region, not a specific ELB – No read-lookups needed – Can increase ELBs in healthy region • More setup needed – Must create Geo + region * (normal CNAME + A, AAAA Aliases in Route53)
    27. 27. PORTABLE CONTROL OF DNS CLOUDS
    28. 28. A Cloud Native Open Source Platform
    29. 29. Feature Set • • • • Do stuff in batches Cleanly handle advanced records Play nice with persistence Don’t do too much • … Use cool things like Dagger
    30. 30. Model ResourceRecordSet is the central class Record types (A, CNAME, etc) extend Map<String, Object> mxData.preference() mxData.get("preference”)
    31. 31. Hello Denominator get denominator from bintray or homebrew create ~/.denominatorconfig name: ultradns-prod provider: ultradns credentials: username: your_user password: your_password denominator -n ultradns-prod zone list
    32. 32. Basic list $ denominator -n ultradns-prod zone [UltraDNS#accountId] ---> POST https://ultraapi.ultradns.com:8443/UltraDNS_WS/v01 HTTP/1.1 [UltraDNS#accountId] <--- HTTP/1.1 200 OK (2062ms) [UltraDNS#zonesOfAccount] ---> POST https://ultraapi.ultradns.com:8443/UltraDNS_WS/v01 HTTP/1.1 [UltraDNS#zonesOfAccount] <--- HTTP/1.1 200 OK (2169ms)
    33. 33. Add Record $ denominator -n ultradns-test record -z ultradnstest.denominator.io. replace -n www.ultradnstest.denominator.io. -t A -d 192.0.2.1 [UltraDNS#recordsInZoneByNameAndType] ---> POST https://ultraapi.ultradns.com:8443/UltraDNS_WS/v01 HTTP/1.1 [UltraDNS#recordsInZoneByNameAndType] <--- HTTP/1.1 200 OK (1663ms) [UltraDNS#createRRPoolInZoneForNameAndType] ---> POST https://ultraapi.ultradns.com:8443/UltraDNS_WS/v01 HTTP/1.1 [UltraDNS#createRRPoolInZoneForNameAndType] <--- HTTP/1.1 200 OK (2108ms) [UltraDNS#createRecordInRRPoolInZone] ---> POST https://ultraapi.ultradns.com:8443/UltraDNS_WS/v01 HTTP/1.1 [UltraDNS#createRecordInRRPoolInZone] <--- HTTP/1.1 200 OK (3263ms)
    34. 34. From Java mgr = Denominator.create(”ultradns”, (username, password)) for (Zone zone : mgr.api(). processZone(zone); } mgr. . ()) { (“denominator.io.”) (a("www.denominator.io.", 300, "192.0.2.1"));
    35. 35. Thanks! adrianco jdamick colmmacc everett-toews digitalsanctum quidryan cfieber davidmc24 (github ids)
    36. 36. Takeaway Geo (Directional) DNS helps you manage the flow of traffic based on location. Vendors engagement in OSS >> better place for availability Denominator is a multi-cloud DNS abstraction built as a library and a cli. https://github.com/Netflix/denominator https://groups.google.com/forum/#!forum/denominator-user http://www.linkedin.com/in/adrianforrestcole @adrianfcole #netflixoss @denominatorOSS
    37. 37. Denominator Sidebar Dagger A fast dependency injector for Android and Java.
    38. 38. Dagger • Guice for libraries, particularly android • Speed and Simplicity over features • Extension averse, feature conservative • Friendly forks
    39. 39. Declare Dependencies class DNSAPIManager { } @Inject ZoneApi zoneApi; ... class Route53ZoneApi implements ZoneApi { } @Inject Route53 route53; ...
    40. 40. Satisfy Dependencies @Module(injects = DNSApiManager.class … class Route53Module { @Provides ZoneApi zoneApi(Route53ZoneApi zone) { return zones; } @Provides @Singleton Route53 route53(Feign feign, Route53Target target) { return feign.newInstance(target); } ...
    41. 41. Create the Graph manager = Denominator.create(new Route53Module()); class Denominator { public static DNSApiManager create(Object module) { ObjectGraph objectGraph = ObjectGraph.create(module); return objectGraph.get(DNSApiManager.class); } ... }
    42. 42. javac with compiler dependencies { compile "com.squareup.dagger:dagger” // 52k! provided "com.squareup.dagger:dagger-compiler” ... } Dagger’s compiler writes binding classes instead of reflection binding at runtime.
    43. 43. Takeaway Dagger is a leaner version of Guice, great for android and libraries. http://square.github.io/dagger/ https://groups.google.com/forum/#!forum/dagger-discuss

    ×