1.
Living on the Edge
Adrian Cole
@adrianfcole #netflixoss @denominatorOSS
http://www.linkedin.com/in/adrianforrestcole

2.
adrian
• engineer at Square
• founded apache jclouds
• focus on (small) libraries
* Worked exclusively on what’s in this deck while at Netflix!

3.
How Netflix Streaming Works
Geo DNS at Netflix
Denominator

4.
Netflix Member Web Site Home Page
Personalization Driven – What goes on to make this?

5.
How Netflix Streaming Works
Consumer
Electronics
User Data
Web Site or
Discovery API
AWS Cloud
Services
Personalization
CDN Edge
Locations
DRM
Customer Device
(PC, PS3, TV…)
Streaming API
QoS Logging
OpenConnect
CDN Boxes
CDN
Management and
Steering
Content Encoding

6.
Content Delivery Service
Open Source Hardware Design + FreeBSD, bird, nginx

7.
November 2012 Traffic

8.
Real Web Server Dependencies Flow
(Netflix Home page business transaction as seen by AppDynamics)
Each icon is
three to a few
hundred
instances
across three
AWS zones
Cassandra
memcached
Start Here
Three Personalization movie group
choosers (for US, Canada and Latam)
Web service
S3 bucket

9.
Netflix entrypoints
are Geo DNS
Geo == Directional

10.
DNS Things
• Nameserver
– Server that listens on port 53 for queries
• Resolver
– Client that makes queries
• API
– Creates and controls configuration and data on
the nameservers.

11.
DNS Lingo
• Zone
– Name (ex. denominator.io.)
– Id (if ambiguous)
• Records
– Name (ex. www.denominator.io.)
– Type (ex. CNAME)
– TTL (ex. 300)
– RData (ex. myLB-1234567890.us-east1.elb.amazonaws.com)

12.
Record Set
• Records visible to the resolver that have the
same name and type (also TTL).
• Ex. If www.denominator.io has 4 ip
addresses, they can be in the same recordset.
• Concept is helpful for DNSSEC

13.
Geo (Directional) Record Set
Visible to clients in specific territories
Group or Qualifier: maybe “US-WEST”
Territories: subset directed to this rrset

14.
EU-WEST-1
cbp.nccp.netflix.com
Zone A
Western
US or
Canada?
Zone C
Cassandra Replicas
Europe?
Zone B
Cassandra Replicas
Cassandra Replicas
Everybody
Else
US-EAST-1
US-WEST-2
Zone A
Zone B
Zone C
Zone A
Zone B
Zone C
Cassandra Replicas
Cassandra Replicas
Cassandra Replicas
Cassandra Replicas
Cassandra Replicas
Cassandra Replicas

15.
Changing territories reshapes traffic

16.
DNS for Region
Failover
aka: why this deck is labeled
advanced

17.
Thanks!
UltraDNS jonbodner
Route53 colmmacc
(github ids)

18.
Route53 Alias
Reference to an AWS resource such as a
load balancer (ELB).
Appears to the client as an ddress record
set.
if IPv4 or
for IPv6

19.
Example setup for failover
• Point Geo record set to a normal, site-specific
.
• These
s point to a Route53 ELB lias.

20.
cbp.nccp.netflix.com
US-WEST-2
US-EAST-1
cbp.nccp.us-west-2.dynprod.netflix.com
cbp.nccp.us-east-1.dynprod.netflix.com
cbp.nccp.us-west-2.dynprod.netflix.net
cbp.nccp.us-east-1.dynprod.netflix.net
nccp-cbp-frontend1065034783.us-west2.elb.amazonaws.com.
nccp-cbp-frontend512191143.us-east1.elb.amazonaws.com.

21.
(at least) 2 failover options
• When Route53 API is available
• When Route53 API isn’t available, but your
normal DNS is

22.
On Failover
When Route53 API is available…
Update the route53 alias to point to the healthy
region.

23.
cbp.nccp.netflix.com
US-WEST-2
US-EAST-1
cbp.nccp.us-west-2.dynprod.netflix.com
cbp.nccp.us-east-1.dynprod.netflix.com
cbp.nccp.us-west-2.dynprod.netflix.net
cbp.nccp.us-east-1.dynprod.netflix.net
nccp-cbp-frontend1065034783.us-west2.elb.amazonaws.com.
nccp-cbp-frontend512191143.us-east1.elb.amazonaws.com.
US-EAST-1 alias indirectly points to the same ELB

24.
On Failover
When Route53 API isn’t available, but your
normal DNS is…
Update normal CNAME for each for each host
and geo group pointing to a healthy ELB.

25.
cbp.nccp.netflix.com
US-WEST-2
US-EAST-1
cbp.nccp.us-west-2.dynprod.netflix.com
cbp.nccp.us-east-1.dynprod.netflix.com
cbp.nccp.us-west-2.dynprod.netflix.net
cbp.nccp.us-east-1.dynprod.netflix.net
dualstack.nccp-cbp-frontend1065034783.us-west2.elb.amazonaws.com.
dualstack.nccp-cbpfrontend-512191143.useast-1.elb.amazonaws.com.
US-EAST-1 CNAME points to the US-WEST-1 CNAME

26.
Implications
• You are pointing to a healthy region, not a
specific ELB
– No read-lookups needed
– Can increase ELBs in healthy region
• More setup needed
– Must create Geo + region * (normal CNAME +
A, AAAA Aliases in Route53)

27.
PORTABLE CONTROL OF DNS CLOUDS

28.
A Cloud Native Open Source Platform

29.
Feature Set
•
•
•
•
Do stuff in batches
Cleanly handle advanced records
Play nice with persistence
Don’t do too much
• … Use cool things like Dagger

30.
Model
ResourceRecordSet is the central class
Record types (A, CNAME, etc) extend
Map<String, Object>
mxData.preference()
mxData.get("preference”)

31.
Hello Denominator
get denominator from bintray or homebrew
create ~/.denominatorconfig
name: ultradns-prod
provider: ultradns
credentials:
username: your_user
password: your_password
denominator -n ultradns-prod zone list

32.
Basic list
$ denominator -n ultradns-prod zone
[UltraDNS#accountId] ---> POST https://ultraapi.ultradns.com:8443/UltraDNS_WS/v01 HTTP/1.1
[UltraDNS#accountId] <--- HTTP/1.1 200 OK (2062ms)
[UltraDNS#zonesOfAccount] ---> POST https://ultraapi.ultradns.com:8443/UltraDNS_WS/v01 HTTP/1.1
[UltraDNS#zonesOfAccount] <--- HTTP/1.1 200 OK (2169ms)

33.
Add Record
$ denominator -n ultradns-test record -z ultradnstest.denominator.io.
replace -n www.ultradnstest.denominator.io. -t A -d 192.0.2.1
[UltraDNS#recordsInZoneByNameAndType] ---> POST https://ultraapi.ultradns.com:8443/UltraDNS_WS/v01 HTTP/1.1
[UltraDNS#recordsInZoneByNameAndType] <--- HTTP/1.1 200 OK (1663ms)
[UltraDNS#createRRPoolInZoneForNameAndType] ---> POST https://ultraapi.ultradns.com:8443/UltraDNS_WS/v01 HTTP/1.1
[UltraDNS#createRRPoolInZoneForNameAndType] <--- HTTP/1.1 200 OK
(2108ms)
[UltraDNS#createRecordInRRPoolInZone] ---> POST https://ultraapi.ultradns.com:8443/UltraDNS_WS/v01 HTTP/1.1
[UltraDNS#createRecordInRRPoolInZone] <--- HTTP/1.1 200 OK (3263ms)

34.
From Java
mgr = Denominator.create(”ultradns”,
(username, password))
for (Zone zone : mgr.api().
processZone(zone);
}
mgr.
.
()) {
(“denominator.io.”)
(a("www.denominator.io.", 300, "192.0.2.1"));

35.
Thanks!
adrianco jdamick colmmacc
everett-toews digitalsanctum
quidryan cfieber davidmc24
(github ids)

36.
Takeaway
Geo (Directional) DNS helps you manage the flow of traffic based on location.
Vendors engagement in OSS >> better place for availability
Denominator is a multi-cloud DNS abstraction built as a library and a cli.
https://github.com/Netflix/denominator
https://groups.google.com/forum/#!forum/denominator-user
http://www.linkedin.com/in/adrianforrestcole
@adrianfcole #netflixoss @denominatorOSS

37.
Denominator Sidebar
Dagger
A fast dependency injector for
Android and Java.

38.
Dagger
• Guice for libraries, particularly android
• Speed and Simplicity over features
• Extension averse, feature conservative
• Friendly forks

39.
Declare
Dependencies
class DNSAPIManager {
}
@Inject ZoneApi zoneApi;
...
class Route53ZoneApi implements ZoneApi {
}
@Inject Route53 route53;
...

40.
Satisfy Dependencies
@Module(injects = DNSApiManager.class …
class Route53Module {
@Provides
ZoneApi zoneApi(Route53ZoneApi zone) {
return zones;
}
@Provides
@Singleton
Route53 route53(Feign feign,
Route53Target target) {
return feign.newInstance(target);
}
...

41.
Create the Graph
manager = Denominator.create(new Route53Module());
class Denominator {
public static DNSApiManager create(Object module) {
ObjectGraph objectGraph
= ObjectGraph.create(module);
return objectGraph.get(DNSApiManager.class);
}
...
}

42.
javac with compiler
dependencies {
compile "com.squareup.dagger:dagger” // 52k!
provided "com.squareup.dagger:dagger-compiler”
...
}
Dagger’s compiler writes
binding classes instead of
reflection binding at runtime.

43.
Takeaway
Dagger is a leaner version of Guice, great for android
and libraries.
http://square.github.io/dagger/
https://groups.google.com/forum/#!forum/dagger-discuss