Living on the edge at Netflix - Adrian Cole


Published on

Life on the edge is innovating a system that is ultimately responsible for 1/3 of all internet bandwidth usage in the US. Ever wonder how Netflix does it? This talk will overview a combination of services and open source projects that comprise the edge network, including Neflix Denominator. Discussions will include how we deal with DNS, distributed load balancing as well as discovery of mid-tier services. You'll understand which service components are involved with edge traffic management. When you leave, you'll have enough context to create your own life on the Edge with NetflixOSS!

Published in: Technology

Living on the edge at Netflix - Adrian Cole

  1. 1. Living on the EdgeJune 2013Adrian Cole@adrianfcole #netflixoss @denominatorOSS
  2. 2. How Netflix Streaming WorksQuick DNS PrimerDenominator Deep DiveCloud Prize
  3. 3. Netflix Member Web Site Home PagePersonalization Driven – What goes on to make this?
  4. 4. How Netflix Streaming WorksCustomer Device(PC, PS3, TV…)Web Site orDiscovery APIUser DataPersonalizationStreaming APIDRMQoS LoggingOpenConnectCDN BoxesCDNManagement andSteeringContent EncodingConsumerElectronicsAWS CloudServicesCDN EdgeLocations
  5. 5. Content Delivery ServiceOpen Source Hardware Design + FreeBSD, bird, nginx
  6. 6. November 2012 Traffic
  7. 7. Real Web Server Dependencies Flow(Netflix Home page business transaction as seen by AppDynamics)Start HerememcachedCassandraWeb serviceS3 bucketThree Personalization movie groupchoosers (for US, Canada and Latam)Each icon isthree to a fewhundredinstancesacross threeAWS zones
  8. 8. Asgard
  9. 9. Portable DNS ControlCassandra ReplicasZone ACassandra ReplicasZone BCassandra ReplicasZone CRegional Load BalancersCassandra ReplicasZone ACassandra ReplicasZone BCassandra ReplicasZone CRegional Load BalancersUltraDNSAWSRoute53A portable way to manage multiple DNS providers from JavaDenominatorDynECT
  11. 11. A Cloud Native Open Source Platform
  12. 12. Feature Set• Do stuff in batches• Cleanly handle advanced records• Play nice with persistence• Don’t do too much• … Use cool things
  13. 13. Hello Denominatormgr = Denominator.create("ultradns”,credentials(username, password))for (Iterator<String> zone =mgr.getApi().getZoneApi().list();zone.hasNext();) {processZone(;}
  14. 14. DNS Lingo• Zone– Name (ex.• Records– Name (ex.– Type (ex. CNAME)– TTL (ex. 300)– RData (ex.
  15. 15. Record Set• Records visible to the resolver that have thesame name and type (also TTL).• Ex. If has 4 ipaddresses, they can be in the same recordset.• Concept is helpful for DNSSEC
  16. 16. DNS Things• Nameserver– Server that listens on port 53 for queries• Resolver– Client that makes queries• API– Creates and controls configuration and data onthe nameservers.
  17. 17.
  18. 18. We wire with Dagger@Provides@SingletonRoute53Api provideApi(Supplier<Credentials>credentials) {return ContextBuilder.newBuilder(“route53”).credentialsSupplier(credentials).buildApi(Route53Api.class);}
  19. 19. We adapt vendor-specifics• Backends are jclouds apis, except mock.• The back the denominator model@Provides@SingletonZoneApi provideZoneApi(DynECTApi jclouds) {return new DynECTZoneApi(jclouds);}
  20. 20. Denominator ModelExtensible types are Forwarding MapsmxData.getPreference()mxData.get("preference”)profile.getType()profile.get(“type”)
  21. 21. ResourceRecordSet ProfilesServer-side configuration for advancedfeatures like health-checks and visibility.rrset.getProfiles().size() == 0 // not specialrrset.getProfiles().get(0).get(“type”)// could be “geo”
  22. 22. Availability by reshaping traffic
  23. 23. Geo ProfileVisible to clients in specific territoriesgeo.getType() == “geo” // or geo.get(”type”)geo.getGroup() // like US-Westgeo.getTerritories() // subset of territories
  24. 24. Example code// select the existing territories in USexisting = geoApi.getByNameTypeAndGroup("", "CNAME", "US");// refine to exclude californiaGeo existingGeo = toProfile(Geo.class).apply(existing);Multimap<String, String> update = filterValues(existingGeo.getRegions(), not(equalTo("California")));// apply the updategeoApi.applyRegionsToNameTypeAndGroup(update, "", "CNAME", "US");
  25. 25. Airline
  26. 26. Our CLI is Airline./denominator -p route53 record add --name --typeA --ec2-public-ipv4
  27. 27. Airline is a git-like CLI builderWe use a process called “really executable jar”so that ./denominator works.class GeoResourceRecordSetCommand extendsDenominatorCommand {@Option(type = OptionType.GROUP,required = true,name = { "-z", "--zone" },description = "zone name to affect. ex.")public String zoneName;}
  29. 29. Netflix Cloud PrizeBoosting the @denominatorOSSEcosystem
  30. 30. EntrantsNetflixEngineeringJudges WinnersNominationsConforms toRulesWorkingCodeCommunityTractionCategoriesRegistrationOpenedMarch 13GithubApacheLicensedContributionsGithubClose EntriesSeptember 15GithubAwardCeremonyDinnerNovemberAWSRe:InventTen PrizeCategories$10K cash$5K AWSAWSRe:InventTicketsTrophy
  31. 31. Functionality and scale now, portability comingMoving from parts to a platform in 2013Netflix is fostering an ecosystemRapid Evolution - Low MTBIAMSH(Mean Time Between Idea And Making Stuff Happen)
  32. 32. TakeawayDenominator is a multi-cloud DNS abstraction built as a library and a cli.Fixing DNS APIs, one vendor at a time!forum/denominator-dev!forum/denominator-user #netflixoss @denominatorOSS