Asgard, the Grails App that Deploys Netflix to the Cloud

// Feed your brain GR8Conf Europe 2012 gr8.technologies.each{ June 6th-8th yourBrain << it Copenhagen, Denmark }The Grails App that Deploys Netﬂix to the CloudJoe Sondow, Netﬂix @joesondow #gr8conf

Slides onlinehttp://slideshare.net/joesondow @joesondow #gr8conf

Agenda

AgendaWho am I? Why am I here?

AgendaWho am I? Why am I here?Asgard, Screen shots

AgendaWho am I? Why am I here?Asgard, Screen shotsNetﬂix, Freedom and Responsibility

AgendaWho am I? Why am I here?Asgard, Screen shotsNetﬂix, Freedom and ResponsibilityRegional URLs

AgendaWho am I? Why am I here?Asgard, Screen shotsNetﬂix, Freedom and ResponsibilityRegional URLsCloud Deployment Model

AgendaWho am I? Why am I here?Asgard, Screen shotsNetﬂix, Freedom and ResponsibilityRegional URLsCloud Deployment ModelOfﬂine Development

AgendaWho am I? Why am I here?Asgard, Screen shotsNetﬂix, Freedom and ResponsibilityRegional URLsCloud Deployment ModelOfﬂine DevelopmentWhy not the AWS Console?

Who am I? @joesondow #gr8conf

Who am I?Joe Sondow @joesondow #gr8conf

Who am I?Joe SondowNetﬂix since 2010 @joesondow #gr8conf

Who am I?Joe SondowNetﬂix since 2010Asgard lead @joesondow #gr8conf

Who am I?Joe SondowNetﬂix since 2010Asgard leadGrails @joesondow #gr8conf

Who am I?Joe SondowNetﬂix since 2010Asgard leadGrailsjQuery @joesondow #gr8conf

Why am I here? @joesondow #gr8conf

Why am I here?

Why am I here?Sell you something

Why am I here?Sell you somethingDiscuss business plans

Why am I here?Sell you somethingDiscuss business plansAnswer technical questions

Why am I here?Sell you somethingDiscuss business plansAnswer technical questionsBe a smaller ﬁsh in AWS

Why am I here?Sell you somethingDiscuss business plansAnswer technical questionsBe a smaller ﬁsh in AWSGive back to community

Why am I here?Sell you somethingDiscuss business plansAnswer technical questionsBe a smaller ﬁsh in AWSGive back to communitySteal your engineers

Asgard

AsgardScreen shots

AsgardApplication list

AsgardAuto Scaling Group list

AsgardCluster deployment, ready for fast rollback

Asgard

AsgardCloud deployment

AsgardCloud deploymentCloud management

AsgardCloud deploymentCloud managementStarted 2010

AsgardCloud deploymentCloud managementStarted 2010Open source in 2012

AsgardCloud deploymentCloud managementStarted 2010Open source in 2012 Follow @NetﬂixOSS

AsgardCloud deploymentCloud managementStarted 2010Open source in 2012 Follow @NetﬂixOSS100’s of Jira tickets

AsgardCloud deploymentCloud managementStarted 2010Open source in 2012 Follow @NetﬂixOSS100’s of Jira ticketsActively developed

Joe Sondow Clay McCoy Jason Gritman @joesondow @claymccoy @jgritmanAsgard TeamJune 2012(The dolphin writes integration tests)

With more than 25 million streaming members in theUnited States, Canada, Latin America, the UnitedKingdom and Ireland, Netﬂix, Inc. (NASDAQ: NFLX) isthe worlds leading internet subscription service forenjoying movies and TV programs.(from http://ir.netﬂix.com)

Freedom and Responsibility

Freedom and ResponsibilityCloud SOA

Freedom and ResponsibilityCloud SOA100’s of services

Freedom and ResponsibilityCloud SOA100’s of servicesSmall teams

Freedom and ResponsibilityCloud SOA100’s of servicesSmall teamsIndependent releases

Freedom and ResponsibilityCloud SOA100’s of servicesSmall teamsIndependent releasesControlled chaos

Regional URLs

Regional URLsLet’s see some code

Regional URLsCloud object URLs were unique, conventional, intuitiveDefault Grails URL mapping is excellent

Regional URLsLater, Netﬂix needed to use multiple Amazon regions

Regional URLsRegion is another URL dimensionAdd URL mapping, but don’t break existing links

Regional URLsRegionService.groovy (excerpt)

Regional URLsSome controllers should never be regionalRegionService detects RegionAgnostic annotation

Cloud deployment model

Cloud deployment modelApplications and Clusters

Cloud deployment model Auto Scaling Group

Cloud deployment model Auto Scaling Group Launch Conﬁguration

Cloud deployment model Elastic Load Balancer Auto Scaling Group Launch Conﬁguration

Cloud deployment model Elastic Load Balancer Auto Scaling Group Launch Conﬁguration Amazon Machine Image

Cloud deployment model Elastic Load Balancer Auto Scaling Group Security Group Launch Conﬁguration Amazon Machine Image

Cloud deployment model Elastic Load Balancer Auto Scaling Group Security Instances Group Launch Conﬁguration Amazon Machine Image

Cloud deployment model Search

Cloud deployment model API Search

Cloud deployment model Ratings API Search

Cloud deployment modelStreaming Starts Ratings API Search

Cloud deployment modelStreaming Starts Ratings API SearchAutocomplete

Cloud deployment model Sign UpStreaming Starts Ratings API SearchAutocomplete

Cloud deployment model Sign UpStreaming Starts Ratings Application Application Application API SearchAutocomplete Application Application Application

Inventing the Application

Inventing the Application Problem: Application is not an Amazon concept Solution: Create an Application domain in SimpleDB Enforce naming conventions on Amazon objects

Fast Rollback

Fast RollbackOptimism causes outages

Fast RollbackOptimism causes outagesProduction trafﬁc is unique

Fast RollbackOptimism causes outagesProduction trafﬁc is uniqueKeep old version running

Fast RollbackOptimism causes outagesProduction trafﬁc is uniqueKeep old version runningSwitch trafﬁc to new version

Fast RollbackOptimism causes outagesProduction trafﬁc is uniqueKeep old version runningSwitch trafﬁc to new versionMonitor results

Fast RollbackOptimism causes outagesProduction traffic is uniqueKeep old version runningSwitch traffic to new versionMonitor resultsRevert traffic quickly

Fast Rollback

Fast Rollback api-frontend api-usprod-v007

Fast Rollback api-frontend api-usprod-v007 api-usprod-v008

Fast Rollback api-frontend api-usprod-v007

Inventing the Cluster

Inventing the Cluster Problem: Two ASGs with one function but different names Solution: Append version number in reserved format Parse ASG name to determine long-term “cluster”

Inventing the ClusterInstead of keeping a database in sync, use namingconventions to store the source in truth in Amazon’s APIapi Applicationapi-usprod Clusterapi-usprod-v007 Auto Scaling Groupapi-usprod-v008 Auto Scaling Group

Ofﬂine Development

Ofﬂine DevelopmentMakes on a plane

Ofﬂine DevelopmentMock dataMock behaviorSystem property switch offline=true

Mock Datahttp://asgardprod/us-east-1/autoScaling/list.json

Mock DataParse JSON

Mock BehaviorOverride Amazon Java client methods

System Propertygrails run-app -Dofﬂine=true

Why not the AWS console?

Why not the AWS console?No, not to keep me busy

Why not the AWS console?

Why not the AWS console?Hide keys

Why not the AWS console?Hide keysCustomize model

Why not the AWS console?Hide keysCustomize modelEnforce conventions

Why not the AWS console?Hide keysCustomize modelEnforce conventionsAutomate workﬂow

Why not the AWS console?Hide keysCustomize modelEnforce conventionsAutomate workﬂowLog changes

Why not the AWS console?Hide keysCustomize modelEnforce conventionsAutomate workﬂowLog changesIntegrate systems

Why not the AWS console?Hide keysCustomize modelEnforce conventionsAutomate workﬂowLog changesIntegrate systemsCreate REST API

@NetﬂixOSS

@NetﬂixOSShttp://techblog.netﬂix.com

@NetﬂixOSShttp://techblog.netﬂix.comOpen source announcement...

@NetﬂixOSSThank you

@NetﬂixOSSThank youQuestions? @joesondow slideshare.net/joesondow linkedin.com/in/joesondow

http://gr8conf.eu	1071
http://groovyflow.com	822
http://www.scoop.it	318
https://twitter.com	41
http://www.groovyflow.com	38
http://localhost	37
http://feeds.feedburner.com	23
http://lanyrd.com	10
http://www.mefeedia.com	10
http://www.linkedin.com	4
http://gr8conf.org	3
http://webcache.googleusercontent.com	2
http://site7.miydim.com	1
https://si0.twimg.com	1
http://50.19.243.188	1
http://translate.googleusercontent.com	1
https://twimg0-a.akamaihd.net	1

Asgard, the Grails App that Deploys Netflix to the Cloud

by Joe Sondow on Jun 08, 2012

Accessibility

Categories

Upload Details

Usage Rights

17 Embeds 2,384

Statistics

Asgard, the Grails App that Deploys Netflix to the Cloud Presentation Transcript