Asgard: Using Grails to Deploy Netflix to AWS (Extended Slides)

// Feed your brain GR8Conf US 2012 gr8.technologies.each{ July 30-31 yourBrain << it Minneapolis, MN, US }The Grails App that Deploys Netﬂix to the CloudExtended Edition SlidesJoe Sondow, Netﬂix @joesondow #gr8conf

Slides onlinehttp://slideshare.net/joesondow @joesondow #gr8conf

Who am I? @joesondow #gr8conf

Who am I?Still time to sneak out to another session @joesondow #gr8conf

Who am I? @joesondow #gr8conf

Who am I?Joe Sondow @joesondow #gr8conf

Who am I?Joe SondowNetﬂix since 2010 @joesondow #gr8conf

Who am I?Joe SondowNetﬂix since 2010Asgard lead @joesondow #gr8conf

Who am I?Joe SondowNetﬂix since 2010Asgard leadGrails @joesondow #gr8conf

Who am I?Joe SondowNetﬂix since 2010Asgard leadGrailsjQuery @joesondow #gr8conf

Why am I here? @joesondow #gr8conf

Why am I here?

Why am I here?Sell you something

Why am I here?Sell you somethingDiscuss business plans

Why am I here?Sell you somethingDiscuss business plansAnswer technical questions

Why am I here?Sell you somethingDiscuss business plansAnswer technical questionsBe a smaller ﬁsh in AWS

Why am I here?Sell you somethingDiscuss business plansAnswer technical questionsBe a smaller ﬁsh in AWSGive back to community

Why am I here?Sell you somethingDiscuss business plansAnswer technical questionsBe a smaller ﬁsh in AWSGive back to communitySteal your engineers

Asgard

AsgardScreen shots

AsgardApplication list

AsgardAuto Scaling Group list

AsgardCluster deployment, ready for fast rollback

Asgard

AsgardApplication deployment

AsgardApplication deploymentCloud management

AsgardApplication deploymentCloud managementStarted 2010

AsgardApplication deploymentCloud managementStarted 2010Open source June 2012

AsgardApplication deploymentCloud managementStarted 2010Open source June 2012http://netﬂix.github.com

AsgardApplication deploymentCloud managementStarted 2010Open source June 2012http://netﬂix.github.com100’s of Jira tickets

AsgardApplication deploymentCloud managementStarted 2010Open source June 2012http://netﬂix.github.com100’s of Jira ticketsActively developed

User forumhttps://groups.google.com/group/AsgardUsers

Source code and downloadhttps://github.com/Netﬂix/asgard

Joe Sondow Clay McCoy Jason Gritman @joesondow @claymccoy @jgritmanAsgard TeamJuly 2012

With more than 25 million streaming members in theUnited States, Canada, Latin America, the UnitedKingdom and Ireland, Netﬂix, Inc. (NASDAQ: NFLX) isthe worlds leading internet subscription service forenjoying movies and TV programs.(from http://ir.netﬂix.com)

Freedom and Responsibility

Freedom and ResponsibilityCorporate culture and the Cloud

Freedom and Responsibility

Freedom and ResponsibilityCloud SOA

Freedom and ResponsibilityCloud SOA100’s of services

Freedom and ResponsibilityCloud SOA100’s of servicesSmall teams

Freedom and ResponsibilityCloud SOA100’s of servicesSmall teamsIndependent releases

Freedom and ResponsibilityCloud SOA100’s of servicesSmall teamsIndependent releasesControlled chaos

Regional URLs

Regional URLsLet’s see some code

Regional URLsCloud object URLs were unique, conventional, intuitiveDefault Grails URL mapping is excellent

Regional URLsLater, Netﬂix needed to use multiple Amazon regions

Regional URLsRegion is another URL dimensionAdd URL mapping, but don’t break existing links

Regional URLsRegionService.groovy

Regional URLsRegionService.groovyGrails may soon provide a better hook

Regional URLsSome controllers should never be regionalRegionService detects RegionAgnostic annotation

Cloud deployment model

Cloud deployment modelApplications and Clusters

Cloud deployment model Auto Scaling Group

Cloud deployment model Auto Scaling Group Launch Conﬁguration

Cloud deployment model Elastic Load Balancer Auto Scaling Group Launch Conﬁguration

Cloud deployment model Elastic Load Balancer Auto Scaling Group Launch Conﬁguration Amazon Machine Image

Cloud deployment model Elastic Load Balancer Auto Scaling Group Security Group Launch Conﬁguration Amazon Machine Image

Cloud deployment model Elastic Load Balancer Auto Scaling Group Security Instances Group Launch Conﬁguration Amazon Machine Image

Cloud deployment model Search

Cloud deployment model API Search

Cloud deployment model Ratings API Search

Cloud deployment modelStreaming Starts Ratings API Search

Cloud deployment model Streaming Starts RatingsAutocomplete API Search

Cloud deployment model Sign Up Streaming Starts RatingsAutocomplete API Search

Cloud deployment model Sign Up Streaming Starts Ratings Application Application ApplicationAutocomplete API Search Application Application Application

Inventing the Application

Inventing the Application Problem: Application is not an Amazon concept

Inventing the Application Problem: Application is not an Amazon concept Solution: Create an Application domain in SimpleDB Enforce naming conventions on Amazon objects

Fast Rollback

Fast RollbackOptimism causes outages

Fast RollbackOptimism causes outagesProduction trafﬁc is unique

Fast RollbackOptimism causes outagesProduction trafﬁc is uniqueKeep old version running

Fast RollbackOptimism causes outagesProduction trafﬁc is uniqueKeep old version runningSwitch trafﬁc to new version

Fast RollbackOptimism causes outagesProduction trafﬁc is uniqueKeep old version runningSwitch trafﬁc to new versionMonitor results

Fast RollbackOptimism causes outagesProduction traffic is uniqueKeep old version runningSwitch traffic to new versionMonitor resultsRevert traffic quickly

Fast Rollback

Fast Rollback api-frontend api-usprod-v007

Fast Rollback api-frontend api-usprod-v007 api-usprod-v008

Fast Rollback api-frontend api-usprod-v007

Inventing the Cluster

Inventing the Cluster Problem: Two ASGs with one function but different names

Inventing the Cluster Problem: Two ASGs with one function but different names Solution: Append version number in reserved format Parse ASG name to determine long-term “cluster”

Inventing the ClusterInstead of keeping a database in sync, use namingconventions to store the source in truth in Amazon’s APIapi Applicationapi-usprod Clusterapi-usprod-v007 Auto Scaling Groupapi-usprod-v008 Auto Scaling Group

Database Aversion

Database AversionStoring metadata on cloud objects

Database Aversion

Database Aversion Naming conventions

Database Aversion Naming conventions Tagging conventions

Database Aversion Naming conventions Tagging conventions No GORM domain objects

Database Aversion Naming conventions Tagging conventions No GORM domain objects AWS Java SDK

Database Aversion Naming conventions Tagging conventions No GORM domain objects AWS Java SDK Less to go out of sync

Database Aversion Naming conventions Tagging conventions No GORM domain objects AWS Java SDK Less to go out of sync Shared source of truth

Open Source, Closed Conﬁg

Open Source, Closed ConﬁgPull company-speciﬁc details out of Asgard

Open Source, Closed Conﬁg

Open Source, Closed Conﬁg Asgard for Netflix is configured to use company- specific extension points such as standard utility links for instances

Open Source, Closed Conﬁg Out-of-the-box Asgard installation has no instance utility links

Open Source, Closed Config Netflix specific $ASGARD_HOME/Config.groovylink { // Avoid GStrings here because these Strings are stored dynamic templates for arbitrary server names. String logUrlStart = http://${server}:7777 String configUrlStart = http://${server}:9999/AdminConfig instanceLinkGroupingsToLinkTemplateLists = [ Logs: [ new TextLinkTemplate(logUrlStart + /Admin/list?view=tomcat/catalina.out, catalina.out), new TextLinkTemplate(logUrlStart + /Admin/list, Log File Archive), new TextLinkTemplate(logUrlStart + /Admin/threaddumps, Thread Dumps), new TextLinkTemplate(logUrlStart + /AdminProxy, Admin Proxy Info), new TextLinkTemplate(logUrlStart + /AdminStatus, Admin Proxy Status), new TextLinkTemplate(logUrlStart + /GC/index, GC Visualization) ], Netflix Configuration: [ new TextLinkTemplate(configUrlStart + /prop.html, NetflixConfiguration Properties Console), new TextLinkTemplate(configUrlStart + /libs.html, Libraries Console), new TextLinkTemplate(configUrlStart + /machineProps, Machine Readable Properties), new TextLinkTemplate(configUrlStart + /webapp/META-INF/MANIFEST.MF, Manifest), ] ]}

Open Source, Closed Config grails-app/conf/Config.groovy references external configuration file ~/.asgard/Config.groovy https://github.com/Netflix/asgard/blob/master/grails-app/conf/Config.groovyasgardHome = System.getenv(ASGARD_HOME) ?: System.getProperty(ASGARD_HOME) ?: "${System.getProperty(user.home)}/.asgard"// Locations to search for config files that get merged into the main config.// Config files can either be Java properties files or ConfigSlurper scripts.grails.config.locations = [ "file:${asgardHome}/Config.groovy", classpath:sourceVersion.properties]

Open Source, Closed Config External Config.groovy also hold the AWS account credentials, or references for finding them.grails { awsAccounts=["178123456789"] awsAccountNames=["178123456789":"prod"]}secret { accessId="AKIAILAND0E3TMSJA8BA" secretKey="Od0AMY/dEC3P0/wZQjpmRtm2MVZc9330epDk+pwm"}cloud { accountName="prod" publicResourceAccounts=["amazon"]}

Caching the Cloud

Caching the CloudResponsive, massive, multi-region metadata

Caching the Cloud

Caching the Cloud Large counts

Caching the Cloud Large counts Many types

Caching the Cloud Large counts Many types Complex relationships

Caching the Cloud Large counts Many types Complex relationships Multiple regions

Caching the Cloud Large counts Many types Complex relationships Multiple regions Consistent single objects

Caching the Cloud Large counts Many types Complex relationships Multiple regions Consistent single objects Eventually consistent lists

Caching the Cloudclass Caches { final CachedMap<AppRegistration> allApplications final CachedMap<ApplicationMetrics> allApplicationMetrics final CachedMap<HardwareProfile> allHardwareProfiles final MultiRegionCachedMap<MetricAlarm> allAlarms final MultiRegionCachedMap<ApplicationInstance> allApplicationInstances final MultiRegionCachedMap<AutoScalingGroup> allAutoScalingGroups final MultiRegionCachedMap<AvailabilityZone> allAvailabilityZones final MultiRegionCachedMap<Cluster> allClusters final MultiRegionCachedMap<DBInstance> allDBInstances final MultiRegionCachedMap<DBSecurityGroup> allDBSecurityGroups final MultiRegionCachedMap<DBSnapshot> allDBSnapshots final MultiRegionCachedMap<String> allDomains final MultiRegionCachedMap<FastProperty> allFastProperties final MultiRegionCachedMap<Image> allImages final MultiRegionCachedMap<Instance> allInstances final MultiRegionCachedMap<InstanceTypeData> allInstanceTypes final MultiRegionCachedMap<KeyPairInfo> allKeyPairs final MultiRegionCachedMap<LaunchConfiguration> allLaunchConfigurations // etc. // etc. // etc.}

Caching the Cloudclass AwsRdsService implements CacheInitializer, InitializingBean { MultiRegionAwsClient<AmazonRDS> awsClient Caches caches void initializeCaches() { caches.allDBInstances.ensureSetUp({ Region region -> retrieveDBInstances(region) }) } private List<DBInstance> retrieveDBInstances(Region region) { awsClient.by(region).describeDBInstances(new DescribeDBInstancesRequest()).getDBInstances() } Collection<DBInstance> getDBInstances(UserContext userContext) { caches.allDBInstances.by(userContext.region).list() }}

Caching the Cloud

Visual Language for the Cloud

Visual Language for the CloudTango open source icons

Visual Language for the Cloud AWS is intimidating

Visual Language for the Cloud AWS is intimidating Many object types

Visual Language for the Cloud AWS is intimidating Many object types Help newbie users

Visual Language for the Cloud AWS is intimidating Many object types Help newbie users Reduce cognitive load

Visual Language for the Cloud AWS is intimidating Many object types Help newbie users Reduce cognitive load Make it easy

Visual Language for the Cloud AWS is intimidating Many object types Help newbie users Reduce cognitive load Make it easy Avoid surprises

Visual Language for the CloudAt a glance, these nav bar items look alike.

Visual Language for the CloudSome screens have multiple actionbuttons that look too similar.

Visual Language for the Cloud Because of naming conventions, these links look alike.

Visual Language for the Cloud The indicators for the current AWS region are too easy to miss.

Visual Language for the CloudThese availability zonesare important torecognize at a glancebut their names looksimilar, and they appearon many screens.

Tango Icons

Tango Iconshttp://tango.freedesktop.org/

Tango Iconshttp://tango.freedesktop.org/http://tango.freedesktop.org/Tango_Icon_Theme_Guidelines

Tango Iconshttp://tango.freedesktop.org/http://tango.freedesktop.org/Tango_Icon_Theme_Guidelineshttp://commons.wikimedia.org/wiki/Tango_icons

Tango Iconshttp://tango.freedesktop.org/http://tango.freedesktop.org/Tango_Icon_Theme_Guidelineshttp://commons.wikimedia.org/wiki/Tango_iconsUsed by Firefox, Jenkins, GIMP, OpenOfﬁce, VMWare

REST API in Grails

REST API in GrailsEnable external mashups with cloud data

REST API in Grails

REST API in GrailsApplicationController.groovy

Ofﬂine Development

Ofﬂine DevelopmentMakes on a plane

Ofﬂine DevelopmentMock dataMock behaviorSystem property switch offline=true

Mock Datahttp://asgardprod/us-east-1/autoScaling/list.json

Mock DataParse JSON

Mock BehaviorOverride Amazon Java client methods

System Propertygrails run-app -Dofﬂine=true

Why not the AWS console?

Why not the AWS console?No, not to keep me busy

Why not the AWS console?

Why not the AWS console?Hide keys

Why not the AWS console?Hide keysCustomize model

Why not the AWS console?Hide keysCustomize modelEnforce conventions

Why not the AWS console?Hide keysCustomize modelEnforce conventionsAutomate workﬂow

Why not the AWS console?Hide keysCustomize modelEnforce conventionsAutomate workﬂowLog changes

Why not the AWS console?Hide keysCustomize modelEnforce conventionsAutomate workﬂowLog changesIntegrate systems

Why not the AWS console?Hide keysCustomize modelEnforce conventionsAutomate workﬂowLog changesIntegrate systemsCreate REST API

@NetﬂixOSS

@NetﬂixOSShttp://techblog.netﬂix.com

@NetflixOSShttp://techblog.netflix.comhttp://netflix.github.com

http://github.com/Netﬂix/asgardThank you

http://github.com/Netﬂix/asgardThank youQuestions? github.com/joesondow @joesondow slideshare.net/joesondow linkedin.com/in/joesondow

http://www.scoop.it	444
https://twitter.com	8
http://us-w1.rockmelt.com	2
https://twimg0-a.akamaihd.net	1
http://webcache.googleusercontent.com	1
http://www.linkedin.com	1

Asgard: Using Grails to Deploy Netflix to AWS (Extended Slides)

by Joe Sondow on Aug 03, 2012

Accessibility

Categories

Upload Details

Usage Rights

6 Embeds 457

Statistics

Asgard: Using Grails to Deploy Netflix to AWS (Extended Slides) Presentation Transcript