// Feed your brain               GR8Conf US 2012        gr8.technologies.each{                July 30-31          yourBrai...
Slides onlinehttp://slideshare.net/joesondow   @joesondow #gr8conf
Who am I? @joesondow #gr8conf
Who am I?Still time to sneak out to another session   @joesondow #gr8conf
Who am I? @joesondow #gr8conf
Who am I?Joe Sondow @joesondow #gr8conf
Who am I?Joe SondowNetflix since 2010 @joesondow #gr8conf
Who am I?Joe SondowNetflix since 2010Asgard lead @joesondow #gr8conf
Who am I?Joe SondowNetflix since 2010Asgard leadGrails @joesondow #gr8conf
Who am I?Joe SondowNetflix since 2010Asgard leadGrailsjQuery @joesondow #gr8conf
Why am I here? @joesondow #gr8conf
Why am I here?
Why am I here?Sell you something
Why am I here?Sell you somethingDiscuss business plans
Why am I here?Sell you somethingDiscuss business plansAnswer technical questions
Why am I here?Sell you somethingDiscuss business plansAnswer technical questionsBe a smaller fish in AWS
Why am I here?Sell you somethingDiscuss business plansAnswer technical questionsBe a smaller fish in AWSGive back to commun...
Why am I here?Sell you somethingDiscuss business plansAnswer technical questionsBe a smaller fish in AWSGive back to commun...
Asgard
Asgard
AsgardScreen shots
AsgardApplication list
AsgardAuto Scaling Group list
AsgardCluster deployment, ready for fast rollback
Asgard
AsgardApplication deployment
AsgardApplication deploymentCloud management
AsgardApplication deploymentCloud managementStarted 2010
AsgardApplication deploymentCloud managementStarted 2010Open source June 2012
AsgardApplication deploymentCloud managementStarted 2010Open source June 2012http://netflix.github.com
AsgardApplication deploymentCloud managementStarted 2010Open source June 2012http://netflix.github.com100’s of Jira tickets
AsgardApplication deploymentCloud managementStarted 2010Open source June 2012http://netflix.github.com100’s of Jira tickets...
User forumhttps://groups.google.com/group/AsgardUsers
Source code and downloadhttps://github.com/Netflix/asgard
Joe Sondow   Clay McCoy   Jason Gritman  @joesondow   @claymccoy   @jgritmanAsgard TeamJuly 2012
With more than 25 million streaming members in theUnited States, Canada, Latin America, the UnitedKingdom and Ireland, Net...
Freedom and Responsibility
Freedom and ResponsibilityCorporate culture and the Cloud
Freedom and Responsibility
Freedom and ResponsibilityCloud SOA
Freedom and ResponsibilityCloud SOA100’s of services
Freedom and ResponsibilityCloud SOA100’s of servicesSmall teams
Freedom and ResponsibilityCloud SOA100’s of servicesSmall teamsIndependent releases
Freedom and ResponsibilityCloud SOA100’s of servicesSmall teamsIndependent releasesControlled chaos
Regional URLs
Regional URLsLet’s see some code
Regional URLsCloud object URLs were unique, conventional, intuitiveDefault Grails URL mapping is excellent
Regional URLsLater, Netflix needed to use multiple Amazon regions
Regional URLsRegion is another URL dimensionAdd URL mapping, but don’t break existing links
Regional URLsRegionService.groovy
Regional URLsRegionService.groovy
Regional URLsRegionService.groovyGrails may soon provide a better hook
Regional URLsSome controllers should never be regionalRegionService detects RegionAgnostic annotation
Cloud deployment model
Cloud deployment modelApplications and Clusters
Cloud deployment model
Cloud deployment model   Auto Scaling      Group
Cloud deployment model    Auto Scaling       Group    Launch  Configuration
Cloud deployment model                   Elastic Load                     Balancer    Auto Scaling       Group    Launch  ...
Cloud deployment model                    Elastic Load                      Balancer    Auto Scaling       Group    Launch...
Cloud deployment model                    Elastic Load                      Balancer    Auto Scaling       Group          ...
Cloud deployment model                    Elastic Load                      Balancer    Auto Scaling       Group          ...
Cloud deployment model                    Elastic Load                      Balancer    Auto Scaling       Group          ...
Cloud deployment model                    Elastic Load                      Balancer    Auto Scaling       Group          ...
Cloud deployment model                    Elastic Load                      Balancer    Auto Scaling       Group          ...
Cloud deployment model
Cloud deployment model                    Search
Cloud deployment model            API                    Search
Cloud deployment model                     Ratings            API                    Search
Cloud deployment modelStreaming Starts                         Ratings                   API                         Search
Cloud deployment model Streaming Starts                          RatingsAutocomplete        API                          S...
Cloud deployment model                    Sign Up Streaming Starts                              RatingsAutocomplete       ...
Cloud deployment model                      Sign Up Streaming Starts                                       Ratings        ...
Inventing the Application
Inventing the Application Problem: Application is not an Amazon concept
Inventing the Application Problem: Application is not an Amazon concept Solution: Create an Application domain in SimpleDB...
Fast Rollback
Fast RollbackOptimism causes outages
Fast RollbackOptimism causes outagesProduction traffic is unique
Fast RollbackOptimism causes outagesProduction traffic is uniqueKeep old version running
Fast RollbackOptimism causes outagesProduction traffic is uniqueKeep old version runningSwitch traffic to new version
Fast RollbackOptimism causes outagesProduction traffic is uniqueKeep old version runningSwitch traffic to new versionMonitor...
Fast RollbackOptimism causes outagesProduction traffic is uniqueKeep old version runningSwitch traffic to new versionMonitor...
Fast Rollback
Fast Rollback                     api-frontend   api-usprod-v007
Fast Rollback                     api-frontend   api-usprod-v007                  api-usprod-v008
Fast Rollback                     api-frontend   api-usprod-v007                  api-usprod-v008
Fast Rollback                     api-frontend   api-usprod-v007                  api-usprod-v008
Fast Rollback                     api-frontend   api-usprod-v007                  api-usprod-v008
Fast Rollback                     api-frontend   api-usprod-v007
Inventing the Cluster
Inventing the Cluster Problem: Two ASGs with one function but different names
Inventing the Cluster Problem: Two ASGs with one function but different names Solution: Append version number in reserved ...
Inventing the ClusterInstead of keeping a database in sync, use namingconventions to store the source in truth in Amazon’s...
Database Aversion
Database AversionStoring metadata on cloud objects
Database Aversion
Database Aversion Naming conventions
Database Aversion Naming conventions Tagging conventions
Database Aversion Naming conventions Tagging conventions No GORM domain objects
Database Aversion Naming conventions Tagging conventions No GORM domain objects AWS Java SDK
Database Aversion Naming conventions Tagging conventions No GORM domain objects AWS Java SDK Less to go out of sync
Database Aversion Naming conventions Tagging conventions No GORM domain objects AWS Java SDK Less to go out of sync Shared...
Open Source, Closed Config
Open Source, Closed ConfigPull company-specific details out of Asgard
Open Source, Closed Config
Open Source, Closed Config
Open Source, Closed Config                   Asgard for                   Netflix is                   configured to       ...
Open Source, Closed Config    Out-of-the-box Asgard installation    has no instance utility links
Open Source, Closed Config           Netflix specific $ASGARD_HOME/Config.groovylink {      // Avoid GStrings here because the...
Open Source, Closed Config grails-app/conf/Config.groovy references external configuration file ~/.asgard/Config.groovy https:/...
Open Source, Closed Config External Config.groovy also hold the AWS account credentials, or references for finding them.grail...
Caching the Cloud
Caching the CloudResponsive, massive, multi-region metadata
Caching the Cloud
Caching the Cloud Large counts
Caching the Cloud Large counts Many types
Caching the Cloud Large counts Many types Complex relationships
Caching the Cloud Large counts Many types Complex relationships Multiple regions
Caching the Cloud Large counts Many types Complex relationships Multiple regions Consistent single objects
Caching the Cloud Large counts Many types Complex relationships Multiple regions Consistent single objects Eventually cons...
Caching the Cloudclass Caches {    final CachedMap<AppRegistration> allApplications    final CachedMap<ApplicationMetrics> a...
Caching the Cloudclass Caches {    final CachedMap<AppRegistration> allApplications    final CachedMap<ApplicationMetrics> a...
Caching the Cloudclass AwsRdsService implements CacheInitializer, InitializingBean {    MultiRegionAwsClient<AmazonRDS> aw...
Caching the Cloudclass AwsRdsService implements CacheInitializer, InitializingBean {    MultiRegionAwsClient<AmazonRDS> aw...
Caching the Cloudclass AwsRdsService implements CacheInitializer, InitializingBean {    MultiRegionAwsClient<AmazonRDS> aw...
Caching the Cloudclass AwsRdsService implements CacheInitializer, InitializingBean {    MultiRegionAwsClient<AmazonRDS> aw...
Caching the Cloud
Visual Language for the Cloud
Visual Language for the CloudTango open source icons
Visual Language for the Cloud
Visual Language for the Cloud AWS is intimidating
Visual Language for the Cloud AWS is intimidating Many object types
Visual Language for the Cloud AWS is intimidating Many object types Help newbie users
Visual Language for the Cloud AWS is intimidating Many object types Help newbie users Reduce cognitive load
Visual Language for the Cloud AWS is intimidating Many object types Help newbie users Reduce cognitive load Make it easy
Visual Language for the Cloud AWS is intimidating Many object types Help newbie users Reduce cognitive load Make it easy A...
Visual Language for the Cloud
Visual Language for the CloudAt a glance, these nav bar items look alike.
Visual Language for the CloudAt a glance, these nav bar items look alike.
Visual Language for the Cloud
Visual Language for the CloudSome screens have multiple actionbuttons that look too similar.
Visual Language for the CloudSome screens have multiple actionbuttons that look too similar.
Visual Language for the Cloud
Visual Language for the Cloud  Because of naming conventions,  these links look alike.
Visual Language for the Cloud  Because of naming conventions,  these links look alike.
Visual Language for the Cloud
Visual Language for the Cloud           The indicators for the           current AWS region are           too easy to miss.
Visual Language for the Cloud           The indicators for the           current AWS region are           too easy to miss.
Visual Language for the CloudThese availability zonesare important torecognize at a glancebut their names looksimilar, and...
Visual Language for the CloudThese availability zonesare important torecognize at a glancebut their names looksimilar, and...
Visual Language for the Cloud
Tango Icons
Tango Iconshttp://tango.freedesktop.org/
Tango Iconshttp://tango.freedesktop.org/http://tango.freedesktop.org/Tango_Icon_Theme_Guidelines
Tango Iconshttp://tango.freedesktop.org/http://tango.freedesktop.org/Tango_Icon_Theme_Guidelineshttp://commons.wikimedia.o...
Tango Iconshttp://tango.freedesktop.org/http://tango.freedesktop.org/Tango_Icon_Theme_Guidelineshttp://commons.wikimedia.o...
REST API in Grails
REST API in GrailsEnable external mashups with cloud data
REST API in Grails
REST API in Grails
REST API in Grails
REST API in Grails
REST API in Grails
REST API in Grails
REST API in Grails
REST API in Grails
REST API in Grails
REST API in GrailsApplicationController.groovy
Offline Development
Offline DevelopmentMakes on a plane
Offline DevelopmentMock dataMock behaviorSystem property switch offline=true
Mock Datahttp://asgardprod/us-east-1/autoScaling/list.json
Mock DataParse JSON
Mock BehaviorOverride Amazon Java client methods
System Propertygrails run-app -Doffline=true
Why not the AWS console?
Why not the AWS console?No, not to keep me busy
Why not the AWS console?
Why not the AWS console?
Why not the AWS console?Hide keys
Why not the AWS console?Hide keysCustomize model
Why not the AWS console?Hide keysCustomize modelEnforce conventions
Why not the AWS console?Hide keysCustomize modelEnforce conventionsAutomate workflow
Why not the AWS console?Hide keysCustomize modelEnforce conventionsAutomate workflowLog changes
Why not the AWS console?Hide keysCustomize modelEnforce conventionsAutomate workflowLog changesIntegrate systems
Why not the AWS console?Hide keysCustomize modelEnforce conventionsAutomate workflowLog changesIntegrate systemsCreate REST...
@NetflixOSS
@NetflixOSShttp://techblog.netflix.com
@NetflixOSShttp://techblog.netflix.comhttp://netflix.github.com
http://github.com/Netflix/asgardThank you
http://github.com/Netflix/asgardThank youQuestions?  github.com/joesondow  @joesondow  slideshare.net/joesondow  linkedin.c...
Asgard: Using Grails to Deploy Netflix to AWS (Extended Slides)
Asgard: Using Grails to Deploy Netflix to AWS (Extended Slides)
Upcoming SlideShare
Loading in...5
×

Asgard: Using Grails to Deploy Netflix to AWS (Extended Slides)

3,090

Published on

Overview and technical exploration of Asgard, an open source, Grails-based, graphical web console created by Netflix for application deployment and cloud management in the Amazon cloud. Presented at the GR8 Conference in Minneapolis, MN, July 30, 2012.

PDF version: https://github.com/joesondow/GR8-US-2012/blob/master/Asgard-JoeSondow/Asgard-GR8Conf-US-2012-07.pdf

Published in: Technology, Business
0 Comments
11 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
3,090
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
103
Comments
0
Likes
11
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Asgard: Using Grails to Deploy Netflix to AWS (Extended Slides)

    1. 1. // Feed your brain GR8Conf US 2012 gr8.technologies.each{ July 30-31 yourBrain << it Minneapolis, MN, US }The Grails App that Deploys Netflix to the CloudExtended Edition SlidesJoe Sondow, Netflix @joesondow #gr8conf
    2. 2. Slides onlinehttp://slideshare.net/joesondow @joesondow #gr8conf
    3. 3. Who am I? @joesondow #gr8conf
    4. 4. Who am I?Still time to sneak out to another session @joesondow #gr8conf
    5. 5. Who am I? @joesondow #gr8conf
    6. 6. Who am I?Joe Sondow @joesondow #gr8conf
    7. 7. Who am I?Joe SondowNetflix since 2010 @joesondow #gr8conf
    8. 8. Who am I?Joe SondowNetflix since 2010Asgard lead @joesondow #gr8conf
    9. 9. Who am I?Joe SondowNetflix since 2010Asgard leadGrails @joesondow #gr8conf
    10. 10. Who am I?Joe SondowNetflix since 2010Asgard leadGrailsjQuery @joesondow #gr8conf
    11. 11. Why am I here? @joesondow #gr8conf
    12. 12. Why am I here?
    13. 13. Why am I here?Sell you something
    14. 14. Why am I here?Sell you somethingDiscuss business plans
    15. 15. Why am I here?Sell you somethingDiscuss business plansAnswer technical questions
    16. 16. Why am I here?Sell you somethingDiscuss business plansAnswer technical questionsBe a smaller fish in AWS
    17. 17. Why am I here?Sell you somethingDiscuss business plansAnswer technical questionsBe a smaller fish in AWSGive back to community
    18. 18. Why am I here?Sell you somethingDiscuss business plansAnswer technical questionsBe a smaller fish in AWSGive back to communitySteal your engineers
    19. 19. Asgard
    20. 20. Asgard
    21. 21. AsgardScreen shots
    22. 22. AsgardApplication list
    23. 23. AsgardAuto Scaling Group list
    24. 24. AsgardCluster deployment, ready for fast rollback
    25. 25. Asgard
    26. 26. AsgardApplication deployment
    27. 27. AsgardApplication deploymentCloud management
    28. 28. AsgardApplication deploymentCloud managementStarted 2010
    29. 29. AsgardApplication deploymentCloud managementStarted 2010Open source June 2012
    30. 30. AsgardApplication deploymentCloud managementStarted 2010Open source June 2012http://netflix.github.com
    31. 31. AsgardApplication deploymentCloud managementStarted 2010Open source June 2012http://netflix.github.com100’s of Jira tickets
    32. 32. AsgardApplication deploymentCloud managementStarted 2010Open source June 2012http://netflix.github.com100’s of Jira ticketsActively developed
    33. 33. User forumhttps://groups.google.com/group/AsgardUsers
    34. 34. Source code and downloadhttps://github.com/Netflix/asgard
    35. 35. Joe Sondow Clay McCoy Jason Gritman @joesondow @claymccoy @jgritmanAsgard TeamJuly 2012
    36. 36. With more than 25 million streaming members in theUnited States, Canada, Latin America, the UnitedKingdom and Ireland, Netflix, Inc. (NASDAQ: NFLX) isthe worlds leading internet subscription service forenjoying movies and TV programs.(from http://ir.netflix.com)
    37. 37. Freedom and Responsibility
    38. 38. Freedom and ResponsibilityCorporate culture and the Cloud
    39. 39. Freedom and Responsibility
    40. 40. Freedom and ResponsibilityCloud SOA
    41. 41. Freedom and ResponsibilityCloud SOA100’s of services
    42. 42. Freedom and ResponsibilityCloud SOA100’s of servicesSmall teams
    43. 43. Freedom and ResponsibilityCloud SOA100’s of servicesSmall teamsIndependent releases
    44. 44. Freedom and ResponsibilityCloud SOA100’s of servicesSmall teamsIndependent releasesControlled chaos
    45. 45. Regional URLs
    46. 46. Regional URLsLet’s see some code
    47. 47. Regional URLsCloud object URLs were unique, conventional, intuitiveDefault Grails URL mapping is excellent
    48. 48. Regional URLsLater, Netflix needed to use multiple Amazon regions
    49. 49. Regional URLsRegion is another URL dimensionAdd URL mapping, but don’t break existing links
    50. 50. Regional URLsRegionService.groovy
    51. 51. Regional URLsRegionService.groovy
    52. 52. Regional URLsRegionService.groovyGrails may soon provide a better hook
    53. 53. Regional URLsSome controllers should never be regionalRegionService detects RegionAgnostic annotation
    54. 54. Cloud deployment model
    55. 55. Cloud deployment modelApplications and Clusters
    56. 56. Cloud deployment model
    57. 57. Cloud deployment model Auto Scaling Group
    58. 58. Cloud deployment model Auto Scaling Group Launch Configuration
    59. 59. Cloud deployment model Elastic Load Balancer Auto Scaling Group Launch Configuration
    60. 60. Cloud deployment model Elastic Load Balancer Auto Scaling Group Launch Configuration Amazon Machine Image
    61. 61. Cloud deployment model Elastic Load Balancer Auto Scaling Group Security Group Launch Configuration Amazon Machine Image
    62. 62. Cloud deployment model Elastic Load Balancer Auto Scaling Group Security Instances Group Launch Configuration Amazon Machine Image
    63. 63. Cloud deployment model Elastic Load Balancer Auto Scaling Group Security Instances Group Launch Configuration Amazon Machine Image
    64. 64. Cloud deployment model Elastic Load Balancer Auto Scaling Group Security Instances Group Launch Configuration Amazon Machine Image
    65. 65. Cloud deployment model Elastic Load Balancer Auto Scaling Group Security Instances Group Launch Configuration Amazon Machine Image
    66. 66. Cloud deployment model
    67. 67. Cloud deployment model Search
    68. 68. Cloud deployment model API Search
    69. 69. Cloud deployment model Ratings API Search
    70. 70. Cloud deployment modelStreaming Starts Ratings API Search
    71. 71. Cloud deployment model Streaming Starts RatingsAutocomplete API Search
    72. 72. Cloud deployment model Sign Up Streaming Starts RatingsAutocomplete API Search
    73. 73. Cloud deployment model Sign Up Streaming Starts Ratings Application Application ApplicationAutocomplete API Search Application Application Application
    74. 74. Inventing the Application
    75. 75. Inventing the Application Problem: Application is not an Amazon concept
    76. 76. Inventing the Application Problem: Application is not an Amazon concept Solution: Create an Application domain in SimpleDB Enforce naming conventions on Amazon objects
    77. 77. Fast Rollback
    78. 78. Fast RollbackOptimism causes outages
    79. 79. Fast RollbackOptimism causes outagesProduction traffic is unique
    80. 80. Fast RollbackOptimism causes outagesProduction traffic is uniqueKeep old version running
    81. 81. Fast RollbackOptimism causes outagesProduction traffic is uniqueKeep old version runningSwitch traffic to new version
    82. 82. Fast RollbackOptimism causes outagesProduction traffic is uniqueKeep old version runningSwitch traffic to new versionMonitor results
    83. 83. Fast RollbackOptimism causes outagesProduction traffic is uniqueKeep old version runningSwitch traffic to new versionMonitor resultsRevert traffic quickly
    84. 84. Fast Rollback
    85. 85. Fast Rollback api-frontend api-usprod-v007
    86. 86. Fast Rollback api-frontend api-usprod-v007 api-usprod-v008
    87. 87. Fast Rollback api-frontend api-usprod-v007 api-usprod-v008
    88. 88. Fast Rollback api-frontend api-usprod-v007 api-usprod-v008
    89. 89. Fast Rollback api-frontend api-usprod-v007 api-usprod-v008
    90. 90. Fast Rollback api-frontend api-usprod-v007
    91. 91. Inventing the Cluster
    92. 92. Inventing the Cluster Problem: Two ASGs with one function but different names
    93. 93. Inventing the Cluster Problem: Two ASGs with one function but different names Solution: Append version number in reserved format Parse ASG name to determine long-term “cluster”
    94. 94. Inventing the ClusterInstead of keeping a database in sync, use namingconventions to store the source in truth in Amazon’s APIapi Applicationapi-usprod Clusterapi-usprod-v007 Auto Scaling Groupapi-usprod-v008 Auto Scaling Group
    95. 95. Database Aversion
    96. 96. Database AversionStoring metadata on cloud objects
    97. 97. Database Aversion
    98. 98. Database Aversion Naming conventions
    99. 99. Database Aversion Naming conventions Tagging conventions
    100. 100. Database Aversion Naming conventions Tagging conventions No GORM domain objects
    101. 101. Database Aversion Naming conventions Tagging conventions No GORM domain objects AWS Java SDK
    102. 102. Database Aversion Naming conventions Tagging conventions No GORM domain objects AWS Java SDK Less to go out of sync
    103. 103. Database Aversion Naming conventions Tagging conventions No GORM domain objects AWS Java SDK Less to go out of sync Shared source of truth
    104. 104. Open Source, Closed Config
    105. 105. Open Source, Closed ConfigPull company-specific details out of Asgard
    106. 106. Open Source, Closed Config
    107. 107. Open Source, Closed Config
    108. 108. Open Source, Closed Config Asgard for Netflix is configured to use company- specific extension points such as standard utility links for instances
    109. 109. Open Source, Closed Config Out-of-the-box Asgard installation has no instance utility links
    110. 110. Open Source, Closed Config Netflix specific $ASGARD_HOME/Config.groovylink {    // Avoid GStrings here because these Strings are stored dynamic templates for arbitrary server names.    String logUrlStart = http://${server}:7777    String configUrlStart = http://${server}:9999/AdminConfig    instanceLinkGroupingsToLinkTemplateLists = [            Logs: [                    new TextLinkTemplate(logUrlStart + /Admin/list?view=tomcat/catalina.out, catalina.out),                    new TextLinkTemplate(logUrlStart + /Admin/list, Log File Archive),                    new TextLinkTemplate(logUrlStart + /Admin/threaddumps, Thread Dumps),                    new TextLinkTemplate(logUrlStart + /AdminProxy, Admin Proxy Info),                    new TextLinkTemplate(logUrlStart + /AdminStatus, Admin Proxy Status),                    new TextLinkTemplate(logUrlStart + /GC/index, GC Visualization)            ],            Netflix Configuration: [                    new TextLinkTemplate(configUrlStart + /prop.html, NetflixConfiguration Properties Console),                    new TextLinkTemplate(configUrlStart + /libs.html, Libraries Console),                    new TextLinkTemplate(configUrlStart + /machineProps, Machine Readable Properties),                    new TextLinkTemplate(configUrlStart + /webapp/META-INF/MANIFEST.MF, Manifest),            ]    ]}
    111. 111. Open Source, Closed Config grails-app/conf/Config.groovy references external configuration file ~/.asgard/Config.groovy https://github.com/Netflix/asgard/blob/master/grails-app/conf/Config.groovyasgardHome = System.getenv(ASGARD_HOME) ?: System.getProperty(ASGARD_HOME) ?:        "${System.getProperty(user.home)}/.asgard"// Locations to search for config files that get merged into the main config.// Config files can either be Java properties files or ConfigSlurper scripts.grails.config.locations = [        "file:${asgardHome}/Config.groovy",        classpath:sourceVersion.properties]
    112. 112. Open Source, Closed Config External Config.groovy also hold the AWS account credentials, or references for finding them.grails {    awsAccounts=["178123456789"]    awsAccountNames=["178123456789":"prod"]}secret {    accessId="AKIAILAND0E3TMSJA8BA"    secretKey="Od0AMY/dEC3P0/wZQjpmRtm2MVZc9330epDk+pwm"}cloud {    accountName="prod"    publicResourceAccounts=["amazon"]}
    113. 113. Caching the Cloud
    114. 114. Caching the CloudResponsive, massive, multi-region metadata
    115. 115. Caching the Cloud
    116. 116. Caching the Cloud Large counts
    117. 117. Caching the Cloud Large counts Many types
    118. 118. Caching the Cloud Large counts Many types Complex relationships
    119. 119. Caching the Cloud Large counts Many types Complex relationships Multiple regions
    120. 120. Caching the Cloud Large counts Many types Complex relationships Multiple regions Consistent single objects
    121. 121. Caching the Cloud Large counts Many types Complex relationships Multiple regions Consistent single objects Eventually consistent lists
    122. 122. Caching the Cloudclass Caches {    final CachedMap<AppRegistration> allApplications    final CachedMap<ApplicationMetrics> allApplicationMetrics    final CachedMap<HardwareProfile> allHardwareProfiles    final MultiRegionCachedMap<MetricAlarm> allAlarms    final MultiRegionCachedMap<ApplicationInstance> allApplicationInstances    final MultiRegionCachedMap<AutoScalingGroup> allAutoScalingGroups    final MultiRegionCachedMap<AvailabilityZone> allAvailabilityZones    final MultiRegionCachedMap<Cluster> allClusters    final MultiRegionCachedMap<DBInstance> allDBInstances    final MultiRegionCachedMap<DBSecurityGroup> allDBSecurityGroups    final MultiRegionCachedMap<DBSnapshot> allDBSnapshots    final MultiRegionCachedMap<String> allDomains    final MultiRegionCachedMap<FastProperty> allFastProperties    final MultiRegionCachedMap<Image> allImages    final MultiRegionCachedMap<Instance> allInstances    final MultiRegionCachedMap<InstanceTypeData> allInstanceTypes    final MultiRegionCachedMap<KeyPairInfo> allKeyPairs    final MultiRegionCachedMap<LaunchConfiguration> allLaunchConfigurations    // etc.    // etc. // etc.}
    123. 123. Caching the Cloudclass Caches {    final CachedMap<AppRegistration> allApplications    final CachedMap<ApplicationMetrics> allApplicationMetrics    final CachedMap<HardwareProfile> allHardwareProfiles    final MultiRegionCachedMap<MetricAlarm> allAlarms    final MultiRegionCachedMap<ApplicationInstance> allApplicationInstances    final MultiRegionCachedMap<AutoScalingGroup> allAutoScalingGroups    final MultiRegionCachedMap<AvailabilityZone> allAvailabilityZones    final MultiRegionCachedMap<Cluster> allClusters    final MultiRegionCachedMap<DBInstance> allDBInstances    final MultiRegionCachedMap<DBSecurityGroup> allDBSecurityGroups    final MultiRegionCachedMap<DBSnapshot> allDBSnapshots    final MultiRegionCachedMap<String> allDomains    final MultiRegionCachedMap<FastProperty> allFastProperties    final MultiRegionCachedMap<Image> allImages    final MultiRegionCachedMap<Instance> allInstances    final MultiRegionCachedMap<InstanceTypeData> allInstanceTypes    final MultiRegionCachedMap<KeyPairInfo> allKeyPairs    final MultiRegionCachedMap<LaunchConfiguration> allLaunchConfigurations    // etc.    // etc. // etc.}
    124. 124. Caching the Cloudclass AwsRdsService implements CacheInitializer, InitializingBean {    MultiRegionAwsClient<AmazonRDS> awsClient    Caches caches    void initializeCaches() {        caches.allDBInstances.ensureSetUp({ Region region -> retrieveDBInstances(region) })    }    private List<DBInstance> retrieveDBInstances(Region region) {        awsClient.by(region).describeDBInstances(new DescribeDBInstancesRequest()).getDBInstances()    }    Collection<DBInstance> getDBInstances(UserContext userContext) {        caches.allDBInstances.by(userContext.region).list()    }}
    125. 125. Caching the Cloudclass AwsRdsService implements CacheInitializer, InitializingBean {    MultiRegionAwsClient<AmazonRDS> awsClient    Caches caches    void initializeCaches() {        caches.allDBInstances.ensureSetUp({ Region region -> retrieveDBInstances(region) })    }    private List<DBInstance> retrieveDBInstances(Region region) {        awsClient.by(region).describeDBInstances(new DescribeDBInstancesRequest()).getDBInstances()    }    Collection<DBInstance> getDBInstances(UserContext userContext) {        caches.allDBInstances.by(userContext.region).list()    }}
    126. 126. Caching the Cloudclass AwsRdsService implements CacheInitializer, InitializingBean {    MultiRegionAwsClient<AmazonRDS> awsClient    Caches caches    void initializeCaches() {        caches.allDBInstances.ensureSetUp({ Region region -> retrieveDBInstances(region) })    }    private List<DBInstance> retrieveDBInstances(Region region) {        awsClient.by(region).describeDBInstances(new DescribeDBInstancesRequest()).getDBInstances()    }    Collection<DBInstance> getDBInstances(UserContext userContext) {        caches.allDBInstances.by(userContext.region).list()    }}
    127. 127. Caching the Cloudclass AwsRdsService implements CacheInitializer, InitializingBean {    MultiRegionAwsClient<AmazonRDS> awsClient    Caches caches    void initializeCaches() {        caches.allDBInstances.ensureSetUp({ Region region -> retrieveDBInstances(region) })    }    private List<DBInstance> retrieveDBInstances(Region region) {        awsClient.by(region).describeDBInstances(new DescribeDBInstancesRequest()).getDBInstances()    }    Collection<DBInstance> getDBInstances(UserContext userContext) {        caches.allDBInstances.by(userContext.region).list()    }}
    128. 128. Caching the Cloud
    129. 129. Visual Language for the Cloud
    130. 130. Visual Language for the CloudTango open source icons
    131. 131. Visual Language for the Cloud
    132. 132. Visual Language for the Cloud AWS is intimidating
    133. 133. Visual Language for the Cloud AWS is intimidating Many object types
    134. 134. Visual Language for the Cloud AWS is intimidating Many object types Help newbie users
    135. 135. Visual Language for the Cloud AWS is intimidating Many object types Help newbie users Reduce cognitive load
    136. 136. Visual Language for the Cloud AWS is intimidating Many object types Help newbie users Reduce cognitive load Make it easy
    137. 137. Visual Language for the Cloud AWS is intimidating Many object types Help newbie users Reduce cognitive load Make it easy Avoid surprises
    138. 138. Visual Language for the Cloud
    139. 139. Visual Language for the CloudAt a glance, these nav bar items look alike.
    140. 140. Visual Language for the CloudAt a glance, these nav bar items look alike.
    141. 141. Visual Language for the Cloud
    142. 142. Visual Language for the CloudSome screens have multiple actionbuttons that look too similar.
    143. 143. Visual Language for the CloudSome screens have multiple actionbuttons that look too similar.
    144. 144. Visual Language for the Cloud
    145. 145. Visual Language for the Cloud Because of naming conventions, these links look alike.
    146. 146. Visual Language for the Cloud Because of naming conventions, these links look alike.
    147. 147. Visual Language for the Cloud
    148. 148. Visual Language for the Cloud The indicators for the current AWS region are too easy to miss.
    149. 149. Visual Language for the Cloud The indicators for the current AWS region are too easy to miss.
    150. 150. Visual Language for the CloudThese availability zonesare important torecognize at a glancebut their names looksimilar, and they appearon many screens.
    151. 151. Visual Language for the CloudThese availability zonesare important torecognize at a glancebut their names looksimilar, and they appearon many screens.
    152. 152. Visual Language for the Cloud
    153. 153. Tango Icons
    154. 154. Tango Iconshttp://tango.freedesktop.org/
    155. 155. Tango Iconshttp://tango.freedesktop.org/http://tango.freedesktop.org/Tango_Icon_Theme_Guidelines
    156. 156. Tango Iconshttp://tango.freedesktop.org/http://tango.freedesktop.org/Tango_Icon_Theme_Guidelineshttp://commons.wikimedia.org/wiki/Tango_icons
    157. 157. Tango Iconshttp://tango.freedesktop.org/http://tango.freedesktop.org/Tango_Icon_Theme_Guidelineshttp://commons.wikimedia.org/wiki/Tango_iconsUsed by Firefox, Jenkins, GIMP, OpenOffice, VMWare
    158. 158. REST API in Grails
    159. 159. REST API in GrailsEnable external mashups with cloud data
    160. 160. REST API in Grails
    161. 161. REST API in Grails
    162. 162. REST API in Grails
    163. 163. REST API in Grails
    164. 164. REST API in Grails
    165. 165. REST API in Grails
    166. 166. REST API in Grails
    167. 167. REST API in Grails
    168. 168. REST API in Grails
    169. 169. REST API in GrailsApplicationController.groovy
    170. 170. Offline Development
    171. 171. Offline DevelopmentMakes on a plane
    172. 172. Offline DevelopmentMock dataMock behaviorSystem property switch offline=true
    173. 173. Mock Datahttp://asgardprod/us-east-1/autoScaling/list.json
    174. 174. Mock DataParse JSON
    175. 175. Mock BehaviorOverride Amazon Java client methods
    176. 176. System Propertygrails run-app -Doffline=true
    177. 177. Why not the AWS console?
    178. 178. Why not the AWS console?No, not to keep me busy
    179. 179. Why not the AWS console?
    180. 180. Why not the AWS console?
    181. 181. Why not the AWS console?Hide keys
    182. 182. Why not the AWS console?Hide keysCustomize model
    183. 183. Why not the AWS console?Hide keysCustomize modelEnforce conventions
    184. 184. Why not the AWS console?Hide keysCustomize modelEnforce conventionsAutomate workflow
    185. 185. Why not the AWS console?Hide keysCustomize modelEnforce conventionsAutomate workflowLog changes
    186. 186. Why not the AWS console?Hide keysCustomize modelEnforce conventionsAutomate workflowLog changesIntegrate systems
    187. 187. Why not the AWS console?Hide keysCustomize modelEnforce conventionsAutomate workflowLog changesIntegrate systemsCreate REST API
    188. 188. @NetflixOSS
    189. 189. @NetflixOSShttp://techblog.netflix.com
    190. 190. @NetflixOSShttp://techblog.netflix.comhttp://netflix.github.com
    191. 191. http://github.com/Netflix/asgardThank you
    192. 192. http://github.com/Netflix/asgardThank youQuestions? github.com/joesondow @joesondow slideshare.net/joesondow linkedin.com/in/joesondow
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×