Continuous Integration and Deployment Best Practices on AWS

  • 685 views
Uploaded on

Continuous Integration and Deployment Best Practices on AWS

Continuous Integration and Deployment Best Practices on AWS

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
685
On Slideshare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
13
Comments
0
Likes
3

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. CONTINUOUS INTEGRATION AND DEPLOYMENT BEST PRACTICES ON AWS Danilo Poccia – Solutions Architect
  • 2. Compute & Networking Storage & Content Delivery AWS Global Infrastucture Database App Services Analytics Deployment & Management API API API
  • 3. API API API Regions Availability Zones Edge Locations EC2 VPC Route 53 Direct Connect S3 Glacier CloudFront Storage Gateway RDS DynamoDB ElastiCache Redshift SQS SNS SWF SES AppStream CloudSearch Elastic Transcoder IAM CloudTrail CloudWatch CloudFormation Elastic Beanstalk OpsWorks EMR Data Pipeline Kinesis WorkSpaces
  • 4. Regions Availability Zones Edge Locations S3 Glacier CloudFront Storage Gateway RDS DynamoDB ElastiCache Redshift SQS SNS SWF SES AppStream CloudSearch Elastic Transcoder IAM CloudTrail CloudWatch CloudFormation Elastic Beanstalk OpsWorks EMR Data Pipeline Kinesis EC2 VPC Route 53 Direct Connect WorkSpaces
  • 5. Regions Availability Zones Edge Locations EC2 VPC Route 53 DirectConnect S3 Glacier CloudFront Storage Gateway RDS DynamoDB ElastiCache RedshiftSQS SNS SWF SES AppStream CloudSearch ElasticTranscoder IAM CloudTrail CloudWatch CloudFormation Elastic Beanstalk OpsWorks EMR Data Pipeline Kinesis WorkSpaces
  • 6. http://commons.wikimedia.org/wiki/File:Lego_bricks.jpg
  • 7. BUILDING BLOCKS TO IMPLEMENT YOUR IDEA
  • 8. FILE SYNC
  • 9. DATA ANALYSIS
  • 10. DATA ANALYSIS COSTS < VALUE
  • 11. AMAZON EC2 SPOT INSTANCES
  • 12. 2013   280 2011   82 2010   61 2009   48 2008   24 2007   9 Amazon  FPS   Red  Hat  EC2 SimpleDB   CloudFront   EBS   Availability  Zones   Elastic  IPs Relational  Database  Service   Virtual  Private  Cloud   Elastic  Map  Reduce   Auto  Scaling   Reserved  Instances   Elastic  Load  Balancer Simple  Notification  Service   Route  53   RDS  Multi-­‐AZ   Singapore  Region   Identity  Access  Management   Cluster  Instances Elastic  Beanstalk   Simple  Email  Service   CloudFormation   RDS  for  Oracle   ElastiCache Redshift   Kinesis   WorkSpaces   Route  53  Health  Check   OpsWorks   CloudTrail number  of  released  features,  sample  services  described 2012   159Glacier   DynamoDB   Simple  Workflow   CloudSearch   Storage  Gateway   Route  53  Latency  Based  Routing
  • 13. NEW SERVICES + NEW FEATURES 
 = NEW IDEAS
  • 14. AMAZON KINESIS REAL-TIME PROCESSING OF STREAMING DATA AT MASSIVE SCALE
  • 15. !Data! Sources! App.4% ! [Machine! Learning]! !!!!!!!!!!!!!!! ! ! ! AWS!Endpoint! App.1% % [Aggregate!&! De:Duplicate]! !Data! Sources! Data! Sources! !Data! Sources! App.2% % [Metric! Extrac=on]! S3 DynamoDB Redshift App.3% [Sliding! Window! Analysis]! !Data! Sources! Availability Zone Shard 1 Shard 2 Shard N Availability Zone Availability Zone
  • 16. CLICKSTREAM ANALYTICS INTERNET OF THINGS METERING & BILLING EVENT PROCESSING SOCIAL NETWORKS LOG COLLECTION …
  • 17. AMAZON SQS + AUTO SCALING + SPOT INSTANCES ! = FREEMIUM
  • 18. AMAZON SNS MOBILE PUSH NOTIFICATIONS
  • 19. EVEO nline
  • 20. ANY NEW IDEA TO BUILD?
  • 21. THE JOURNEY OF CODE TO PRODUCTION
  • 22. WRITTEN TESTED CONSUMED DEPLOYED
  • 23. ~ MANUFACTURING ~
  • 24. LEAN MANUFACTURING Transport Inventory Motion Waiting Over Production Over Processing Defects Talent! & Skills
  • 25. INFORMATION TECHNOLOGY Transport Inventory Motion Waiting Over Production Over Processing Defects Talent! & Skills
  • 26. OPTIMIZE WITH AWS Transport Inventory Motion Waiting Over Production Over Processing Defects Talent! & Skills
  • 27. OPTIMIZE WITH AWS OPPORTUNITY STRATEGY USE ADJUST
  • 28. = PROGRAMMABLE PLATFORM
  • 29. ! IF YOU CAN DO IT
 YOU CAN AUTOMATE IT
  • 30. “Improving daily work is even more important than doing daily work.” ! ! Gene Kim, Kevin Behr, George Spafford, The Phoenix Project: A Novel About IT, DevOps,
 and Helping Your Business Win
  • 31. CONTINUOUS INTEGRATION
  • 32. DEVELOPER
  • 33. SOURCE CODE REPOSITORY
  • 34. SOURCE CODE REPOSITORY PROJECT MANAGEMENT SERVER
  • 35. SOURCE CODE REPOSITORY PROJECT MANAGEMENT SERVER CONTINUOUS INTEGRATION SERVER
  • 36. SOURCE CODE REPOSITORY PROJECT MANAGEMENT SERVER CONTINUOUS INTEGRATION SERVER PICK
 TASKS
  • 37. SOURCE CODE REPOSITORY PROJECT MANAGEMENT SERVER CONTINUOUS INTEGRATION SERVER SUBMIT
 CODE
  • 38. SOURCE CODE REPOSITORY PROJECT MANAGEMENT SERVER CONTINUOUS INTEGRATION SERVER SCHEDULE
 BUILD
  • 39. SOURCE CODE REPOSITORY PROJECT MANAGEMENT SERVER CONTINUOUS INTEGRATION SERVER AUTOMATIC
 BUILDS
  • 40. SOURCE CODE REPOSITORY PROJECT MANAGEMENT SERVER CONTINUOUS INTEGRATION SERVER CODE
 FETCH
  • 41. SOURCE CODE REPOSITORY PROJECT MANAGEMENT SERVER CONTINUOUS INTEGRATION SERVER CODE QUALITY
 TESTS TEST
 RESULTS
  • 42. SOURCE CODE REPOSITORY PROJECT MANAGEMENT SERVER CONTINUOUS INTEGRATION SERVER BUILD OUTPUT
  • 43. SOURCE CODE REPOSITORY PROJECT MANAGEMENT SERVER CONTINUOUS INTEGRATION SERVER DOCS BINARIES
 & PACKAGES
  • 44. TDD TEST-DRIVEN DEVELOPMENT
  • 45. BDD BEHAVIOR-DRIVEN DEVELOPMENT
  • 46. PAIN POINTS • UNIT TESTS INCOMPLETE • MOCK TESTS MAINTENANCE • EXPENSIVE TEST ENVIRONMENT • TEST ENVIRONMENT ≠ PRODUCTION • DEPLOYMENT CYCLES
  • 47. “Complexity arises when the dependencies among the elements become important.” ! ! Scott E. Page, John H. Miller,
 Complex Adaptive Systems:
 An Introduction to Computational Models of Social Life
  • 48. REAL USERS = REAL FEEDBACK
  • 49. CLOSE THE FEEDBACK LOOP AS QUICKLY AS POSSIBLE
  • 50. CONTINUOUS INTEGRATION CONTINUOUS DEPLOYMENT
  • 51. SOFTWARE DEPLOY
 ≠
 PRODUCT LAUNCH
  • 52. CONTINUOUS DEPLOYMENT SMALL, FREQUENT CHANGES CONSTANTLY INTEGRATING INTO PRODUCTION
  • 53. KEY = ITERATION
  • 54. ITERATION
 =
 MODIFY THE SYSTEM TO BETTER MEET THE EXPECTATIONS OF YOUR USERS
  • 55. FEATURE SWITCH
 FEATURE FLIPPER FEATURE TOGGLE FEATURE FLAG CONDITIONAL FEATURE
  • 56. // To totally enabled a feature ! $server_config[‘foo'] = ‘on’;! ! // To totally disabled a feature! $server_config['foo'] = ‘off’;! ! // Feature with variant turned on for everyone! $server_config['foo'] = 'blue_background';! ! // Enabled for 10% of regular users and all admins! $server_config[‘foo'] = array(!    'enabled' => 10,!    'admin' => 'on',! !); https://github.com/etsy/feature
  • 57. !if (Feature::isEnabled('my_feature')) {! !    // do stuff! !}! ! ! !if (Feature::isEnabled('my_feature')) {! ! ! !    switch (Feature::variant('my_feature')) {! !      case 'foo':! !          // do stuff appropriate for the foo variant! !          break;! !      case 'bar':! !          // do stuff appropriate for the bar variant! !          break;! !    }! !} https://github.com/etsy/feature
  • 58. A / B TESTING CONTENT
  • 59. // Single-variant feature in 50/50 A/B test! ! $server_config['foo'] = array('enabled' => 50);! ! // Multi-variant feature in A/B test! // with 20% of users seeing each variant! // (and 40% left in control group)! ! $server_config['foo'] = array(! !   'enabled' => array(! !       'blue_background'   => 20,! !       'orange_background' => 20,! !       'pink_background'   => 20,! !   ),! !); https://github.com/etsy/feature
  • 60. AWS OPSWORKS MODEL AND MANAGE THE ENTIRE APPLICATION
  • 61. 14 BILLION REQUESTS/MONTH 50 000 DATABASE UPDATES / SEC NO CACHE C A SE STU D Y
  • 62. A / B TESTING NOT JUST FOR CONTENT
  • 63. USER INTERFACE ALGORITHMS ARCHITECTURES COST OPTIMIZATION
  • 64. AWS CLOUDFORMATION INFRASTRUCTURE IS CODE
  • 65. CLOUDFORMATION
 TEMPLATE
  • 66. APPLICATION
 VERSIONS
 +
 INFRASTRUCTURE VERSIONS
  • 67. CLOUDFORMATION TEMPLATE
  • 68. AWS ELASTIC BEANSTALK QUICKLY DEPLOY AND MANAGE APPLICATIONS
  • 69. Production Environment Test Environment master branch test branch Git Repository git commit git aws.push git aws.push Swap URLs
  • 70. AMAZON EC2 API / SDK / CLI
  • 71. for ip in $(! ! aws ec2 describe-instances! --filter Name=instance-state-name,Values=running,! Name=tag:aws:autoscaling:groupName,Values=XXX! --query Reservations[].Instances[].PublicDnsName! --output text! ! ); do ssh -t ec2-user@$ip "/app/update.sh"; done! !
  • 72. ALL TOGETHER NOW! ! ! ! ! !
  • 73. DEPLOYMENTS AT 
 AMAZON.COM ! ! ! 11.6s Mean time between deployments (weekday) 1,079 Max number of deployments in a single hour 10,000 Mean number of hosts simultaneously receiving a deployment 30,000 Max number of hosts simultaneously receiving a deployment
  • 74. CONTINUOUS DEPLOYMENT = CONTINUOUS EXPERIMENTATION
  • 75. CONTINUOUS DEPLOYMENT = CONTINUOUS INNOVATION
  • 76. http://commons.wikimedia.org/wiki/File:Lego_bricks.jpg
  • 77. « Want to increase innovation? Lower the cost of failure » ! Joi Ito
  • 78. CONTINUOUS INTEGRATION AND DEPLOYMENT BEST PRACTICES ON AWS Danilo Poccia – Solutions Architect