Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Zero to the Cloud with @NetflixOSS

3,420 views

Published on

Getting started on AWS is easy, but building a scalable, reliable and performant product in the cloud can be a challenge for startups and enterprises alike. Netflix has famously migrated all our services to the cloud. Along the way, we have open sourced large portions of our platform that helped make this a reality. In this talk, Mike McGarr (Manager, Netflix Build Tools) will provide a survey of the @NetflixOSS products available. Mike will also share patterns and lessons Netflix learned migrating to the cloud. Lastly, Mike will leave you with a roadmap for how to get started with @NetflixOSS on your cloud today. This talk will cover the following @NetflixOSS products, and more!

Published in: Technology

Zero to the Cloud with @NetflixOSS

  1. 1. Zero to the Cloud with @NetflixOSS Mike McGarr mmcgarr@netflix.com @SonOfGarr http://www.MikeMcGarr.com
  2. 2. about me • J. Michael (Mike) McGarr • Engineering Manager, Netflix Build Tools • Founder, DC Continuous Delivery Meetup • Former Director of DevOps, Blackboard • Excella Consulting • Booz Allen Hamilton 11/17/14 @SonOfGarr 2
  3. 3. 11/17/14 @SonOfGarr 3
  4. 4. 11/17/14 @SonOfGarr 4
  5. 5. @NetflixOSS 11/17/14 @SonOfGarr http://netflix.github.io 5
  6. 6. 11/17/14 @SonOfGarr 6
  7. 7. The Netflix Way Everything is “built for three” Fully automated build tools to test and make packages Fully automated machine image bakery 11/17/14 @SonOfGarr 7
  8. 8. The Netflix Way Fully automated image deployment Service teams responsible for both “Dev” and “Ops” Redundancy through multi-region deployment 11/17/14 @SonOfGarr 8
  9. 9. cloud native 11/17/14 @SonOfGarr 9
  10. 10. microservices (soa) 11/17/14 @SonOfGarr 10
  11. 11. 11/17/14 @SonOfGarr 11
  12. 12. Zero to the Cloud • Stand-up your own NetflixOSS cloud • Aminator • BaseAMI • Asgard • Edda • Eureka http://nflx.it/zerotocloud 11/17/14 @SonOfGarr 12
  13. 13. foundations • account setup • user, roles, security groups • jumphost • foundation AMI 11/17/14 @SonOfGarr 13
  14. 14. BUILD/BAKE 11/17/14 @SonOfGarr 14
  15. 15. immutable servers • new os image per code change • never change running instances • avoid configuration drift http://martinfowler.com/bliki/ImmutableServer.html 11/17/14 @SonOfGarr 15
  16. 16. Nebula • Netflix Build Language • Gradle plugins http://github.com/nebula-plugins 11/17/14 @SonOfGarr 16
  17. 17. Nebula Plugins • gradle-ospackage-plugin • gradle-dependency-lock-plugin • gradle-extra-configurations-plugin • gradle-blacklist-plugin • gradle-override-plugin • gradle-stash-plugin • gradle-scm-plugin • gradle-contacts-plugin • gradle-info-plugin • nebula-ospackage-plugin • nebula-test • nebula-project-plugin • nebula-release-plugin • nebula-bintray-plugin • nebula-publishing-plugin • nebula-metrics-plugin 11/17/14 @SonOfGarr 17
  18. 18. gradle-ospackage-plugin • produce rpm or debian packages from Gradle build • can be used for non-java apps 11/17/14 @SonOfGarr 18
  19. 19. 11/17/14 @SonOfGarr 19
  20. 20. Baking • app. code installed as OS package • unit of deployment = AMI 11/17/14 @SonOfGarr 20
  21. 21. BaseAMI 11/17/14 @SonOfGarr 21
  22. 22. Aminator • python library used to “bake” EBS AMI’s • must bake per region • works with system packages, Chef, Puppet, Ansible, etc 11/17/14 @SonOfGarr 22
  23. 23. sudo aminate -e ec2_aptitude_linux -b ubuntu-foundation -n ubuntu-base-ami baseami/build/distributions/baseami_1.0.0_all.deb 11/17/14 @SonOfGarr 23
  24. 24. DEPLOYMENTS 11/17/14 @SonOfGarr 24
  25. 25. AWS “applications” 11/17/14 @SonOfGarr 25
  26. 26. AWS “applications” 11/17/14 @SonOfGarr 26
  27. 27. asgard • Netflix AWS console • Provides “app” abstraction • Hides AWS keys • NetflixOSS https://github.com/Netflix/asgard 11/17/14 @SonOfGarr 27
  28. 28. red/black deployments 11/17/14 @SonOfGarr 28
  29. 29. red/black deployments 11/17/14 @SonOfGarr 29
  30. 30. red/black deployments 11/17/14 @SonOfGarr 30
  31. 31. canary releases 11/17/14 @SonOfGarr 31
  32. 32. inventing clusters 11/17/14 @SonOfGarr 32
  33. 33. without asgard aws ec2 create-security-group --group-name asgard-sg aws ec2 authorize-security-group-ingress --group-id sg-123456 --protocol tcp --port 22 --cidr 0.0.0.0/0 aws elb create-load-balancer --load-balancer-name asgard-lb –listeners Protocol=string,LoadBalancerPort=80,InstanceProtocol=http,InstancePort=7001 aws autoscaling create-launch-configuration --launch-configuration-name asgard-lc --image-id ami-XXX --key-name zerotocloud --security-groups asgard-sg --instance-type m3.medium --iam-instance-profile jumphost aws autoscaling create-auto-scaling-group --auto-scaling-group-name asgard-v000 --launch-configuration-name asgard-lc --min-size 1 --max-size 1 --desired-capacity 1 --availability-zones us-west-2b --load-balancer-names asgard-lb --tags ResourceId=string,ResourceType=string,Key=string,Value=string,PropagateAtLaunch=boolean 11/17/14 @SonOfGarr 33
  34. 34. CLOUD INFRASTRUCTURE 11/17/14 @SonOfGarr 34
  35. 35. service discovery? • ephemeral nature of the cloud – instances come and go – ASG’s come and go • need to register/de-register a server • load balance middle-tier services w/o public exposure 11/17/14 @SonOfGarr 35
  36. 36. eureka • REST-based server • Java-based client https://github.com/Netflix/eureka 11/17/14 @SonOfGarr 36
  37. 37. eureka 11/17/14 @SonOfGarr 37
  38. 38. state of the cloud? • your cloud is dynamic: – new instances – IP address reuse – new firewalls • I want to track changes over time – not provided by AWS 11/17/14 @SonOfGarr 38
  39. 39. edda • polls AWS resources • quick search via REST API • changes over time • architecture: – Scala – MongoDB backend 11/17/14 @SonOfGarr 39
  40. 40. cloud persistence • a variety of NoSQL persistence needs • utilize open source • Problems: – not cloud native – poor operational features http://techblog.netflix.com/2011/01/nosql-at-netflix.html 11/17/14 @SonOfGarr 40
  41. 41. sidecar 11/17/14 @SonOfGarr 41
  42. 42. priam Cassandra instance sidecar – backup and recovery – token management – Seed discovery – Configuration – Support AWS environment https://github.com/Netflix/priam 11/17/14 @SonOfGarr 42
  43. 43. raigad ElasticSearch instance sidecar – backup and restore – configured deployments – tribe node deployments – publish metrics – support AWS environment https://github.com/Netflix/raigad 11/17/14 @SonOfGarr 43
  44. 44. exhibitor ZooKeeper instance sidecar – Performs periodic backups – Perform periodic cleaning of ZK log directory – A GUI explorer for viewing ZK nodes – A rich REST API 11/17/14 @SonOfGarr 44
  45. 45. SERVICE ARCHITECTURE 11/17/14 @SonOfGarr 45
  46. 46. Java microservices 11/17/14 @SonOfGarr 46
  47. 47. ribbon • cloud IPC library – pluggable load balancing rules – eureka integration – archauis integration – cloud enabled https://github.com/Netflix/ribbon 11/17/14 @SonOfGarr 47
  48. 48. 11/17/14 @SonOfGarr 48
  49. 49. hystrix • Latency & fault tolerance for distributed systems – circuit breaker pattern – realtime operations https://github.com/Netflix/Hystrix 11/17/14 @SonOfGarr 49
  50. 50. 11/17/14 @SonOfGarr 50
  51. 51. hystrix dashboard https://github.com/Netflix/Hystrix/wiki/Dashboard 11/17/14 @SonOfGarr 51
  52. 52. archaius • Java library for dynamic runtime properties https://github.com/Netflix/archaius 11/17/14 @SonOfGarr 52
  53. 53. evcache • distributed in-memory caching – based on memchached & spymemcached • EVCache: – Elastic – Volatile – Cache https://github.com/Netflix/EVCache 11/17/14 @SonOfGarr 53
  54. 54. servo • expose applications metrics – gauges – counters – timers https://github.com/Netflix/servo 11/17/14 @SonOfGarr 54
  55. 55. astayanax • Java Cassandra client • fluent API • cloud native https://github.com/Netflix/astyanax 11/17/14 @SonOfGarr 55
  56. 56. governator • Google Guice extensions – classpath scanning – lifecycle management – field validation – lazy singleton https://github.com/netflix/Governator 11/17/14 @SonOfGarr 56
  57. 57. karyon • nucleus of Netflix platform – governator bootstrap – admin console – health checks – discovery https://github.com/Netflix/karyon 11/17/14 @SonOfGarr 57
  58. 58. prana Generic sidecar w/ HTTP access to Java clients for: – Eureka – Ribbon – Archauis – health checks 11/17/14 @SonOfGarr 58
  59. 59. CHAOS 11/17/14 @SonOfGarr 59
  60. 60. chaos monkey 11/17/14 @SonOfGarr 60
  61. 61. simian army • chaos monkey • janitor monkey • compliance monkey • security monkey https://github.com/Netflix/SimianArmy 11/17/14 @SonOfGarr 61
  62. 62. http://nflx.it/1v4H7yi 11/17/14 @SonOfGarr 62
  63. 63. 11/17/14 @SonOfGarr 63
  64. 64. We are Hiring! Senior Cloud Tools Engineer http://nflx.it/140BnJN Senior Tooling Integration Engineer Mike McGarr mmcgarr@netflix.com @SonOfGarr http://www.mikemcgarr.com http://nflx.it/1xyhkeD 11/17/14 @SonOfGarr 64

×