Uploaded byMike McGarr
Zero to the Cloud with @NetflixOSS
The document discusses Netflix's transition to cloud-native architecture, detailing tools and practices such as automated build tools, microservices, and immutable servers. It highlights various Netflix OSS projects like Asgard, Eureka, and Chaos Monkey that facilitate deployment, service discovery, and resilience in cloud environments. Additionally, it addresses the dynamic nature of cloud infrastructure and the need for effective monitoring and management strategies.
More Related Content
Similar to Zero to the Cloud with @NetflixOSS
Zero to the Cloud with @NetflixOSS
- 1. Zero to theCloud with @NetflixOSS Mike McGarr mmcgarr@netflix.com @SonOfGarr http://www.MikeMcGarr.com
- 2. about me •J. Michael (Mike) McGarr • Engineering Manager, Netflix Build Tools • Founder, DC Continuous Delivery Meetup • Former Director of DevOps, Blackboard • Excella Consulting • Booz Allen Hamilton 11/17/14 @SonOfGarr 2
- 3.
- 4.
- 5. @NetflixOSS 11/17/14 @SonOfGarrhttp://netflix.github.io 5
- 6.
- 7. The Netflix Way Everything is “built for three” Fully automated build tools to test and make packages Fully automated machine image bakery 11/17/14 @SonOfGarr 7
- 8. The Netflix Way Fully automated image deployment Service teams responsible for both “Dev” and “Ops” Redundancy through multi-region deployment 11/17/14 @SonOfGarr 8
- 9. cloud native 11/17/14@SonOfGarr 9
- 10. microservices (soa) 11/17/14@SonOfGarr 10
- 11.
- 12. Zero to theCloud • Stand-up your own NetflixOSS cloud • Aminator • BaseAMI • Asgard • Edda • Eureka http://nflx.it/zerotocloud 11/17/14 @SonOfGarr 12
- 13. foundations • accountsetup • user, roles, security groups • jumphost • foundation AMI 11/17/14 @SonOfGarr 13
- 14.
- 15. immutable servers •new os image per code change • never change running instances • avoid configuration drift http://martinfowler.com/bliki/ImmutableServer.html 11/17/14 @SonOfGarr 15
- 16. Nebula • NetflixBuild Language • Gradle plugins http://github.com/nebula-plugins 11/17/14 @SonOfGarr 16
- 17. Nebula Plugins •gradle-ospackage-plugin • gradle-dependency-lock-plugin • gradle-extra-configurations-plugin • gradle-blacklist-plugin • gradle-override-plugin • gradle-stash-plugin • gradle-scm-plugin • gradle-contacts-plugin • gradle-info-plugin • nebula-ospackage-plugin • nebula-test • nebula-project-plugin • nebula-release-plugin • nebula-bintray-plugin • nebula-publishing-plugin • nebula-metrics-plugin 11/17/14 @SonOfGarr 17
- 18. gradle-ospackage-plugin • producerpm or debian packages from Gradle build • can be used for non-java apps 11/17/14 @SonOfGarr 18
- 19.
- 20. Baking • app.code installed as OS package • unit of deployment = AMI 11/17/14 @SonOfGarr 20
- 21.
- 22. Aminator • pythonlibrary used to “bake” EBS AMI’s • must bake per region • works with system packages, Chef, Puppet, Ansible, etc 11/17/14 @SonOfGarr 22
- 23. sudo aminate -e ec2_aptitude_linux -b ubuntu-foundation -n ubuntu-base-ami baseami/build/distributions/baseami_1.0.0_all.deb 11/17/14 @SonOfGarr 23
- 24.
- 25. AWS “applications” 11/17/14@SonOfGarr 25
- 26. AWS “applications” 11/17/14@SonOfGarr 26
- 27. asgard • NetflixAWS console • Provides “app” abstraction • Hides AWS keys • NetflixOSS https://github.com/Netflix/asgard 11/17/14 @SonOfGarr 27
- 28. red/black deployments 11/17/14@SonOfGarr 28
- 29. red/black deployments 11/17/14@SonOfGarr 29
- 30. red/black deployments 11/17/14@SonOfGarr 30
- 31. canary releases 11/17/14@SonOfGarr 31
- 32. inventing clusters 11/17/14@SonOfGarr 32
- 33. without asgard awsec2 create-security-group --group-name asgard-sg aws ec2 authorize-security-group-ingress --group-id sg-123456 --protocol tcp --port 22 --cidr 0.0.0.0/0 aws elb create-load-balancer --load-balancer-name asgard-lb –listeners Protocol=string,LoadBalancerPort=80,InstanceProtocol=http,InstancePort=7001 aws autoscaling create-launch-configuration --launch-configuration-name asgard-lc --image-id ami-XXX --key-name zerotocloud --security-groups asgard-sg --instance-type m3.medium --iam-instance-profile jumphost aws autoscaling create-auto-scaling-group --auto-scaling-group-name asgard-v000 --launch-configuration-name asgard-lc --min-size 1 --max-size 1 --desired-capacity 1 --availability-zones us-west-2b --load-balancer-names asgard-lb --tags ResourceId=string,ResourceType=string,Key=string,Value=string,PropagateAtLaunch=boolean 11/17/14 @SonOfGarr 33
- 34. CLOUD INFRASTRUCTURE 11/17/14@SonOfGarr 34
- 35. service discovery? •ephemeral nature of the cloud – instances come and go – ASG’s come and go • need to register/de-register a server • load balance middle-tier services w/o public exposure 11/17/14 @SonOfGarr 35
- 36. eureka • REST-basedserver • Java-based client https://github.com/Netflix/eureka 11/17/14 @SonOfGarr 36
- 37.
- 38. state of thecloud? • your cloud is dynamic: – new instances – IP address reuse – new firewalls • I want to track changes over time – not provided by AWS 11/17/14 @SonOfGarr 38
- 39. edda • pollsAWS resources • quick search via REST API • changes over time • architecture: – Scala – MongoDB backend 11/17/14 @SonOfGarr 39
- 40. cloud persistence •a variety of NoSQL persistence needs • utilize open source • Problems: – not cloud native – poor operational features http://techblog.netflix.com/2011/01/nosql-at-netflix.html 11/17/14 @SonOfGarr 40
- 41.
- 42. priam Cassandra instancesidecar – backup and recovery – token management – Seed discovery – Configuration – Support AWS environment https://github.com/Netflix/priam 11/17/14 @SonOfGarr 42
- 43. raigad ElasticSearch instancesidecar – backup and restore – configured deployments – tribe node deployments – publish metrics – support AWS environment https://github.com/Netflix/raigad 11/17/14 @SonOfGarr 43
- 44. exhibitor ZooKeeper instancesidecar – Performs periodic backups – Perform periodic cleaning of ZK log directory – A GUI explorer for viewing ZK nodes – A rich REST API 11/17/14 @SonOfGarr 44
- 45. SERVICE ARCHITECTURE 11/17/14@SonOfGarr 45
- 46. Java microservices 11/17/14@SonOfGarr 46
- 47. ribbon • cloudIPC library – pluggable load balancing rules – eureka integration – archauis integration – cloud enabled https://github.com/Netflix/ribbon 11/17/14 @SonOfGarr 47
- 48.
- 49. hystrix • Latency& fault tolerance for distributed systems – circuit breaker pattern – realtime operations https://github.com/Netflix/Hystrix 11/17/14 @SonOfGarr 49
- 50.
- 51. hystrix dashboard https://github.com/Netflix/Hystrix/wiki/Dashboard 11/17/14 @SonOfGarr 51
- 52. archaius • Javalibrary for dynamic runtime properties https://github.com/Netflix/archaius 11/17/14 @SonOfGarr 52
- 53. evcache • distributedin-memory caching – based on memchached & spymemcached • EVCache: – Elastic – Volatile – Cache https://github.com/Netflix/EVCache 11/17/14 @SonOfGarr 53
- 54. servo • exposeapplications metrics – gauges – counters – timers https://github.com/Netflix/servo 11/17/14 @SonOfGarr 54
- 55. astayanax • JavaCassandra client • fluent API • cloud native https://github.com/Netflix/astyanax 11/17/14 @SonOfGarr 55
- 56. governator • GoogleGuice extensions – classpath scanning – lifecycle management – field validation – lazy singleton https://github.com/netflix/Governator 11/17/14 @SonOfGarr 56
- 57. karyon • nucleusof Netflix platform – governator bootstrap – admin console – health checks – discovery https://github.com/Netflix/karyon 11/17/14 @SonOfGarr 57
- 58. prana Generic sidecarw/ HTTP access to Java clients for: – Eureka – Ribbon – Archauis – health checks 11/17/14 @SonOfGarr 58
- 59.
- 60. chaos monkey 11/17/14@SonOfGarr 60
- 61. simian army •chaos monkey • janitor monkey • compliance monkey • security monkey https://github.com/Netflix/SimianArmy 11/17/14 @SonOfGarr 61
- 62.
- 63.
- 64. We are Hiring! Senior Cloud Tools Engineer http://nflx.it/140BnJN Senior Tooling Integration Engineer Mike McGarr mmcgarr@netflix.com @SonOfGarr http://www.mikemcgarr.com http://nflx.it/1xyhkeD 11/17/14 @SonOfGarr 64
Editor's Notes
- #4 ----- Meeting Notes (10/9/14 12:28) ----- voice track here
- #6 47 repos
- #22 ----- Meeting Notes (11/17/14 13:47) ----- redraw as two separate Amis
- #24 -e = environment -b = starting AMI -n = use a named image aminator adds the –ebs to the AMI name
- #45 problems: ZK statically configured limited tools to manage the ensemble backup/restore visualization periodic cleanup
- #54 Ephemeral - The data stored is for a short duration as specified by its TTL(Time To Live). Volatile - The data can disappear any time (Evicted). Cache – An in-memory key-value store.