Netflix oss season 1 episode 3

147,423 views

Published on

lightning talks, overview and cloud prize update

Published in: Technology, Self Improvement
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
147,423
On SlideShare
0
From Embeds
0
Number of Embeds
140,462
Actions
Shares
0
Downloads
95
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide

Netflix oss season 1 episode 3

  1. 1. NetflixOSS Season 1, Episode 3
  2. 2. Lightning talks
  3. 3. Pytheas ● Web based resource and rich UI framework ● Dashboards ● Web console ● Exploring / visualizing structured + unstructured data
  4. 4. Pytheas - based on OSS frameworks ● Guice ● Jersey ● FreeMarker ● JQuery ● DataTables ● D3 ● JQuery-UI ● Bootstrap
  5. 5. Pytheas - Modular Design ● Single jar per module ● Bundles ○ Guice Module ○ Jersey Resources ○ FreeMarker Templates ○ CSS + Img + JS code ○ Properties configuration ○ Pytheas Module class ● Minimal bootstrap code to wire modules together into a dashboard
  6. 6. Pytheas What's next - Angular JS - css/js preprocessing - Scaffold generation https://github.com/Netflix/pytheas
  7. 7. Conformity Monkey Help keep your instances and clusters following best practices
  8. 8. Conformity Monkey ● Mark and Notify approach ● Customize rules ● Customize rule set ● Add new rules ● Customize execution time and frequency ● Can migrate to other cloud platforms
  9. 9. Sample Conformity Rules ● Security Groups ● Instance Configuration In Eureka ● ELB and ASG mismatch ● Cluster Naming Conventions
  10. 10. Conformity Monkey https://github.com/ Netflix/SimianArmy
  11. 11. Zuul ● Netflix's edge tier ○ In place of Nginx, Apache Http or Traffic Server ● Dynamic filters that act on HTTP requests ● Filters written in Groovy / Dynamically updated ○ Any JVM language Supported
  12. 12. Zuul within Netflix ELB Zuul API Service Netflix Services ELB Zuul Streaming Service ELB Zuul Website Service
  13. 13. Request Lifecycle HTTP Request "pre" filters "routing" filter(s) "post" filters Origin Server "custom" filters Http Request Http Response "error" filters
  14. 14. How we Use Zuul ● Authentication ● Insights ● Stress Testing ● Canary Testing ● Dynamic Routing ● Load Shedding ● Security ● Static Response handling ● Isthmus
  15. 15. Zuul Components
  16. 16. https://github.com/Netflix/zuul http://techblog.netflix.com/2013/06/announcing-zuul-edge-service-in-cloud.html
  17. 17. ICE: AWS Usage Tool ● Background ● Design ● Usage ● Next Steps
  18. 18. Background ● There is increased resource usage in the cloud ● That is very decentralized ● But lacks broad and deep insights about resource usage
  19. 19. ICE Design Approach ● Cloud resource usage has many dimensions: account, region, zone, product, usage type, user-defined tags ● ICE writer tags each line item in billing file with different tags ● ICE UI provides a way for users to query cloud resource usage freely in arbitrary dimensions
  20. 20. ICE Usage ● ICE provides a birds-eye view of large and complex cloud landscape from a usage and cost perspective. ● ICE UI is easy and intended to be used by: ● Managers ● Reservation Planners ● Engineers ● ICE OSS at https://github.com/Netflix/ice
  21. 21. ICE Next Steps ● Accessing billing files though IAM role and allow multiple payer accounts ● Create Wiki page and Google Group ● Configurable cost alerts
  22. 22. Genie - Motivation
  23. 23. Genie - Configuration Service ● Abstraction of physical details of back-end Hadoop clusters in the cloud ○ Status: up, out of service, or terminated ○ Cluster naming/tagging for job submissions ○ Site-specific Hadoop, Hive and Pig configurations
  24. 24. Genie - Job Execution Service ● Simple API to run Hadoop, Hive and Pig jobs ○ Auto-magic submission of jobs to the right Hadoop cluster ○ Abstracting away cluster details from clients
  25. 25. Genie - Use Cases Enabled at Netflix ● Running 100% of all SLA jobs, and a high percentage of ad-hoc jobs ● Running nightly short-lived “bonus” clusters to augment ETL processing ● Re-routing traffic between clusters ● “Red/black” pushes for clusters ● Attaching stand-alone gateways to clusters
  26. 26. http://techblog.netflix.com/2013/06/genie-is-out-of-bottle.html https://github.com/Netflix/genie
  27. 27. Lipstick Visualization of Pig workflows at a logical level. ● quickly spot common errors / inefficiencies ● visually monitor execution ● empowers users to support themselves ● facilitates communication between infrastructure team and users
  28. 28. Lipstick
  29. 29. Lipstick in our Architecture
  30. 30. Lipstick UI
  31. 31. Lipstick UI - Pig Workflow Closeup
  32. 32. https://github.com/Netflix/Lipstick http://techblog.netflix.com/2013/06/introducing-lipstick-on-apache-pig.html
  33. 33. Overview and Roadmap
  34. 34. 34 Projects and counting...
  35. 35. Excellent mindshare and vendor adoption Too hard to get started for individual users State of NetflixOSS
  36. 36. Where to Start? ● Recipe app(s) ● AMI's - Asgard available, more coming! ● Hangouts ● Workshop(s) ● One button install.... not there yet.
  37. 37. Build, Deploy, Tools Libraries Libraries , Cont-d. Services ZK / C* Big Data Recipe Apps Aminator Archaius Karyon Eureka Curator Genie Recipes- RSS Asgard Blitz4J Netflix Graph Edda Exhibitor Lipstick Flux Capacitor GCViz Denominator Ribbon EvCache Astyanax ACME Air Ice EvCache RxJava Turbine Priam Simian Army Feign Servo Zuul CassJMeter Pytheas Governator Hystrix Categories
  38. 38. 3rd Party demos
  39. 39. Acme Air OSS Sample and Benchmark ● Fictitious airline sample application ● Mobile app (iOS/Android) and desktop browser versions ● NodeJS and Java versions with multiple NoSQL data tier impls ● Benchmarking framework that allows “Web Scale” runs ● Previously run at 4.3 Billion customer facing REST calls per second (50,000 req/sec) using IBM middleware and cloud ● Re-implemented using Netflix OSS ● Microservices architecture ● Eureka, Hystrix, Karyon, Ribbon ● Asgard ● Run at (peak) ~ 13,000 req/sec – 1.1 Billion per day ● Replicate this yourself in ~ 4 hours with the OSS Learn more at http://bit.ly/acmeairblog and from @aspyker
  40. 40. Eucalyptus, AWS, and NetflixOSS Asgard manages autoscaling across both clouds. Eucalyptus/AWS compatibility makes it possible! Some NetflixOSS services can run in either cloud Simian Army Edda Aminator Priam Servo 30+ NetflixOSS services ...and more on the way!
  41. 41. Confidential and Proprietary ● Multi Data Center Support ● Single Dashboard to manage OpenStack & AWS ● Access multiple Tenants in Single Login ● One-Stop-Shop to manage ● Compute, Storage, Network ● Security ● Cloud Templates ● Asgard Framework and Architecture ● Why Asgard? ● Janitor Monkey for OpenStack ● Future Plans ● Demo HIGHLIGHTS
  42. 42. Riot Games rCloud ●Aminator +chef-solo provisioner ●Asgard ●Edda +Security Audits ●Simian Army ●Ice ●Eureka ●Archaius + Configuration Service ●Ribbon, Karyon, Hystrix ●And more
  43. 43. Progress and Updates NetflixOSS Cloud Prize
  44. 44. State of the Cloud Prize Forks of github.com/netflix/cloud-prize ~700 Mailchimp signup form complete ~120 Public active submissions ~20 Two more months for procrastination! Stickers in return for Mailchimp signup...
  45. 45. Bootcamp/Workshops AMIs, Charms, Cookbooks, Ansible Models More incentives and joint prizes Make it easier to get started
  46. 46. Juju Charm Championship https://juju.ubuntu.com/charm-championship/ Helping solve ease-of-deploy for NetflixOSS Mark Shuttleworth joins Netflix Cloud Prize judges Adrian Cockcroft joins Charm Championship judges Ubuntu Juju
  47. 47. Citrix are offering $10K prize for best integration with NetflixOSS Apache Cloudstack
  48. 48. First prototype! Cloud Prize Trophy
  49. 49. Inspiration
  50. 50. Cloudified
  51. 51. Cloud Monkey It Bleeps! Designed by bleeplabs.com
  52. 52. Q+A and Demos Food and Drink in Building B

×