Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Netflix OSS Meetup Season 4 Episode 4

2,005 views

Published on

In this episode, we will focus on open sourcing how we run Netflix's open source program. Netflix has been using and contributing to open source for several years. Over the years, Netflix has released over one hundred Netflix Open Source (aka NetflixOSS) libraries, servers, and technologies. Netflix engineers benefit by accepting contributions and gathering feedback with key collaborators around the world. Users of NetflixOSS from many industries benefit from our solutions including Big Data, Build and Delivery Tools, Runtime Services and Libraries, Data Persistence, Insight, Reliability and Performance, Security and User Interface. With such a large and mature open source program, Netflix has worked on approaches and tools that help manage and improve the NetflixOSS source offerings and communities. Netflix has taken a different approach to building support for open source as compared to other Internet scale companies. Come to this session to learn about the unique approaches Netflix has taken to both distribute and automate the responsibilities of building a world-class open source program.

Published in: Technology
  • Be the first to comment

Netflix OSS Meetup Season 4 Episode 4

  1. 1. Netflix Open Source Season 4, Episode 4
  2. 2. Introduction
  3. 3. Why does Netflix Open Source? Improve Engineering ● Great feedback from wider community ● Collaborate through open code Recruit new and retain engineering talent ● Hard problems are openly worked on
  4. 4. Industry Alignment Why does Netflix Open Source? Netflix moves to cloud 2008 2013 2016
  5. 5. Netflix Github - http://netflix.github.io
  6. 6. Netflix’s approach to open source Form a small cross-functional team working group that centralizes OSS competence, assisting decentralized teams working with OSS spend less time focusing on the administrative aspects (legal, tooling, branding, monitoring, and community promotion).
  7. 7. Open source enabler - OSS Interest Group ● Internal mailing list ● Meets once per month ● Topics from developers ● Help each other with common problems
  8. 8. Agenda Assisting open source at Netflix ● Github management and security ● Build, CI and release tools Engaging in open source ● Transparency & OSS maturity ● Fostering communities @rusmeshenberg @SonOfGarr @rfletcherEW mikegrima
  9. 9. Agenda Open Source Offices and the TODO Group Nithya A. Ruff Director, Open Source Office, Western Digital @nithyaruff
  10. 10. Github Management & Security Mike Grima
  11. 11. Security Tools - Sensitive Data Leakage ● We scan source code for ○ Access keys, passwords, tokens, hostnames ● We scan code automatically and frequently
  12. 12. Scumblr
  13. 13. Security Tools - User Management ● Provide tools and automation for user access ○ Adding / Removing users ○ Performing privileged tasks ● We aim for self-service as much as possible!
  14. 14. Github Organizational Management ● Management must be easy ○ Otherwise, teams will go it alone (BAD) ● Less is more: fewer orgs = Good
  15. 15. Github Organizational Management BYOGID: ● User links to internal ID ● All tools then can associate identity Two Factor Auth Enforcement ● Automation to boot users who don’t ● Be careful - education on recovery!
  16. 16. Github Organizational Management ● Owners ○ Limited group - due to power ○ Broker owner actions via ChatOps ● Netflixer group ○ Full write permissions on all repos ● Outside collaborators ○ Added by netflixers, validated over time
  17. 17. ChatOps for GitHub Management
  18. 18. ChatOps for GitHub Management
  19. 19. ChatOps for GitHub Management
  20. 20. More advanced commands & DUO!
  21. 21. Building Netflix OSS Mike McGarr
  22. 22. OSS builds needs... ● Consistent build automation ● Continuous integration ● Release software versions ● Publish to JCenter and Maven central ● Validate license files ● Simple and consistent
  23. 23. http://nebula-plugins.github.io
  24. 24. build.gradle file
  25. 25. ● Add contacts to build.gradle ● Bundle build metadata into .jar ● Publish .jar files to Bintray ● Git tag to build release versions ● Optional: lock dependencies Reduce boilerplate
  26. 26. ● Jenkins on Cloudbees ● Setup builds with the Job DSL plugin Continuous integration ● What we didn’t like? ○ Lacked declarative builds ○ Config not in source ○ Complex setup ○ Not as common in OSS community
  27. 27. Travis CI
  28. 28. All commits to master ./gradlew -Prelease.travisci=true build snapshot Releases w/ Nebula + TravisCI Every pull request ./gradlew build Commit tagged with vX.Y.Z-rc# ./gradlew -Prelease.travisci=true candidate Commit tagged with vX.Y.Z ./gradlew -Prelease.travisci=true final
  29. 29. https://jcenter.bintray.com Distributing the binaries
  30. 30. What happens when...
  31. 31. Backup source Netflix OSS ● Backup Github repos to internal Git git remote add internal ssh://git@netflix.internal.net/foo/bar.git
  32. 32. Transparency and OSS Maturity Ruslan Meshenberg @rusmeshenberg
  33. 33. 4 seasons of NetflixOSS Many OSS Projects In 4 years
  34. 34. All wildly successful? Some - yes Some… not so much
  35. 35. What are some of the challenges? Lack of OSS transparency / direction
  36. 36. What are some of the challenges? Internal / OSS Divergence
  37. 37. What are some of the challenges? Maturing and EOL of projects
  38. 38. What are some of the challenges? Separating ideas from code
  39. 39. What are some of the challenges? All leading to variable levels of support
  40. 40. What are we doing about it? Data to the rescue! Org Health Tracking Project Health Tracking Backlog of PRs and Issues
  41. 41. Overall Org Health Tracking Metrics we track ● Issues ○ open, closed, TTC ● Pull Requests ○ open, closed, TTC ● Last commit timing ● Stars/forks ● Num contributors
  42. 42. Project Health Tracking github.com/ Netflix/ OSSTracker
  43. 43. Transparency about project lifecycle OSSMETADATA file: ● Active ● Maintenance ● Archived
  44. 44. Project Ownership All projects have ● Development lead, Management lead ● Shepherd from OSS function area Only projects with active leads stay active!
  45. 45. Transparency about project evolution
  46. 46. Transparency about project evolution
  47. 47. Converging internal and OSS Less maintenance for us You get exactly what we use
  48. 48. Fostering communities Rob Fletcher
  49. 49. What’s in it for us? Tangible contributions ● Leverage enhancements made for other cloud providers ● Titus integration made easier ● Role-based authentication Intangible benefits ● Influencing the conversation ● Validation of concept & implementation ● Recruitment ● Retention
  50. 50. Spinnaker contributions — Clouddriver
  51. 51. Spinnaker contributions — Orca
  52. 52. Encouraging engagement Contributors ● Public roadmap ● “no” > ignoring people ● We can’t do everything — encourage contributions ● Review community PRs & issues regularly ● Make the 1st step easy ● Don’t let docs mislead Open tools ● GitHub ● Slack ● Readme.io ● Travis CI
  53. 53. You gotta do it every day… that’s the hard part
  54. 54. TODO Group Nithya A. Ruff
  55. 55. • Many of us who ran open source program offices shared a private mailing list to commiserate… • It was an avenue to discuss issues in private and even find ways to collaborate on open source projects… • Focused on Silicon Valley companies initially • In 2014 we had an idea of scaling and opening up the the community more… • Announced the TODO Group @Scale 2014 conference!
  56. 56. • TODO Group is a group of companies who want to – collaborate on best practices on running open source program offices – share open source policies and training material – codify quality criteria for well-run open source projects and communities – build and share tools to maintain those quality standards • As we scaled our open source programs, we realized we all built similar tools for the purposes of corporate scale open source… • What is corporate scale open source?
  57. 57. • Corporate participants in open source have a number of unique concerns: – scale (i.e., Google and Microsoft have many open source projects) – insights – cultural – legal / governance • Companies doing open source generally want to be good community citizens, to be open and inclusive, to operate meritocracies. They also need to run a business and be aware of responsibilities to their employees, shareholders and the broader community.
  58. 58. • To establish the TODO Group as a legitimate legal entity, we partnered with the Linux Foundation to make the TODO Group an official collaborative project! • The LF helps with legal paperwork, running events and gives the TODO Group access to its 650+ members
  59. 59. Western Digital and Open Source • Started the Open Source office at SanDisk to engage with the community and go past consumption and compliance • Branded it Open @ SanDisk and became a visible supporter of events and communities • Increased contribution and competency inside the company around open source development models • With the acquisition by Western Digital, created a single office across WD, HGST and SanDisk
  60. 60. Demo Stations Conductor (workflow)
  61. 61. Questions?

×