This document summarizes Netflix's efforts to evolve their open source projects. It discusses establishing clear ownership and lifecycles for projects (active, retired, experimental). It also describes a new dashboard called the Netflix OSS Tracker to monitor project health metrics. The rest of the document demonstrates this Spinnaker continuous delivery platform that Netflix has open sourced and discusses Google's involvement in contributing to and adopting Spinnaker.

1. Netflix Open Source
Season 4, Episode 1
Feb 2016

2. Agenda
● Welcome
● Netflix Open Source Evolution
○ @aspyker
● Spinnaker
○ @ajordens, @mjduftler, @rcarunmsft, @dstengle
● Demo stations and reception

3. Open Source since 2010
● Architecture has shaped public cloud usage
○ Immutability, red/black deploys, chaos, regional
and worldwide high availability
● 102 Open Source Projects in Netflix org

4. Showing some age
● Across our projects
● Open issues
○ Astyanax (148), Asgard (106), Ice (74),
Total (1631)
● Open pull requests
○ Astyanax (44), Ice (23), Asgard (18),
Total (258)
● Some issues & PR’s have
been around for a while
Open Issues Per Project
Open PR’s Per Project

5. What are we doing to evolve?
● Establish ownership and group logically
○ Management and engineering aligned with business
● Classify projects into lifecycles
○ Clearly identify active vs. retired projects
○ Clearly identify experimental projects
● Track active projects
○ Actively work to improve project “health”

6. Establish Ownership and Grouping
● Consistency across
related projects
● Document how area
fits together
● Focus on OSS health
of functional area
OSS Area Shepherds

7. Already taking action
● Astyanax, Asgard, ICE are either not
used or being transitioned away
● Starting to document this better

8. ● OSSMETADATA tagging file
osslifecycle=
active | maintenance | archived
● You can see these today
○ Eventually badges in README.md
● Allows for a tool to automate lifecycle
verification
How do we automate this?

9. Our Dashboard: Netflix OSS Tracker
OSS Tracker
Nightly Jobs
Project Ownership
Database
Time Series
Project Statistics
OSS Tracker
Console
● Collects metrics on every project every day
● Allows shepherds to see combined stats for their area
● Allows project owners to see daily metrics
Whitepages
Service

10. github.com/Netflix/osstracker

11. OSS Tracker Future (Want to help?)
● Pluggable whitepages & database systems
● Collect more metrics from github
○ Time bucketed (last 60 days vs. forever)
○ Internal vs. external contributors
○ Tagging to allow for issues as features, etc.
● Create single consumable health score
● Emailing reports to shepherds and owners
● Better user interface

13. The Netflix Continuous Delivery Platform
Development began in 2014
Open sourced 11/2015
Spinnaker

14. Evolution of CD at Netflix
Asgard
Open Sourced in 2012
Was the primary deployment
tool until 2015
A better AWS Console
Codified Conventions
- Red/Black
- Naming
Pitfalls
Single Account
No first-class API
Mimir
Internal Netflix
Script Support (Jenkins)
Flexible Workflows
Used the “Asgard API”
Edge Center
Internal Netflix
Canaries
Automation
Spinnaker
Application-centric UI
Flexible Pipelines
RESTful API
Multi Account
Canaries (Netflix-only)
https://www.flickr.com/photos/spidermandragon5/2922128673

15. ● Intuitive UI (Self-service)
● Moving Beyond Simple Red/Black
● Out-of-the-box Best Practices
!! Community Participation !!
Goals

16. Abstraction
UI Components
Pipeline Stages

18. Let’s Take a Tour

22. Container
Support
PluggableComponents
UI and Services
Declarative
Continuous
Delivery
Performance
Documentation
Community Site /
Whitepapers /
FAQ

23. ● Netflix and Spinnaker principles align with
Google’s
○ Massive infrastructure, services at
scale
○ Immutable infrastructure
○ Continuous delivery
○ Traceability
○ Self-service tools
Why is Google investing in Spinnaker?

24. ● >30,000 developers in 40+ offices
● 13,000+ projects under active development
● 30k submissions per day (1 every 3 seconds)
● Single monolithic code tree with mixed language code
● Development on one branch - submissions at head
● All builds from source
● 30+ sustained code changes per minute with 90+ peaks
● 50% of code changes monthly
● 150+ million test cases / day, > 150 years of test / day
● Supports continuous deployment for all Google teams!
Internal Google Development

25. ● Developing out GCP toolset
● Our internal platform and tooling is container
oriented
● Spinnaker is a great starting point for VM
oriented tooling
● Build out functionality by contributing to OSS
that aligns with our practices and goals
○ Bazel, Kubernetes, ...
Google Cloud Platform

26. Let’s See it in Action

31. MAKE
NEXT
POSSIBLE

32. ● Professional Services
NetflixOSS
○ Microservices and Cloud
● Spinnaker Launch Contribs
○ Installation and VM Images
○ Docs
Kenzan & Spinnaker

34. ● Reduce Complexity of a true
E2E Spinnaker CD Setup
● Support AWS and GCP
Terraform Spinnaker

36. What you get
● Spinnaker
● Jenkins
● Apt repo
● Networking
● Example App
● Jobs and Pipelines

38. What’s Next
● Productionized Terraform
Deployment
● More sophisticated continuous
deployment pipeline examples

39. Demo Stations
● Spinnaker
○ Snag Spinnaker stickers
○ Netflix Spinnaker demos
○ See Google-Spinnaker integration
○ Meet the Azure team
● Kenzan
○ Spinnaker Terraform demo
○ Learn about their NetflixOSS consulting

40. Demo Stations
Come hear about Dynomite updates
● Dynamo layer to make a non-distributed
data store into distributed
○ Redis/memcache, HA scale-out, encrypted cross-region replication
● Quorum consistency
● Data warm up
● Speed improvements in latest release
○ 12 Dynomite nodes >1M OPS

41. Wrapup
● Thanks for attending!
● Join us for
○ Food and drinks
○ Demo stations