Dependency Management at Scale
In an environment where software is constantly changing and new versions of a library should be distributed across thousands of projects, how can you know that your projects are using the right version of a given dependency? What if a OSS library introduces a security vulnerability and you need to make sure that no one is using it in your company? What if an internal library introduces a bad change and you need everyone to upgrade/downgrade? Automating these for hundreds or thousands of engineers is crucial. At Netflix, engineers are not immune to the cost of dependency updates. Library owners publish new versions of their code without a comprehensive understanding of the organizational impact. Application owners ingest new library versions that can fail in obvious or subtle ways, leading to decreased confidence and slower organizational velocity. But these are problems we understand, and tooling can help. After years of evolving our build, we've developed a few conceptual models of dependency management. Dependency management is hard, and in all cases there are compromises, and you the build owner should be conscious of which choices you're making and what else is available. On this session, we'll introduce some tools we've developed at Netflix which attack dependency issues on a large scale to make it easier for every JVM engineer at Netflix and how we react to the scenarios mentioned above. While most of our tooling is build around Gradle, the learnings from this talk can be applied to other build tools such as Maven.
Recommended
Recommended
More Related Content
What's hot
What's hot (6)
Similar to Dependency Management at Scale
Similar to Dependency Management at Scale (20)
More from Roberto Pérez Alcolea
More from Roberto Pérez Alcolea (7)
Recently uploaded
Recently uploaded (20)
Dependency Management at Scale
- 2. Roberto Pérez Alcolea Senior Software Engineer @ Netflix Productivity Engineering rperezalcolea@netflix.com @rpalcolea DEPENDENCY MANAGEMENT AT SCALE
- 3. •Many repositories (~3k) •Binary integration (JARs) •Microservices with fat clients •Hundreds of engineers DEPENDENCY MANAGEMENT AT SCALE Context
- 4. DEPENDENCY MANAGEMENT AT SCALE Builds per week: local + CI
- 5. Common problems in dependency management
- 6. Adding or upgrading a dependency causes: •NoSuchMethodError •NoSuchFieldError •NoClassDefFoundError •error: cannot find symbol DEPENDENCY MANAGEMENT AT SCALE
- 7. Unaligned dependencies Transitive dependency bumps a single module but not the whole family DEPENDENCY MANAGEMENT AT SCALE
- 8. Picking up dependencies with bugs or security vulnerabilities DEPENDENCY MANAGEMENT AT SCALE
- 9. DEPENDENCY MANAGEMENT AT SCALE Publishers lack visibility • Who will I break? • Who is consuming my library? • Who is using this API that I want to change?
- 10. DEPENDENCY MANAGEMENT AT SCALE Semantic versioning is not enough • Which version should I use? • Pinning versions is technical debt
- 11. DEPENDENCY MANAGEMENT AT SCALE Java version is not compatible
- 12. What can publishers do to alleviate pain?
- 13. DEPENDENCY MANAGEMENT AT SCALE Publish a BOM for family of modules •A BOM enables consumers of a library to select consistent versions for artifacts included in that library. •Examples: ‣ jackson-bom ‣ spring-boot-dependencies •Gradle Java Platform Plugin
- 14. •When making a breaking change: ‣If the new library surface is delivered under a new Java package, you could use a new artifact ID. ‣If the breaking change is made in-place and the Java package is kept the same, use the same artifact ID. •Avoid colliding classes (same package/class under multiple artifactId) DEPENDENCY MANAGEMENT AT SCALE Rename artifacts and packages together
- 15. •Do not upgrade JDK version in your library if you have consumers in older versions •Alternatives ‣Multi Release Jars ‣Separate JARs: Variant awareness with Gradle More: Multi-release JARs - Good or bad idea? DEPENDENCY MANAGEMENT AT SCALE Support the minimum Java version of your consumers
- 16. •Shading is a process where a dependency is relocated to a different Java package and copied into the same jar as the code that relies on that dependency. DEPENDENCY MANAGEMENT AT SCALE Last resource… shading
- 17. Improving consumer experience with Nebula
- 18. This plugin provides general purpose rule types on top of Gradle resolution strategies and module metadata, allowing rules to be published, versioned, shared between projects, and optionally dependency locked. •Duplicate classes caused by changes to group or artifact ids, without renaming packages •Duplicate classes caused by bundle dependencies, which do not conflict resolve against the 'regular' dependencies for that library •Lack of version alignment between libraries, where version alignment is needed for compatibility •Ensuring a minimum version of a library Resolution Rules Plugin DEPENDENCY MANAGEMENT AT SCALE
- 22. How do I know who uses my library?
- 23. •Collection of services and UIs that enable artifact observability and the ability to effect change in the Netflix ecosystem. •High level goal is to decrease time and effort needed to propagate change through the Netflix ecosystem •Useful for Library deprecation and Security Vulnerabilities Astrid DEPENDENCY MANAGEMENT AT SCALE
- 24. Astrid - Artifact Management DEPENDENCY MANAGEMENT AT SCALE
- 25. Astrid - Artifact Management DEPENDENCY MANAGEMENT AT SCALE
- 26. Astrid - Campaigns DEPENDENCY MANAGEMENT AT SCALE
- 28. CVE + Astrid + Nebula DEPENDENCY MANAGEMENT AT SCALE Practical Approach to Automate the Discovery & Eradication of Open-Source Software Vulnerabilities Aladdin Almubayed | 2019
- 29. What else can we do?
- 30. •Distributed Refactoring •Managed Source: automatic PRs for dependency locks, nebula wrapper and lint rules updates. What else can we do? Evolving Continuous Integration at Netflix (Elise McCallum) | 2019 DEPENDENCY MANAGEMENT AT SCALE
- 32. •When a new version of a library is published, build all the consumers •Lack of testing in projects didn’t provide enough confidence for this model Publisher Feedback DEPENDENCY MANAGEMENT AT SCALE Gradle Summit - Netflix | Dependencies, Distributed Code and Engineering Velocity - Mike McGarr | 2017
- 33. Last thoughts… •Dependency management is hard •Dependency hell is inevitable, let’s try to reduce the pain •Build often, release often. Avoid conflict resolution by limiting version skew •Producers and consumers play an important role DEPENDENCY MANAGEMENT AT SCALE