Have you been in the situation where your project builds as expected but a few hours later it won’t anymore? Does this include symptoms such as ClassNotFoundError or NoSuchMethodError exceptions? Well, you are not alone, you probably are or have been in ‘Dependency Hell’ and believe me, we all have been there. It doesn’t matter if you are about to start a green field project or touching an existing one, most likely you will suffer from this at some point in your career. Unfortunately, compilation issues are not the only symptoms of being in ‘Dependency Hell’, another problems are (and not limited to): - Intentional and unintentional Breaking API changes in libraries - Multiple components depending on the same module but with different, incompatible, APIs. - Multiple libraries providing the same feature (multiple logging libraries). - Incompatible versions of a runtime. - Misaligned dependencies of a component - Depending on transitive dependencies that eventually disappear from our projects This list can be extensive and overwhelming! As a developer I just want my build to work! Fortunately, Gradle Build Tool Team has heavily invested engineering efforts for many years in their Dependency Management machinery to address these issues from both sides, the consumer of modules but more importantly, better software modeling for library producers. In this session we will navigate, as library consumers, how we can influence the dependency resolution machinery in Gradle to keep your build in good state and reproducible. In addition, we will explore a few Gradle features to do better software modeling in order to publish good metadata that benefits consumers of our libraries. Some familiarity with Gradle is recommended but not required. Repository with examples can be found in https://github.com/rpalcolea/escaping-dependency-hell-talk-demos-2023 About our speaker Roberto is an experienced software engineer with focus in the JVM ecosystem and developer productivity He has several years of experience using technologies for the JVM. He's an active maintainer of Netflix Nebula Plugins and occasional contributor to the Gradle Build tool Currently works at Netflix in the JVM ecosystem team. The JVM Ecosystem Team provides the user experience for JDK lifecycle, dependency management, building, packaging, and publishing JVM-based libraries and applications through providing tools, automation, and guidance to thousands of engineers at Netflix

1. Escaping
Dependency Hell
A deep dive into Gradle's dependency
management features

2. NETFLIX
SECTION DIVIDER
Roberto Perez Alcolea
Senior Software Engineer @ Netflix
JVM Ecosystem
@rpalcolea
Who am I?

3. — Software Dependencies
— Why Dependency Management
is important?
— Dependency Management: The
Basics
— Dependency Management: Deep
Dive
— Q&A
Agenda

4. Software
Dependencies

5. A relationship between software components where
one component relies on the other to work properly.
Two types:
● Direct: explicitly defined and used by a software
component
● Transitive: A library or module indirectly used by a
software component.
Software Dependency

6. An automated technique for declaring, resolving, and
using functionality required by a project
What is dependency management?

7. Why is
Dependency
Management
important?

8. Developer’s frustration

9. But also…

10. ⚠ ⚠ ⚠ ⚠ ⚠ ⚠ ⚠ ⚠ ⚠
⚠ ⚠
Free and Open Source
Software (FOSS) constitutes
70-90% of any given piece of
modern software solutions
⚠ ⚠ ⚠ ⚠ ⚠ ⚠ ⚠ ⚠ ⚠
⚠ ⚠
A Summary of Census II: Open Source Software Application Libraries the World Depends On

11. We have Maven Central!

12. Keeps growing…
Images from https://www.sonatype.com/state-of-the-software-supply-chain/introduction

13. Images from https://www.sonatype.com/state-of-the-software-supply-chain/introduction
However, there are risks

14. Remember log4shell?

16. And…
Projects evolve fast so
our dependencies

17. The reality is…
Most of times, these
problems come from
transitive dependencies

19. At Netflix, we are
not immune to this

20. NETFLIX
TEXT
Netflix Ecosystem
— 3.5k JVM based repositories
(really an eventually consistent
distributed monolith)
— Binary Integration (JARs)
— Thousands of Builds per day
— Mix of internal and external
libraries/frameworks
— Thousands of artifacts generated
per day that could be deployed at
any time

21. Thankfully, Gradle Dependency
Management Machinery is very
powerful for these scenarios

22. 🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨
🚨🚨🚨
The following can be
applied to any jvm based
project
🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨

23. Dependency
Management
(the basics)

24. Built-in support for declaring
dependencies
Example in https://github.com/rpalcolea/escaping-dependency-hell-talk-demos-2023/tree/main/1-dependency-management-basics

25. View and Debug Dependencies
Example in https://github.com/rpalcolea/escaping-dependency-hell-talk-demos-2023/tree/main/1-dependency-management-basics

26. Powerful Build Scans
Example in https://github.com/rpalcolea/escaping-dependency-hell-talk-demos-2023/tree/main/1-dependency-management-basics

27. Configurations
Images from https://docs.gradle.org/current/userguide/java_plugin.html

28. Dependency Resolution
Two phases until the graph is complete
● When a new dependency is added to the
graph, perform conflict resolution to
determine which version should be added to
the graph.
● When a specific dependency, that is a
module with a version, is identified as part of
the graph, retrieve its metadata so that its
dependencies can be added in turn
(transitives).

29. Conflicts will happen!
Example in https://github.com/rpalcolea/escaping-dependency-hell-talk-demos-2023/tree/main/1-dependency-management-basics
Images from https://jacomet.dev/jfokus23_dependency_management/

30. Version conflict
● Multiple paths to dependency
○ Disagree on version
● Optimistic upgrade strategy
○ Highest version that satisfies all
Example in https://github.com/rpalcolea/escaping-dependency-hell-talk-demos-2023/tree/main/1-dependency-management-basics
Images from https://jacomet.dev/jfokus23_dependency_management/

31. Declaring repositories
Example in https://github.com/rpalcolea/escaping-dependency-hell-talk-demos-2023/tree/main/2-configuring-repositories
The order of declaration determines how Gradle will check for dependencies at
runtime. If Gradle finds a module descriptor in a particular repository, it will attempt
to download all of the artifacts for that module from the same repository.

32. Dependency Cache
Located in $GRADLE_USER_HOME/caches
● A file-based store of downloaded artifacts, including binaries like jars as
well as raw downloaded metadata like POM files and Ivy files.
● A binary store of resolved module metadata, including the results of
resolving dynamic versions, module descriptors, and artifacts.
Example in https://github.com/rpalcolea/escaping-dependency-hell-talk-demos-2023/tree/main/2-configuring-repositories

33. Dependency Cache
For builds on ephemeral environments (ex. containers) you could:
● Copying the dependency cache into each container
● Sharing a read-only dependency cache between multiple containers

34. Dependency
Management
(Deep Dive)

35. Downgrading dependencies
Example in https://github.com/rpalcolea/escaping-dependency-hell-talk-demos-2023/tree/main/3-downgrading-dependencies
Useful for cases where we require an older version and transitives break us

36. Avoid use of exclude!
● Excluding a transitive dependency might lead to runtime errors if
external libraries do not properly function without them
● If excludes are applied globally, there is a huge performance
impact
● No visibility in dependency insight or build scans

37. Upgrading transitive dependencies
Example in https://github.com/rpalcolea/escaping-dependency-hell-talk-demos-2023/tree/main/4-defining-constraints
The version definition for org.apache.logging.log4j:log4j-core:2.21.0 is only taken into account if log4j-core is
brought in as transitive dependency, since log4j-core is not defined as dependency in the project

38. Cross module version conflict
There are libraries that are expected to work together
within a single version. Examples:
● com.fasterxml.jackson-*
● org.springframework:spring-*
● org.springframework.boot:spring-boot-*
● and many others
You want alignment of these dependencies!

39. Option 1: Use BOMs if available!
Example in https://github.com/rpalcolea/escaping-dependency-hell-talk-demos-2023/tree/main/5-use-boms
If you have a family of libraries, consider creating your own!

40. Option 2: Create a virtual platform!
Example in https://github.com/rpalcolea/escaping-dependency-hell-talk-demos-2023/tree/main/6-virtual-platform-alignment

41. Example in https://github.com/rpalcolea/escaping-dependency-hell-talk-demos-2023/tree/main/7-module-replacements
Useful for dependency graphs that accidentally contain multiple
implementations of the same APIs!
Dealing with implementation conflicts

42. Example in https://github.com/rpalcolea/escaping-dependency-hell-talk-demos-2023/tree/main/8-reject-dependency-versions
Sometimes we just don’t want to see certain dependencies in our graph
Rejecting versions

43. We have seen plenty of code so far in
one project and we can re-use it! 😊
with…

44. Gradle Plugins!
Example in https://github.com/rpalcolea/escaping-dependency-hell-talk-demos-2023/tree/main/9-sample-gradle-plugin

45. Existing OSS Plugins
There are a few at your disposal!
● Nebula (Netflix)
○ Gradle Jakarta EE Migration Plugin
○ Nebula Resolution Rules
● Logging capabilities gradle plugin

46. 🔒🔒🔒🔒🔒🔒🔒
🔒
Don’t forget to lock your
dependencies if you use
dynamic versions!
🔒🔒🔒🔒🔒🔒🔒

47. Dependency Locking
Example in https://github.com/rpalcolea/escaping-dependency-hell-talk-demos-2023/tree/main/10-dependency-lock

48. And there
is more 😊!
But we might never end

49. Other things to explore:
● Fail on version conflict
● Consistency resolution across classpaths
● Sharing versions across projects (catalogs)
● Feature Variants
● Dependency Verification

50. Last thoughts…
— Software evolves fast and we
integrate binaries constantly
— Dependency Hell is real, we just
want to reduce the pain
sometimes
— Yes, Dependency Management is
hard! And can be frustrating but
Gradle has a rich feature set to
overcome Dependency
Management Challenges and
provide great insight into your
project’s dependency graphs

51. Questions?
Roberto Perez Alcolea
rperezalcolea@netflix.com

52. Thank
you!