SlideShare a Scribd company logo
1 of 64
Download to read offline
Gene Kim
Author, Researcher
“The Unicorn Project,”
Co-Author: “The Phoenix Project,” “DevOps Handbook,”
“Accelerate”
Exemplars, Laggards, and Hoarders
A Data-driven Look at Open Source Software Supply Chains
Dr. Stephen Magill
CEO, MuseDev
Principal Scientist, Galois, Inc.
@stephenmagill
@stephenmagill @RealGeneKim
• Open Source Software is everywhere
• Nat Friedman, CEO, GitHub: “99% of new software projects include open
source”
• How do these teams you depend on manage updates / security /
testing?
• “You are inviting thousands of developers into your code” when you use open
source dependencies
• Will they help or hurt you? (Erica Brescia, COO, GitHub)
• Which practices correspond to good component security outcomes
and therefore good security for your software?
Problem Statement
State of DevOps Research
• State of DevOps Report (2013-2019)
• Dr. Nicole Forsgren, Jez Humble, Gene Kim
• Cross population study spanning over 35K respondents
• Identified “IT performance” and the factors that predicts:
• Deployment Frequency
• Deployment Lead Time
• Deploy Success Rate
• Mean Time to Restore
Source: Google/DORA: 2018 State Of DevOps Report:
https://cloudplatformonline.com/2018-state-of-devops.html
@stephenmagill @RealGeneKim
• Our goal: Study what structures and practices are correlated
with exemplary outcomes (fast time to update, fast time to
remediate security vulnerabilities)
• Will we find the same trends we do in the enterprise, with faster
delivery correlating with good “business” outcomes?
Goals
Session ID:
The Opportunity: Study the Java Maven Ecosystem!
Session ID:
The Opportunity: Study the Java Maven Ecosystem!
Clojure
Haskell
Dr. Stephen Magill (Galois)
Gene Kim (IT Revolution)
Bruce Mayhew (Sonatype)
Gazi Mahmud (Sonatype)
Thanks also to:
Kevin Witten, Derek Weeks,
and Matt Howard
@stephenmagill @RealGeneKim
• Hypothesis 1: Projects that release frequently have better
outcomes.
• Hypothesis 2: Projects that update dependencies more
frequently are generally more secure.
• Hypothesis 3: Projects with fewer dependencies will stay more
up to date.
• Hypothesis 4: More popular projects will be better about staying
up to date.
Hypotheses *
310,888
Java components
4.2 Million
Artifacts (JARs)
6,952
GitHub Repos
27,704
8.9% with known
vulnerabilities
@RealGeneKim@stephenmagill
MavenCentral
TODO: Dependency Graph
Visualization
TODO: Dependency Graph
Visualization
@RealGeneKim@stephenmagill
@RealGeneKim@stephenmagill
Maven Central
@RealGeneKim@stephenmagill
Recent
@RealGeneKim@stephenmagill
Connected
@RealGeneKim@stephenmagill
Correct
Versioning
@RealGeneKim@stephenmagill
All of the above
for dependencies
@RealGeneKim@stephenmagill
Updated
Dependencies
13.6% of the total population
Attributes Measure
Popularity Avg. daily Central Repository downloads
Release Frequency Avg. period between releases
Development Activity Avg. commits per month
Size of Team Avg. unique monthly contributors
Presence of CI Presence of popular cloud CI systems
Foundation Support Associated with an open source foundation
Security Based on reported vulnerabilities
Update Lag Based on dependency updates
@RealGeneKim@stephenmagill
@stephenmagill @RealGeneKim
• Popularity
• Main component: Average number of downloads per day from The Central Repository.
• Also used the Libraries.io dataset: Number of GitHub stars, forks, and pull requests.
• Sonatype Nexus IQ Server: Popularity score based on how frequently components are seen by the Nexus IQ
repository scanning service
• Commit activity
• SCM Commits per Month – average number of commits per month (Perceval)
• Developer Team Size –average number of unique developers committing each month (Perceval)
• (8 core VM scanning repositories for three days: Clojure wrapper around Perceval and jq)
• Presence of Continuous Integration (CI): as measured by the detection of any CI-related
configuration files in the source code repository (e.g., Travis, Jenkins, CircleCI, etc.).
• Clojure program retrieving HTML from GitHub repo, regular expressions to detect CI
Data Gathered: Repositories*
We used the CHAOSS Perceval utility to gather GitHub commit data, we gathered the number of commits per
month for twelve months, as well as the number of unique developers committing during each month.
Thank you to CHAOSS and Libaries.io for your amazing tools and data!
@stephenmagill @RealGeneKim
• Support Type: support for the component comes from an open source foundation, a
commercial organization, or is not officially supported by any organization (e.g., a
personal project).
• Number of Dependencies: the maximum count of dependencies for any given component
across all versions in the study period, as measured by the dependencies in the Maven
pom.xml file.
• Stale Dependencies (fewer is better): the average percentage of out-of-date component
dependencies (i.e., a newer version has been released) present when the component has
a new release.
• Release Period (shorter is better): average time in days each component version spends
as the “current” release. A shorter average release period equates to more frequent
releases.
Data Gathered: Project-Level *
ITPERF (2013-2019) Software Supply Chain (2019)
Deployment Frequency
• Commits / month *
• Releases / month
• Commits / dev / month *
Deployment Lead Time
• PR lead time
• Issue resolution time
Deploy Success Rate • API Breakage rate, Build Breakage Rate, PR Breakage Rate
Mean Time to Restore
• MTTR (mean time to remediate security vulnerabilities) *
• MTTU (mean time to update available components) *
• Age of stale / vulnerable dependencies *
Org Perf
• Stars / Popularity / Download count *
Thoughts On ITPERF <-> SSC Metrics
* Explored in this year’s research: 2019 State of the Software Supply Chain
@RealGeneKim
Hypothesis 1
Projects that release frequently have better outcomes.
(State of DevOps Report shows decisively that
shorter deployment lead times and
higher release frequency
improves outcomes)
@stephenmagill
*
@RealGeneKim
Hypothesis 1
Projects that release frequently have better outcomes.
(VALIDATED)
@stephenmagill
*
Projects that release most frequently (top 20%):
are 5x more popular (Maven Central downloads, GitHub stars and forks)
have 79% more developers
have 12% greater foundation support rates.
@RealGeneKim@stephenmagill
Attributes Measure
Popularity Avg. daily Central Repository downloads
Release Frequency Avg. period between releases
Development Speed Avg. commits per month
Size of Team Avg. unique monthly contributors
Presence of CI Presence of popular cloud CI systems
Foundation Support Associated with an open source foundation
Security Based on reported vulnerabilities
Update Speed Based on dependency updates
@RealGeneKim@stephenmagill
Dependency-Level Metrics
Security: Time to Remediate (TTR)
@RealGeneKim@stephenmagill
Security: Time to Remediate (TTR)
@RealGeneKim@stephenmagill
Security: Time to Remediate (TTR)
@RealGeneKim@stephenmagill
Security: Time to Remediate (TTR)
@RealGeneKim@stephenmagill
Security: Time to Remediate (TTR)
@RealGeneKim@stephenmagill
Security: Time to Remediate (TTR)
@RealGeneKim
B Vulnerable Time
@stephenmagill
Security: Time to Remediate (TTR)
@RealGeneKim
C Vulnerable Time
@stephenmagill
Security: Time to Remediate (TTR)
@RealGeneKim
C Remediation Time
@stephenmagill
Security: Time to Update (TTU)
@RealGeneKim
C Update Time (for B)
@stephenmagill
Security: Time to Update (TTU)
@RealGeneKim
C Update Time (for A)
@stephenmagill
Security: Stale Dependencies
@RealGeneKim
Stale Dependency
@stephenmagill
@RealGeneKim
The Key Dependency Metrics
(per-update)
Time to Remediate
Time to Update
Stale Dependencies
@stephenmagill
@RealGeneKim
The Key Dependency Metrics
(per-project)
Median Time to Remediate
Median Time to Update
Median Stale Dependencies
@stephenmagill
Time to Remediate Security Vulnerabilities*
Time to Remediate Security Vulnerabilities
Do these update
quickly in general?
Time to Remediate vs. Time to Update Dependencies (TTU)
Time to Remediate (TTR) vs. Time to Update (TTU) *
@RealGeneKim
Pearson correlation 0.6
@stephenmagill
Most projects stay secure by staying up to date.
55% have MTTR and MTTU within 20% of each other.
Only 15% of projects with worse than average MTTU
manage to maintain better than average MTTR.
@RealGeneKim@stephenmagill
Time to Remediate (TTR) vs. Time to Update (TTU)
@RealGeneKim
Pearson correlation 0.6
@stephenmagill
Most projects stay secure by staying up to date.
55% have MTTR and MTTU within 20% of each other.
Only 15% of projects with worse than average MTTU
manage to maintain better than average MTTR.
@RealGeneKim@stephenmagill
Hypothesis 2
Projects that update dependencies more frequently
are generally more secure.
@RealGeneKim@stephenmagill
Hypothesis 2
Projects that update dependencies more frequently
are generally more secure.
(VALIDATED)
@RealGeneKim@stephenmagill
*PrimeFaces CVE-2017-1000486: published 1/3/2018; vuln unreported as CVE; was fixed in
2/2016; cryptominers started using it (Source: Jeremy Long: @ctxt)
Hypothesis 3
Projects with fewer dependencies will stay more up to date.
@RealGeneKim@stephenmagill
Hypothesis 3
Projects with fewer dependencies will stay more up to date.
(REFUTED)
Components with more dependencies actually have better MTTU.
@RealGeneKim@stephenmagill
More dependencies
correlate with larger
development teams.
@RealGeneKim
Larger development
teams have 50% faster
MTTU and release 2.6x
more frequently.
@stephenmagill
More dependencies
correlate with larger
development teams.
@RealGeneKim
Larger development
teams have 50% faster
MTTU and release 2.6x
more frequently.
@stephenmagill
@RealGeneKim
Hypothesis 4
More popular projects will be better about staying up to date.
@stephenmagill
Not all popular
projects are
exemplary and
release fast
(10-10K downloads
per day)
@RealGeneKim
Hypothesis 4
More popular projects will be better about staying up to date.
(REFUTED)
There are plenty of popular components with poor MTTU.
Popularity does not correlate with MTTU.
The most popular projects are not statistically different
from others with respect to MTTU.
@stephenmagill
@RealGeneKim
Number of stars or number of forks
IS NOT AN EFFECTIVE HEURISTIC
for selecting which components to use
(if security is important to you)
@stephenmagill
5 Behavioral Clusters for OSS “Suppliers”
@RealGeneKim
Small Exemplar
(606)
Large Exemplar
(595)
Small development
teams (1.6 devs),
exemplary MTTU.
Large development teams (8.9
devs), exemplary MTTU, very
likely to be foundation supported,
11x more popular.
@stephenmagill
Laggards
(521)
Features First
(280)
Cautious
(429)
Poor MTTU, high
stale dependency
count, more likely to
be commercially
supported.
Frequent releases,
but poor TTU.
Still reasonably
popular.
Good TTU,
but seldom
completely up
to date.
@stephenmagill @RealGeneKim
• We conducted survey with 658 respondents who completed it — three clusters
emerged, which we called “high, medium, and low update pain” clusters
• Comparison between “high pain” vs. “low pain” clusters:
• Updating dependencies is painful: 3.2x less likely to strongly agree
• Updating vulnerable components is painful: 2.6x less likely
• We schedule updating dependencies as part of our daily work: 10x more likely
• We strive to use the latest version (or latest-N) of all our dependencies: 6.2x more likely
• We use some process to add a new dependency (e.g., evaluate, approve, standardize,
etc.): 11x more likely
• We have a process to proactively remove problematic or unused dependencies: 9.3x
more likely
• We have automated tools to track, manage, and/or ensure policy compliance of our
dependencies: 12x more likely
Exemplars: Survey Data (N=658) *
Dr. Stephen Magill (Galois)
Gene Kim (IT Revolution)
Bruce Mayhew (Sonatype)
Gazi Mahmud (Sonatype)
Thanks also to:
Kevin Witten, Derek Weeks,
and Matt Howard
stephen@muse.dev
@stephenmagill @RealGeneKim
• Study further breaking changes
• Look at transitive dependencies
• Identify leading indicators, use techniques to assert causation
Year 2 Goals
@stephenmagill @RealGeneKim
@stephenmagill @RealGeneKim
• Ways to detect breaking changes
• Outcomes resulting from Dependabot pull requests?
• For which components are updates quickly and painlessly applied
• For which components are updates never applied (i.e., because they break everything)
• Which components have a disciplined and immutable API that allows for easier
upgrades?
• E.g., Clojure programming language and standard library have had virtually no breaking changes in
12 years
• E.g., React-native: “4 months after not touching it, it no longer builds if you update all the
dependencies”
• Get data on pull request lead time and issue resolution time
• (DONE? Thank you code.gov!)
• Authoritative list of foundation-supported projects?
Help We’re Looking For
Quick Takeaways
Integrate updating dependencies into your daily work!
Contribute dependency updates to components you use!
Don’t make decisions based solely on popularity!
Tell us what hypotheses you would like to see investigated!
@RealGeneKim@stephenmagill
stephen@muse.dev

More Related Content

What's hot

Keeping The Auditor Away: DevOps Audit Compliance Case Studies
Keeping The Auditor Away: DevOps Audit Compliance Case StudiesKeeping The Auditor Away: DevOps Audit Compliance Case Studies
Keeping The Auditor Away: DevOps Audit Compliance Case StudiesGene Kim
 
DevOps Kanban Meet Up 3/22/12
DevOps Kanban Meet Up 3/22/12DevOps Kanban Meet Up 3/22/12
DevOps Kanban Meet Up 3/22/12Gene Kim
 
Infosec at Ludicrous Speeds - Rugged DevOps
Infosec at Ludicrous Speeds - Rugged DevOps Infosec at Ludicrous Speeds - Rugged DevOps
Infosec at Ludicrous Speeds - Rugged DevOps Gene Kim
 
DevOps State of the Union 2015
DevOps State of the Union 2015DevOps State of the Union 2015
DevOps State of the Union 2015Ernest Mueller
 
Winnipeg ISACA Security is Dead, Rugged DevOps
Winnipeg ISACA Security is Dead, Rugged DevOpsWinnipeg ISACA Security is Dead, Rugged DevOps
Winnipeg ISACA Security is Dead, Rugged DevOpsGene Kim
 
ServiceNow ITIL at Ludicrous Speeds - Rugged DevOps
ServiceNow  ITIL at Ludicrous Speeds - Rugged DevOpsServiceNow  ITIL at Ludicrous Speeds - Rugged DevOps
ServiceNow ITIL at Ludicrous Speeds - Rugged DevOpsGene Kim
 
2013 Velocity DevOps Metrics -- It's Not Just For WebOps Any More!
2013 Velocity DevOps Metrics -- It's Not Just For WebOps Any More!2013 Velocity DevOps Metrics -- It's Not Just For WebOps Any More!
2013 Velocity DevOps Metrics -- It's Not Just For WebOps Any More!Gene Kim
 
Kim itSMF New England: ITIL at Ludicrous Speeds - Rugged DevOps 6a
Kim itSMF New England: ITIL at Ludicrous Speeds - Rugged DevOps 6aKim itSMF New England: ITIL at Ludicrous Speeds - Rugged DevOps 6a
Kim itSMF New England: ITIL at Ludicrous Speeds - Rugged DevOps 6aGene Kim
 
2014 State Of DevOps Findings! Velocity Conference
2014 State Of DevOps Findings! Velocity Conference2014 State Of DevOps Findings! Velocity Conference
2014 State Of DevOps Findings! Velocity ConferenceGene Kim
 
PuppetConf2012GeneKim
PuppetConf2012GeneKimPuppetConf2012GeneKim
PuppetConf2012GeneKimGene Kim
 
SecureWorld - Communicating With Your CFO
SecureWorld - Communicating With Your CFOSecureWorld - Communicating With Your CFO
SecureWorld - Communicating With Your CFOGene Kim
 
Kim IT Pro Forum Eugene: IT at Ludicrous Speeds - rugged dev ops
Kim IT Pro Forum Eugene: IT at Ludicrous Speeds - rugged dev opsKim IT Pro Forum Eugene: IT at Ludicrous Speeds - rugged dev ops
Kim IT Pro Forum Eugene: IT at Ludicrous Speeds - rugged dev opsGene Kim
 
Why Everyone Needs DevOps Now - Gene Kim
Why Everyone Needs DevOps Now - Gene KimWhy Everyone Needs DevOps Now - Gene Kim
Why Everyone Needs DevOps Now - Gene KimDynatrace
 
When IT Fails The Business Fails...
When IT Fails The Business Fails...When IT Fails The Business Fails...
When IT Fails The Business Fails...Gene Kim
 
Tui the phoenix project book review
Tui the phoenix project book reviewTui the phoenix project book review
Tui the phoenix project book reviewRudiger Wolf
 
SecureWorld Kim - Infosec at Ludicrous Speeds - Rugged DevOps 6a
SecureWorld   Kim - Infosec at Ludicrous Speeds - Rugged DevOps 6aSecureWorld   Kim - Infosec at Ludicrous Speeds - Rugged DevOps 6a
SecureWorld Kim - Infosec at Ludicrous Speeds - Rugged DevOps 6aGene Kim
 
Top Lessons Learned From The DevOps Handbook
Top Lessons Learned From The DevOps HandbookTop Lessons Learned From The DevOps Handbook
Top Lessons Learned From The DevOps HandbookXebiaLabs
 
2012 Velocity London: DevOps Patterns Distilled
2012 Velocity London: DevOps Patterns Distilled2012 Velocity London: DevOps Patterns Distilled
2012 Velocity London: DevOps Patterns DistilledGene Kim
 
Keynote: The Phoenix Project: Lessons Learned - PuppetConf 2014
Keynote: The Phoenix Project: Lessons Learned - PuppetConf 2014Keynote: The Phoenix Project: Lessons Learned - PuppetConf 2014
Keynote: The Phoenix Project: Lessons Learned - PuppetConf 2014Puppet
 

What's hot (20)

Keeping The Auditor Away: DevOps Audit Compliance Case Studies
Keeping The Auditor Away: DevOps Audit Compliance Case StudiesKeeping The Auditor Away: DevOps Audit Compliance Case Studies
Keeping The Auditor Away: DevOps Audit Compliance Case Studies
 
DevOps Kanban Meet Up 3/22/12
DevOps Kanban Meet Up 3/22/12DevOps Kanban Meet Up 3/22/12
DevOps Kanban Meet Up 3/22/12
 
Infosec at Ludicrous Speeds - Rugged DevOps
Infosec at Ludicrous Speeds - Rugged DevOps Infosec at Ludicrous Speeds - Rugged DevOps
Infosec at Ludicrous Speeds - Rugged DevOps
 
DevOps State of the Union 2015
DevOps State of the Union 2015DevOps State of the Union 2015
DevOps State of the Union 2015
 
Winnipeg ISACA Security is Dead, Rugged DevOps
Winnipeg ISACA Security is Dead, Rugged DevOpsWinnipeg ISACA Security is Dead, Rugged DevOps
Winnipeg ISACA Security is Dead, Rugged DevOps
 
ServiceNow ITIL at Ludicrous Speeds - Rugged DevOps
ServiceNow  ITIL at Ludicrous Speeds - Rugged DevOpsServiceNow  ITIL at Ludicrous Speeds - Rugged DevOps
ServiceNow ITIL at Ludicrous Speeds - Rugged DevOps
 
2013 Velocity DevOps Metrics -- It's Not Just For WebOps Any More!
2013 Velocity DevOps Metrics -- It's Not Just For WebOps Any More!2013 Velocity DevOps Metrics -- It's Not Just For WebOps Any More!
2013 Velocity DevOps Metrics -- It's Not Just For WebOps Any More!
 
Kim itSMF New England: ITIL at Ludicrous Speeds - Rugged DevOps 6a
Kim itSMF New England: ITIL at Ludicrous Speeds - Rugged DevOps 6aKim itSMF New England: ITIL at Ludicrous Speeds - Rugged DevOps 6a
Kim itSMF New England: ITIL at Ludicrous Speeds - Rugged DevOps 6a
 
2014 State Of DevOps Findings! Velocity Conference
2014 State Of DevOps Findings! Velocity Conference2014 State Of DevOps Findings! Velocity Conference
2014 State Of DevOps Findings! Velocity Conference
 
PuppetConf2012GeneKim
PuppetConf2012GeneKimPuppetConf2012GeneKim
PuppetConf2012GeneKim
 
SecureWorld - Communicating With Your CFO
SecureWorld - Communicating With Your CFOSecureWorld - Communicating With Your CFO
SecureWorld - Communicating With Your CFO
 
Kim IT Pro Forum Eugene: IT at Ludicrous Speeds - rugged dev ops
Kim IT Pro Forum Eugene: IT at Ludicrous Speeds - rugged dev opsKim IT Pro Forum Eugene: IT at Ludicrous Speeds - rugged dev ops
Kim IT Pro Forum Eugene: IT at Ludicrous Speeds - rugged dev ops
 
Why Everyone Needs DevOps Now - Gene Kim
Why Everyone Needs DevOps Now - Gene KimWhy Everyone Needs DevOps Now - Gene Kim
Why Everyone Needs DevOps Now - Gene Kim
 
When IT Fails The Business Fails...
When IT Fails The Business Fails...When IT Fails The Business Fails...
When IT Fails The Business Fails...
 
Tui the phoenix project book review
Tui the phoenix project book reviewTui the phoenix project book review
Tui the phoenix project book review
 
SecureWorld Kim - Infosec at Ludicrous Speeds - Rugged DevOps 6a
SecureWorld   Kim - Infosec at Ludicrous Speeds - Rugged DevOps 6aSecureWorld   Kim - Infosec at Ludicrous Speeds - Rugged DevOps 6a
SecureWorld Kim - Infosec at Ludicrous Speeds - Rugged DevOps 6a
 
DevOps and Audit
DevOps and AuditDevOps and Audit
DevOps and Audit
 
Top Lessons Learned From The DevOps Handbook
Top Lessons Learned From The DevOps HandbookTop Lessons Learned From The DevOps Handbook
Top Lessons Learned From The DevOps Handbook
 
2012 Velocity London: DevOps Patterns Distilled
2012 Velocity London: DevOps Patterns Distilled2012 Velocity London: DevOps Patterns Distilled
2012 Velocity London: DevOps Patterns Distilled
 
Keynote: The Phoenix Project: Lessons Learned - PuppetConf 2014
Keynote: The Phoenix Project: Lessons Learned - PuppetConf 2014Keynote: The Phoenix Project: Lessons Learned - PuppetConf 2014
Keynote: The Phoenix Project: Lessons Learned - PuppetConf 2014
 

Similar to GitHub Universe: 2019: Exemplars, Laggards, and Hoarders A Data-driven Look at Open Source Software Supply Chains

DevOps Days Columbus - Derek Weeks - 2019
DevOps Days Columbus - Derek Weeks - 2019DevOps Days Columbus - Derek Weeks - 2019
DevOps Days Columbus - Derek Weeks - 2019Sonatype
 
DevOps and the Importance of Single Source Code Repos 
DevOps and the Importance of Single Source Code Repos DevOps and the Importance of Single Source Code Repos 
DevOps and the Importance of Single Source Code Repos Perforce
 
Why Everyone Needs DevOps Now: 15 Year Study Of High Performing Technology Orgs
Why Everyone Needs DevOps Now: 15 Year Study Of High Performing Technology OrgsWhy Everyone Needs DevOps Now: 15 Year Study Of High Performing Technology Orgs
Why Everyone Needs DevOps Now: 15 Year Study Of High Performing Technology OrgsGene Kim
 
DevOps Patterns Distilled: Implementing The Needed Practices In Practical Steps
DevOps Patterns Distilled: Implementing The Needed Practices In Practical StepsDevOps Patterns Distilled: Implementing The Needed Practices In Practical Steps
DevOps Patterns Distilled: Implementing The Needed Practices In Practical StepsCA Technologies
 
Divine and felonios cyber security devopsdays austin 2018
Divine and felonios cyber security  devopsdays austin 2018Divine and felonios cyber security  devopsdays austin 2018
Divine and felonios cyber security devopsdays austin 2018John Willis
 
My Top Five DevOps Learnings
My Top Five DevOps LearningsMy Top Five DevOps Learnings
My Top Five DevOps LearningsPredix
 
Git into the Flow, with the Ultimate Continuous Delivery Workflow on Heroku
Git into the Flow, with the Ultimate Continuous Delivery Workflow on HerokuGit into the Flow, with the Ultimate Continuous Delivery Workflow on Heroku
Git into the Flow, with the Ultimate Continuous Delivery Workflow on HerokuSalesforce Developers
 
The Rationale for Continuous Delivery
The Rationale for Continuous DeliveryThe Rationale for Continuous Delivery
The Rationale for Continuous DeliveryPerforce
 
Top Lessons Learned While Researching and Writing The DevOps Handbook
Top Lessons Learned While Researching and Writing The DevOps HandbookTop Lessons Learned While Researching and Writing The DevOps Handbook
Top Lessons Learned While Researching and Writing The DevOps HandbookDynatrace
 
On the health of the npm packaging ecosystem
On the health of the npm packaging ecosystemOn the health of the npm packaging ecosystem
On the health of the npm packaging ecosystemTom Mens
 
The Unicorn Project and the Five Ideals.pdf
The Unicorn Project and the Five Ideals.pdfThe Unicorn Project and the Five Ideals.pdf
The Unicorn Project and the Five Ideals.pdfVMware Tanzu
 
Cloud-Native Fundamentals: Accelerating Development with Continuous Integration
Cloud-Native Fundamentals: Accelerating Development with Continuous IntegrationCloud-Native Fundamentals: Accelerating Development with Continuous Integration
Cloud-Native Fundamentals: Accelerating Development with Continuous IntegrationVMware Tanzu
 
Edit Privacy Settings Analytics FREE Collect Leads Micro Focus DevOps Drive-i...
Edit Privacy Settings Analytics FREE Collect Leads Micro Focus DevOps Drive-i...Edit Privacy Settings Analytics FREE Collect Leads Micro Focus DevOps Drive-i...
Edit Privacy Settings Analytics FREE Collect Leads Micro Focus DevOps Drive-i...Serena Software
 
Open Source Insight: Balancing Agility and Open Source Security for DevOps
Open Source Insight: Balancing Agility and Open Source Security for DevOpsOpen Source Insight: Balancing Agility and Open Source Security for DevOps
Open Source Insight: Balancing Agility and Open Source Security for DevOpsBlack Duck by Synopsys
 
Markings of a Healthy OSS Project
Markings of a Healthy OSS ProjectMarkings of a Healthy OSS Project
Markings of a Healthy OSS ProjectClement Ho
 
Is my software ecosystem healthy? It depends!
Is my software ecosystem healthy? It depends!Is my software ecosystem healthy? It depends!
Is my software ecosystem healthy? It depends!Tom Mens
 
Agile Secure Cloud Application Development Management
Agile Secure Cloud Application Development ManagementAgile Secure Cloud Application Development Management
Agile Secure Cloud Application Development ManagementAdam Getchell
 
Hidden Speed Bumps on the Road to "Continuous"
Hidden Speed Bumps on the Road to "Continuous"Hidden Speed Bumps on the Road to "Continuous"
Hidden Speed Bumps on the Road to "Continuous"Sonatype
 

Similar to GitHub Universe: 2019: Exemplars, Laggards, and Hoarders A Data-driven Look at Open Source Software Supply Chains (20)

DevOps Days Columbus - Derek Weeks - 2019
DevOps Days Columbus - Derek Weeks - 2019DevOps Days Columbus - Derek Weeks - 2019
DevOps Days Columbus - Derek Weeks - 2019
 
DevOps and the Importance of Single Source Code Repos 
DevOps and the Importance of Single Source Code Repos DevOps and the Importance of Single Source Code Repos 
DevOps and the Importance of Single Source Code Repos 
 
Why Everyone Needs DevOps Now: 15 Year Study Of High Performing Technology Orgs
Why Everyone Needs DevOps Now: 15 Year Study Of High Performing Technology OrgsWhy Everyone Needs DevOps Now: 15 Year Study Of High Performing Technology Orgs
Why Everyone Needs DevOps Now: 15 Year Study Of High Performing Technology Orgs
 
DevOps Patterns Distilled: Implementing The Needed Practices In Practical Steps
DevOps Patterns Distilled: Implementing The Needed Practices In Practical StepsDevOps Patterns Distilled: Implementing The Needed Practices In Practical Steps
DevOps Patterns Distilled: Implementing The Needed Practices In Practical Steps
 
Divine and felonios cyber security devopsdays austin 2018
Divine and felonios cyber security  devopsdays austin 2018Divine and felonios cyber security  devopsdays austin 2018
Divine and felonios cyber security devopsdays austin 2018
 
My Top Five DevOps Learnings
My Top Five DevOps LearningsMy Top Five DevOps Learnings
My Top Five DevOps Learnings
 
Git into the Flow, with the Ultimate Continuous Delivery Workflow on Heroku
Git into the Flow, with the Ultimate Continuous Delivery Workflow on HerokuGit into the Flow, with the Ultimate Continuous Delivery Workflow on Heroku
Git into the Flow, with the Ultimate Continuous Delivery Workflow on Heroku
 
The Rationale for Continuous Delivery
The Rationale for Continuous DeliveryThe Rationale for Continuous Delivery
The Rationale for Continuous Delivery
 
Top Lessons Learned While Researching and Writing The DevOps Handbook
Top Lessons Learned While Researching and Writing The DevOps HandbookTop Lessons Learned While Researching and Writing The DevOps Handbook
Top Lessons Learned While Researching and Writing The DevOps Handbook
 
On the health of the npm packaging ecosystem
On the health of the npm packaging ecosystemOn the health of the npm packaging ecosystem
On the health of the npm packaging ecosystem
 
The Unicorn Project and the Five Ideals.pdf
The Unicorn Project and the Five Ideals.pdfThe Unicorn Project and the Five Ideals.pdf
The Unicorn Project and the Five Ideals.pdf
 
Cloud-Native Fundamentals: Accelerating Development with Continuous Integration
Cloud-Native Fundamentals: Accelerating Development with Continuous IntegrationCloud-Native Fundamentals: Accelerating Development with Continuous Integration
Cloud-Native Fundamentals: Accelerating Development with Continuous Integration
 
Edit Privacy Settings Analytics FREE Collect Leads Micro Focus DevOps Drive-i...
Edit Privacy Settings Analytics FREE Collect Leads Micro Focus DevOps Drive-i...Edit Privacy Settings Analytics FREE Collect Leads Micro Focus DevOps Drive-i...
Edit Privacy Settings Analytics FREE Collect Leads Micro Focus DevOps Drive-i...
 
Open Source Insight: Balancing Agility and Open Source Security for DevOps
Open Source Insight: Balancing Agility and Open Source Security for DevOpsOpen Source Insight: Balancing Agility and Open Source Security for DevOps
Open Source Insight: Balancing Agility and Open Source Security for DevOps
 
Docker and Devops
Docker and DevopsDocker and Devops
Docker and Devops
 
Markings of a Healthy OSS Project
Markings of a Healthy OSS ProjectMarkings of a Healthy OSS Project
Markings of a Healthy OSS Project
 
Software as Craft
Software as CraftSoftware as Craft
Software as Craft
 
Is my software ecosystem healthy? It depends!
Is my software ecosystem healthy? It depends!Is my software ecosystem healthy? It depends!
Is my software ecosystem healthy? It depends!
 
Agile Secure Cloud Application Development Management
Agile Secure Cloud Application Development ManagementAgile Secure Cloud Application Development Management
Agile Secure Cloud Application Development Management
 
Hidden Speed Bumps on the Road to "Continuous"
Hidden Speed Bumps on the Road to "Continuous"Hidden Speed Bumps on the Road to "Continuous"
Hidden Speed Bumps on the Road to "Continuous"
 

More from Gene Kim

Speaker Recording Tips For Virtual DevOps Enterprise (And Why We're Pre-Recor...
Speaker Recording Tips For Virtual DevOps Enterprise (And Why We're Pre-Recor...Speaker Recording Tips For Virtual DevOps Enterprise (And Why We're Pre-Recor...
Speaker Recording Tips For Virtual DevOps Enterprise (And Why We're Pre-Recor...Gene Kim
 
Kevin Behr: Integrating Controls and Process Improvement
Kevin Behr: Integrating Controls and Process ImprovementKevin Behr: Integrating Controls and Process Improvement
Kevin Behr: Integrating Controls and Process ImprovementGene Kim
 
United2012 Rugged DevOps Rocks
United2012 Rugged DevOps RocksUnited2012 Rugged DevOps Rocks
United2012 Rugged DevOps RocksGene Kim
 
2012 05 corp fin 1c
2012 05 corp fin 1c2012 05 corp fin 1c
2012 05 corp fin 1cGene Kim
 
SecureWorld: Security is Dead, Rugged DevOps 1f
SecureWorld:  Security is Dead, Rugged DevOps 1fSecureWorld:  Security is Dead, Rugged DevOps 1f
SecureWorld: Security is Dead, Rugged DevOps 1fGene Kim
 
2012 SxSW When IT Says No by Gene Kim
2012 SxSW When IT Says No by Gene Kim2012 SxSW When IT Says No by Gene Kim
2012 SxSW When IT Says No by Gene KimGene Kim
 
Security is Dead. Long Live Rugged DevOps: IT at Ludicrous Speed
Security is Dead. Long Live Rugged DevOps: IT at Ludicrous SpeedSecurity is Dead. Long Live Rugged DevOps: IT at Ludicrous Speed
Security is Dead. Long Live Rugged DevOps: IT at Ludicrous SpeedGene Kim
 
2011 09 19 LSPE Dev Ops Cookbook 1a
2011 09 19 LSPE Dev Ops Cookbook 1a2011 09 19 LSPE Dev Ops Cookbook 1a
2011 09 19 LSPE Dev Ops Cookbook 1aGene Kim
 

More from Gene Kim (8)

Speaker Recording Tips For Virtual DevOps Enterprise (And Why We're Pre-Recor...
Speaker Recording Tips For Virtual DevOps Enterprise (And Why We're Pre-Recor...Speaker Recording Tips For Virtual DevOps Enterprise (And Why We're Pre-Recor...
Speaker Recording Tips For Virtual DevOps Enterprise (And Why We're Pre-Recor...
 
Kevin Behr: Integrating Controls and Process Improvement
Kevin Behr: Integrating Controls and Process ImprovementKevin Behr: Integrating Controls and Process Improvement
Kevin Behr: Integrating Controls and Process Improvement
 
United2012 Rugged DevOps Rocks
United2012 Rugged DevOps RocksUnited2012 Rugged DevOps Rocks
United2012 Rugged DevOps Rocks
 
2012 05 corp fin 1c
2012 05 corp fin 1c2012 05 corp fin 1c
2012 05 corp fin 1c
 
SecureWorld: Security is Dead, Rugged DevOps 1f
SecureWorld:  Security is Dead, Rugged DevOps 1fSecureWorld:  Security is Dead, Rugged DevOps 1f
SecureWorld: Security is Dead, Rugged DevOps 1f
 
2012 SxSW When IT Says No by Gene Kim
2012 SxSW When IT Says No by Gene Kim2012 SxSW When IT Says No by Gene Kim
2012 SxSW When IT Says No by Gene Kim
 
Security is Dead. Long Live Rugged DevOps: IT at Ludicrous Speed
Security is Dead. Long Live Rugged DevOps: IT at Ludicrous SpeedSecurity is Dead. Long Live Rugged DevOps: IT at Ludicrous Speed
Security is Dead. Long Live Rugged DevOps: IT at Ludicrous Speed
 
2011 09 19 LSPE Dev Ops Cookbook 1a
2011 09 19 LSPE Dev Ops Cookbook 1a2011 09 19 LSPE Dev Ops Cookbook 1a
2011 09 19 LSPE Dev Ops Cookbook 1a
 

Recently uploaded

Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Farhan Tariq
 
Introduction-to-Wazuh-and-its-integration.pptx
Introduction-to-Wazuh-and-its-integration.pptxIntroduction-to-Wazuh-and-its-integration.pptx
Introduction-to-Wazuh-and-its-integration.pptxmprakaash5
 
A Glance At The Java Performance Toolbox
A Glance At The Java Performance ToolboxA Glance At The Java Performance Toolbox
A Glance At The Java Performance ToolboxAna-Maria Mihalceanu
 
Decarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a realityDecarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a realityIES VE
 
Automation Ops Series: Session 3 - Solutions management
Automation Ops Series: Session 3 - Solutions managementAutomation Ops Series: Session 3 - Solutions management
Automation Ops Series: Session 3 - Solutions managementDianaGray10
 
Infrared simulation and processing on Nvidia platforms
Infrared simulation and processing on Nvidia platformsInfrared simulation and processing on Nvidia platforms
Infrared simulation and processing on Nvidia platformsYoss Cohen
 
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentEmixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentPim van der Noll
 
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...Alkin Tezuysal
 
QMMS Lesson 2 - Using MS Excel Formula.pdf
QMMS Lesson 2 - Using MS Excel Formula.pdfQMMS Lesson 2 - Using MS Excel Formula.pdf
QMMS Lesson 2 - Using MS Excel Formula.pdfROWELL MARQUINA
 
Generative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdfGenerative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdfIngrid Airi González
 
2024 April Patch Tuesday
2024 April Patch Tuesday2024 April Patch Tuesday
2024 April Patch TuesdayIvanti
 
Green paths: Learning from publishers’ sustainability journeys - Tech Forum 2024
Green paths: Learning from publishers’ sustainability journeys - Tech Forum 2024Green paths: Learning from publishers’ sustainability journeys - Tech Forum 2024
Green paths: Learning from publishers’ sustainability journeys - Tech Forum 2024BookNet Canada
 
Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Hiroshi SHIBATA
 
Transport in Open Pits______SM_MI10415MI
Transport in Open Pits______SM_MI10415MITransport in Open Pits______SM_MI10415MI
Transport in Open Pits______SM_MI10415MIRomil Mishra
 
All These Sophisticated Attacks, Can We Really Detect Them - PDF
All These Sophisticated Attacks, Can We Really Detect Them - PDFAll These Sophisticated Attacks, Can We Really Detect Them - PDF
All These Sophisticated Attacks, Can We Really Detect Them - PDFMichael Gough
 
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24Mark Goldstein
 
Accelerating Enterprise Software Engineering with Platformless
Accelerating Enterprise Software Engineering with PlatformlessAccelerating Enterprise Software Engineering with Platformless
Accelerating Enterprise Software Engineering with PlatformlessWSO2
 
Potential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsPotential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsRavi Sanghani
 
Which standard is best for your content?
Which standard is best for your content?Which standard is best for your content?
Which standard is best for your content?Rustici Software
 
Deliver Latency Free Customer Experience
Deliver Latency Free Customer ExperienceDeliver Latency Free Customer Experience
Deliver Latency Free Customer ExperienceOpsTree solutions
 

Recently uploaded (20)

Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...
 
Introduction-to-Wazuh-and-its-integration.pptx
Introduction-to-Wazuh-and-its-integration.pptxIntroduction-to-Wazuh-and-its-integration.pptx
Introduction-to-Wazuh-and-its-integration.pptx
 
A Glance At The Java Performance Toolbox
A Glance At The Java Performance ToolboxA Glance At The Java Performance Toolbox
A Glance At The Java Performance Toolbox
 
Decarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a realityDecarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a reality
 
Automation Ops Series: Session 3 - Solutions management
Automation Ops Series: Session 3 - Solutions managementAutomation Ops Series: Session 3 - Solutions management
Automation Ops Series: Session 3 - Solutions management
 
Infrared simulation and processing on Nvidia platforms
Infrared simulation and processing on Nvidia platformsInfrared simulation and processing on Nvidia platforms
Infrared simulation and processing on Nvidia platforms
 
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentEmixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
 
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
 
QMMS Lesson 2 - Using MS Excel Formula.pdf
QMMS Lesson 2 - Using MS Excel Formula.pdfQMMS Lesson 2 - Using MS Excel Formula.pdf
QMMS Lesson 2 - Using MS Excel Formula.pdf
 
Generative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdfGenerative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdf
 
2024 April Patch Tuesday
2024 April Patch Tuesday2024 April Patch Tuesday
2024 April Patch Tuesday
 
Green paths: Learning from publishers’ sustainability journeys - Tech Forum 2024
Green paths: Learning from publishers’ sustainability journeys - Tech Forum 2024Green paths: Learning from publishers’ sustainability journeys - Tech Forum 2024
Green paths: Learning from publishers’ sustainability journeys - Tech Forum 2024
 
Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024
 
Transport in Open Pits______SM_MI10415MI
Transport in Open Pits______SM_MI10415MITransport in Open Pits______SM_MI10415MI
Transport in Open Pits______SM_MI10415MI
 
All These Sophisticated Attacks, Can We Really Detect Them - PDF
All These Sophisticated Attacks, Can We Really Detect Them - PDFAll These Sophisticated Attacks, Can We Really Detect Them - PDF
All These Sophisticated Attacks, Can We Really Detect Them - PDF
 
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
 
Accelerating Enterprise Software Engineering with Platformless
Accelerating Enterprise Software Engineering with PlatformlessAccelerating Enterprise Software Engineering with Platformless
Accelerating Enterprise Software Engineering with Platformless
 
Potential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsPotential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and Insights
 
Which standard is best for your content?
Which standard is best for your content?Which standard is best for your content?
Which standard is best for your content?
 
Deliver Latency Free Customer Experience
Deliver Latency Free Customer ExperienceDeliver Latency Free Customer Experience
Deliver Latency Free Customer Experience
 

GitHub Universe: 2019: Exemplars, Laggards, and Hoarders A Data-driven Look at Open Source Software Supply Chains

  • 1. Gene Kim Author, Researcher “The Unicorn Project,” Co-Author: “The Phoenix Project,” “DevOps Handbook,” “Accelerate” Exemplars, Laggards, and Hoarders A Data-driven Look at Open Source Software Supply Chains Dr. Stephen Magill CEO, MuseDev Principal Scientist, Galois, Inc. @stephenmagill
  • 2. @stephenmagill @RealGeneKim • Open Source Software is everywhere • Nat Friedman, CEO, GitHub: “99% of new software projects include open source” • How do these teams you depend on manage updates / security / testing? • “You are inviting thousands of developers into your code” when you use open source dependencies • Will they help or hurt you? (Erica Brescia, COO, GitHub) • Which practices correspond to good component security outcomes and therefore good security for your software? Problem Statement
  • 3. State of DevOps Research • State of DevOps Report (2013-2019) • Dr. Nicole Forsgren, Jez Humble, Gene Kim • Cross population study spanning over 35K respondents • Identified “IT performance” and the factors that predicts: • Deployment Frequency • Deployment Lead Time • Deploy Success Rate • Mean Time to Restore Source: Google/DORA: 2018 State Of DevOps Report: https://cloudplatformonline.com/2018-state-of-devops.html
  • 4. @stephenmagill @RealGeneKim • Our goal: Study what structures and practices are correlated with exemplary outcomes (fast time to update, fast time to remediate security vulnerabilities) • Will we find the same trends we do in the enterprise, with faster delivery correlating with good “business” outcomes? Goals
  • 5. Session ID: The Opportunity: Study the Java Maven Ecosystem!
  • 6. Session ID: The Opportunity: Study the Java Maven Ecosystem! Clojure Haskell
  • 7. Dr. Stephen Magill (Galois) Gene Kim (IT Revolution) Bruce Mayhew (Sonatype) Gazi Mahmud (Sonatype) Thanks also to: Kevin Witten, Derek Weeks, and Matt Howard
  • 8. @stephenmagill @RealGeneKim • Hypothesis 1: Projects that release frequently have better outcomes. • Hypothesis 2: Projects that update dependencies more frequently are generally more secure. • Hypothesis 3: Projects with fewer dependencies will stay more up to date. • Hypothesis 4: More popular projects will be better about staying up to date. Hypotheses *
  • 9. 310,888 Java components 4.2 Million Artifacts (JARs) 6,952 GitHub Repos 27,704 8.9% with known vulnerabilities @RealGeneKim@stephenmagill MavenCentral
  • 17. @RealGeneKim@stephenmagill All of the above for dependencies
  • 19. Attributes Measure Popularity Avg. daily Central Repository downloads Release Frequency Avg. period between releases Development Activity Avg. commits per month Size of Team Avg. unique monthly contributors Presence of CI Presence of popular cloud CI systems Foundation Support Associated with an open source foundation Security Based on reported vulnerabilities Update Lag Based on dependency updates @RealGeneKim@stephenmagill
  • 20. @stephenmagill @RealGeneKim • Popularity • Main component: Average number of downloads per day from The Central Repository. • Also used the Libraries.io dataset: Number of GitHub stars, forks, and pull requests. • Sonatype Nexus IQ Server: Popularity score based on how frequently components are seen by the Nexus IQ repository scanning service • Commit activity • SCM Commits per Month – average number of commits per month (Perceval) • Developer Team Size –average number of unique developers committing each month (Perceval) • (8 core VM scanning repositories for three days: Clojure wrapper around Perceval and jq) • Presence of Continuous Integration (CI): as measured by the detection of any CI-related configuration files in the source code repository (e.g., Travis, Jenkins, CircleCI, etc.). • Clojure program retrieving HTML from GitHub repo, regular expressions to detect CI Data Gathered: Repositories* We used the CHAOSS Perceval utility to gather GitHub commit data, we gathered the number of commits per month for twelve months, as well as the number of unique developers committing during each month. Thank you to CHAOSS and Libaries.io for your amazing tools and data!
  • 21. @stephenmagill @RealGeneKim • Support Type: support for the component comes from an open source foundation, a commercial organization, or is not officially supported by any organization (e.g., a personal project). • Number of Dependencies: the maximum count of dependencies for any given component across all versions in the study period, as measured by the dependencies in the Maven pom.xml file. • Stale Dependencies (fewer is better): the average percentage of out-of-date component dependencies (i.e., a newer version has been released) present when the component has a new release. • Release Period (shorter is better): average time in days each component version spends as the “current” release. A shorter average release period equates to more frequent releases. Data Gathered: Project-Level *
  • 22. ITPERF (2013-2019) Software Supply Chain (2019) Deployment Frequency • Commits / month * • Releases / month • Commits / dev / month * Deployment Lead Time • PR lead time • Issue resolution time Deploy Success Rate • API Breakage rate, Build Breakage Rate, PR Breakage Rate Mean Time to Restore • MTTR (mean time to remediate security vulnerabilities) * • MTTU (mean time to update available components) * • Age of stale / vulnerable dependencies * Org Perf • Stars / Popularity / Download count * Thoughts On ITPERF <-> SSC Metrics * Explored in this year’s research: 2019 State of the Software Supply Chain
  • 23. @RealGeneKim Hypothesis 1 Projects that release frequently have better outcomes. (State of DevOps Report shows decisively that shorter deployment lead times and higher release frequency improves outcomes) @stephenmagill *
  • 24. @RealGeneKim Hypothesis 1 Projects that release frequently have better outcomes. (VALIDATED) @stephenmagill *
  • 25. Projects that release most frequently (top 20%): are 5x more popular (Maven Central downloads, GitHub stars and forks) have 79% more developers have 12% greater foundation support rates. @RealGeneKim@stephenmagill
  • 26. Attributes Measure Popularity Avg. daily Central Repository downloads Release Frequency Avg. period between releases Development Speed Avg. commits per month Size of Team Avg. unique monthly contributors Presence of CI Presence of popular cloud CI systems Foundation Support Associated with an open source foundation Security Based on reported vulnerabilities Update Speed Based on dependency updates @RealGeneKim@stephenmagill Dependency-Level Metrics
  • 27. Security: Time to Remediate (TTR) @RealGeneKim@stephenmagill
  • 28. Security: Time to Remediate (TTR) @RealGeneKim@stephenmagill
  • 29. Security: Time to Remediate (TTR) @RealGeneKim@stephenmagill
  • 30. Security: Time to Remediate (TTR) @RealGeneKim@stephenmagill
  • 31. Security: Time to Remediate (TTR) @RealGeneKim@stephenmagill
  • 32. Security: Time to Remediate (TTR) @RealGeneKim B Vulnerable Time @stephenmagill
  • 33. Security: Time to Remediate (TTR) @RealGeneKim C Vulnerable Time @stephenmagill
  • 34. Security: Time to Remediate (TTR) @RealGeneKim C Remediation Time @stephenmagill
  • 35. Security: Time to Update (TTU) @RealGeneKim C Update Time (for B) @stephenmagill
  • 36. Security: Time to Update (TTU) @RealGeneKim C Update Time (for A) @stephenmagill
  • 38. @RealGeneKim The Key Dependency Metrics (per-update) Time to Remediate Time to Update Stale Dependencies @stephenmagill
  • 39. @RealGeneKim The Key Dependency Metrics (per-project) Median Time to Remediate Median Time to Update Median Stale Dependencies @stephenmagill
  • 40. Time to Remediate Security Vulnerabilities*
  • 41. Time to Remediate Security Vulnerabilities Do these update quickly in general?
  • 42. Time to Remediate vs. Time to Update Dependencies (TTU)
  • 43. Time to Remediate (TTR) vs. Time to Update (TTU) * @RealGeneKim Pearson correlation 0.6 @stephenmagill
  • 44. Most projects stay secure by staying up to date. 55% have MTTR and MTTU within 20% of each other. Only 15% of projects with worse than average MTTU manage to maintain better than average MTTR. @RealGeneKim@stephenmagill
  • 45. Time to Remediate (TTR) vs. Time to Update (TTU) @RealGeneKim Pearson correlation 0.6 @stephenmagill
  • 46. Most projects stay secure by staying up to date. 55% have MTTR and MTTU within 20% of each other. Only 15% of projects with worse than average MTTU manage to maintain better than average MTTR. @RealGeneKim@stephenmagill
  • 47. Hypothesis 2 Projects that update dependencies more frequently are generally more secure. @RealGeneKim@stephenmagill
  • 48. Hypothesis 2 Projects that update dependencies more frequently are generally more secure. (VALIDATED) @RealGeneKim@stephenmagill *PrimeFaces CVE-2017-1000486: published 1/3/2018; vuln unreported as CVE; was fixed in 2/2016; cryptominers started using it (Source: Jeremy Long: @ctxt)
  • 49. Hypothesis 3 Projects with fewer dependencies will stay more up to date. @RealGeneKim@stephenmagill
  • 50. Hypothesis 3 Projects with fewer dependencies will stay more up to date. (REFUTED) Components with more dependencies actually have better MTTU. @RealGeneKim@stephenmagill
  • 51. More dependencies correlate with larger development teams. @RealGeneKim Larger development teams have 50% faster MTTU and release 2.6x more frequently. @stephenmagill
  • 52. More dependencies correlate with larger development teams. @RealGeneKim Larger development teams have 50% faster MTTU and release 2.6x more frequently. @stephenmagill
  • 53. @RealGeneKim Hypothesis 4 More popular projects will be better about staying up to date. @stephenmagill
  • 54. Not all popular projects are exemplary and release fast (10-10K downloads per day)
  • 55. @RealGeneKim Hypothesis 4 More popular projects will be better about staying up to date. (REFUTED) There are plenty of popular components with poor MTTU. Popularity does not correlate with MTTU. The most popular projects are not statistically different from others with respect to MTTU. @stephenmagill
  • 56. @RealGeneKim Number of stars or number of forks IS NOT AN EFFECTIVE HEURISTIC for selecting which components to use (if security is important to you) @stephenmagill
  • 57. 5 Behavioral Clusters for OSS “Suppliers” @RealGeneKim Small Exemplar (606) Large Exemplar (595) Small development teams (1.6 devs), exemplary MTTU. Large development teams (8.9 devs), exemplary MTTU, very likely to be foundation supported, 11x more popular. @stephenmagill Laggards (521) Features First (280) Cautious (429) Poor MTTU, high stale dependency count, more likely to be commercially supported. Frequent releases, but poor TTU. Still reasonably popular. Good TTU, but seldom completely up to date.
  • 58. @stephenmagill @RealGeneKim • We conducted survey with 658 respondents who completed it — three clusters emerged, which we called “high, medium, and low update pain” clusters • Comparison between “high pain” vs. “low pain” clusters: • Updating dependencies is painful: 3.2x less likely to strongly agree • Updating vulnerable components is painful: 2.6x less likely • We schedule updating dependencies as part of our daily work: 10x more likely • We strive to use the latest version (or latest-N) of all our dependencies: 6.2x more likely • We use some process to add a new dependency (e.g., evaluate, approve, standardize, etc.): 11x more likely • We have a process to proactively remove problematic or unused dependencies: 9.3x more likely • We have automated tools to track, manage, and/or ensure policy compliance of our dependencies: 12x more likely Exemplars: Survey Data (N=658) *
  • 59. Dr. Stephen Magill (Galois) Gene Kim (IT Revolution) Bruce Mayhew (Sonatype) Gazi Mahmud (Sonatype) Thanks also to: Kevin Witten, Derek Weeks, and Matt Howard stephen@muse.dev
  • 60. @stephenmagill @RealGeneKim • Study further breaking changes • Look at transitive dependencies • Identify leading indicators, use techniques to assert causation Year 2 Goals
  • 62. @stephenmagill @RealGeneKim • Ways to detect breaking changes • Outcomes resulting from Dependabot pull requests? • For which components are updates quickly and painlessly applied • For which components are updates never applied (i.e., because they break everything) • Which components have a disciplined and immutable API that allows for easier upgrades? • E.g., Clojure programming language and standard library have had virtually no breaking changes in 12 years • E.g., React-native: “4 months after not touching it, it no longer builds if you update all the dependencies” • Get data on pull request lead time and issue resolution time • (DONE? Thank you code.gov!) • Authoritative list of foundation-supported projects? Help We’re Looking For
  • 63. Quick Takeaways Integrate updating dependencies into your daily work! Contribute dependency updates to components you use! Don’t make decisions based solely on popularity! Tell us what hypotheses you would like to see investigated! @RealGeneKim@stephenmagill

Editor's Notes

  1. Stephen: I’ve been doing academic research in software analysis, security, and programming languages for more than 15 years, first as part of my Ph.D. work at Carnegie Mellon and then at other universities and industry research labs. Over the last few years I’ve been getting more and more interested in the “practice” of software: open source development practices, how Enterprises approach software, and how to best contribute to these communities by improving tools and practices. Gene: I’ve been studying high performers since 1999; started when I was CTO and founder of a company called Tripwire; One of the most fun things I’ve ever worked on was the State of DevOps Report, along with Dr. Nicole Forsgren and Jez Humble, which resulted in the Accelerate book My area of passion these days are studying how DevOps principles and patterns are being adopted in large complex organizations; I run the DevOps Enterprise Summit; and I have a book coming out in 1.5 weeks called the Unicorn Project.
  2. It was with gratitude and extreme enthusiasm that I jumped into this project, because as someone who loves the Clojure programming language, we benefit everyday from the Maven ecosystem.
  3. It was this extensive use of open source in high performing DevOps teams that led to an intersection of research that I had been leading for five years in the SSC report, And research that Dr. Steven Magill, Gene Kim, Bruce Mayhew, Gazi Mahmud, and I embarked upon a year ago You see, Gene Kim shared the Three Ways of DevOps inside The Phoenix Project, with the first way being “Emphasize performance of the entire system and never pass a defect downstream.”
  4. i
  5. i
  6. I want to talk a little bit about the dataset so first of all we focused on Java projects publish to Maven Central there about 260,000 of those and then we apply the number of filters to get down to a course out of components that we that we felt we could analyze well  
  7. I want to talk a little bit about the dataset so first of all we focused on Java projects publish to Maven Central there about 260,000 of those and then we apply the number of filters to get down to a course out of components that we that we felt we could analyze well   so those filters were first of all we looked in the last 5 years I can develop development Trends culture tools and Technology of changed over time we wanted to find things that hold today and in the last five years we also throughout components that we didn't have enough data about to really draw conclusions so for example we wanted to measure release frequency the average time between new releases the component is only put one release out there there's never been a follow-up release we can't even met her by chance they don't use any open source libraries and they're not used by any other projects so there's sort of isolated off of it all by themselves so when we apply all of these we get down to a course out of 36000 components and our research and for those components  
  8. we looked at a number of different attributes things like popularity the size of the development team development Speed release speed and so forth from any of these we have data across those entire 36,000   so for example popularity we Define as the average daily Maven Central downloads and we have that data for every component of other things like size of development team we get that from GitHub data associated with the project so we only have that for the projects that are on guitar and they're about ten thousand of those   I answer most of these attributes are self-explanatory there's a couple of the bottom though that warrant a little bit more discussion so security and update speed a little bit more complicated because of the complexity of Open Source Supply chains
  9. CI didn’t matter: a pretty startling result, but in hindsight, makes some sense… rather like how version control is not predictive of performance — it’s necessary, but far from sufficient.
  10. so now we can look at a couple of these attributes and say you know does this faster is better relationship hold an open source and look at release frequency vs. popularity   and this is one of the hypotheses that we enter this project with let's see if we can find data to validate this hypothesis that projects that release frequently have better outcomes and in fact we find support for this so if you look at the top 20% by release frequency that group is 5 times more popular than the rest of the population attracts on average 79% more developers to contribute to the project and has 12% of Greater rates of foundation support should view these as a descriptive statistics about the population, where we could see the correlation and not the causation
  11. Gene: Surprise: do larger teams lead to more frequent releases, or does popularity and active projects attract more developers
  12. we looked at a number of different attributes things like popularity the size of the development team development Speed release speed and so forth from any of these we have data across those entire 36,000   so for example popularity we Define as the average daily Maven Central downloads and we have that data for every component of other things like size of development team we get that from GitHub data associated with the project so we only have that for the projects that are on guitar and they're about ten thousand of those   I answer most of these attributes are self-explanatory there's a couple of the bottom though that warrant a little bit more discussion so security and update speed a little bit more complicated because of the complexity of Open Source Supply chains
  13. What were we looking to measure across these 36,000 projects? Here is a visualization of three components A, B, &C and the dependency relationship between them time is marching along from left to right ====== how those are Define so here we have an example of so the weight of you this chart is so version 2.2 of B comes out then version 2.2 of a wave inversion 2.3 l A and so on   left or right the lines show dependency relationships.   so for example version 2.2 of C depends on version 2.2 of B and then we also have a vulnerability disclosure represented here   so there's a point in time at which there's a vulnerability reported against component B there's a and then B releases version 2.3 to mitigate that there's a. Of time where B is vulnerable and because C includes be as a dependency there's a. Of time where C is vulnerable and so we can measure each of these times   but if you think about it from Cedar Point of View the important time frame to think about is how long it takes him to respond to the release of the past version of B is really the first opportunity C has to know mitigate this Downstream security risks that use imported via his software supply chain security relevant metric that we measure and we call that time to remediate TTR   we also just measure update time in general and so that's a new release of these or see is you know take some time to incorporate that new releases be that's the update time for B is not a time for A as well even though there's no security vulnerability against that   every new release or some Associated time to update   we looked at is this notion of stale dependency so we often see a project release and maybe some of its dependencies wheel will be updated to the latest version but others others will be behind   you see that happening here with C where a version 2.3 has been released at the point where C version 2.2 comes out but C is not using that they're not using latest version of its time to update and steel dependency switch or just general update hygiene mix and
  14. What were we looking to measure across these 36,000 projects? Here is a visualization of three components A, B, &C and the dependency relationship between them time is marching along from left to right ====== how those are Define so here we have an example of so the weight of you this chart is so version 2.2 of B comes out then version 2.2 of a wave inversion 2.3 l A and so on   left or right the lines show dependency relationships.   so for example version 2.2 of C depends on version 2.2 of B and then we also have a vulnerability disclosure represented here   so there's a point in time at which there's a vulnerability reported against component B there's a and then B releases version 2.3 to mitigate that there's a. Of time where B is vulnerable and because C includes be as a dependency there's a. Of time where C is vulnerable and so we can measure each of these times   but if you think about it from Cedar Point of View the important time frame to think about is how long it takes him to respond to the release of the past version of B is really the first opportunity C has to know mitigate this Downstream security risks that use imported via his software supply chain security relevant metric that we measure and we call that time to remediate TTR   we also just measure update time in general and so that's a new release of these or see is you know take some time to incorporate that new releases be that's the update time for B is not a time for A as well even though there's no security vulnerability against that   every new release or some Associated time to update   we looked at is this notion of stale dependency so we often see a project release and maybe some of its dependencies wheel will be updated to the latest version but others others will be behind   you see that happening here with C where a version 2.3 has been released at the point where C version 2.2 comes out but C is not using that they're not using latest version of its time to update and steel dependency switch or just general update hygiene mix and
  15. What were we looking to measure across these 36,000 projects? Here is a visualization of three components A, B, &C and the dependency relationship between them time is marching along from left to right ====== how those are Define so here we have an example of so the weight of you this chart is so version 2.2 of B comes out then version 2.2 of a wave inversion 2.3 l A and so on   left or right the lines show dependency relationships.   so for example version 2.2 of C depends on version 2.2 of B and then we also have a vulnerability disclosure represented here   so there's a point in time at which there's a vulnerability reported against component B there's a and then B releases version 2.3 to mitigate that there's a. Of time where B is vulnerable and because C includes be as a dependency there's a. Of time where C is vulnerable and so we can measure each of these times   but if you think about it from Cedar Point of View the important time frame to think about is how long it takes him to respond to the release of the past version of B is really the first opportunity C has to know mitigate this Downstream security risks that use imported via his software supply chain security relevant metric that we measure and we call that time to remediate TTR   we also just measure update time in general and so that's a new release of these or see is you know take some time to incorporate that new releases be that's the update time for B is not a time for A as well even though there's no security vulnerability against that   every new release or some Associated time to update   we looked at is this notion of stale dependency so we often see a project release and maybe some of its dependencies wheel will be updated to the latest version but others others will be behind   you see that happening here with C where a version 2.3 has been released at the point where C version 2.2 comes out but C is not using that they're not using latest version of its time to update and steel dependency switch or just general update hygiene mix and
  16. What were we looking to measure across these 36,000 projects? Here is a visualization of three components A, B, &C and the dependency relationship between them time is marching along from left to right ====== how those are Define so here we have an example of so the weight of you this chart is so version 2.2 of B comes out then version 2.2 of a wave inversion 2.3 l A and so on   left or right the lines show dependency relationships.   so for example version 2.2 of C depends on version 2.2 of B and then we also have a vulnerability disclosure represented here   so there's a point in time at which there's a vulnerability reported against component B there's a and then B releases version 2.3 to mitigate that there's a. Of time where B is vulnerable and because C includes be as a dependency there's a. Of time where C is vulnerable and so we can measure each of these times   but if you think about it from Cedar Point of View the important time frame to think about is how long it takes him to respond to the release of the past version of B is really the first opportunity C has to know mitigate this Downstream security risks that use imported via his software supply chain security relevant metric that we measure and we call that time to remediate TTR   we also just measure update time in general and so that's a new release of these or see is you know take some time to incorporate that new releases be that's the update time for B is not a time for A as well even though there's no security vulnerability against that   every new release or some Associated time to update   we looked at is this notion of stale dependency so we often see a project release and maybe some of its dependencies wheel will be updated to the latest version but others others will be behind   you see that happening here with C where a version 2.3 has been released at the point where C version 2.2 comes out but C is not using that they're not using latest version of its time to update and steel dependency switch or just general update hygiene mix and
  17. What were we looking to measure across these 36,000 projects? Here is a visualization of three components A, B, &C and the dependency relationship between them time is marching along from left to right ====== how those are Define so here we have an example of so the weight of you this chart is so version 2.2 of B comes out then version 2.2 of a wave inversion 2.3 l A and so on   left or right the lines show dependency relationships.   so for example version 2.2 of C depends on version 2.2 of B and then we also have a vulnerability disclosure represented here   so there's a point in time at which there's a vulnerability reported against component B there's a and then B releases version 2.3 to mitigate that there's a. Of time where B is vulnerable and because C includes be as a dependency there's a. Of time where C is vulnerable and so we can measure each of these times   but if you think about it from Cedar Point of View the important time frame to think about is how long it takes him to respond to the release of the past version of B is really the first opportunity C has to know mitigate this Downstream security risks that use imported via his software supply chain security relevant metric that we measure and we call that time to remediate TTR   we also just measure update time in general and so that's a new release of these or see is you know take some time to incorporate that new releases be that's the update time for B is not a time for A as well even though there's no security vulnerability against that   every new release or some Associated time to update   we looked at is this notion of stale dependency so we often see a project release and maybe some of its dependencies wheel will be updated to the latest version but others others will be behind   you see that happening here with C where a version 2.3 has been released at the point where C version 2.2 comes out but C is not using that they're not using latest version of its time to update and steel dependency switch or just general update hygiene mix and
  18. In this visualization, we wanted to understand and measure A vuln is found in B 2.2 B 2.3 updates it
  19. We also wanted to understand how fast C would update all dependencies, including A
  20. I want to focus on the security relevant part for just a bit because of what Derek was saying about the the prevalence of vulnerabilities in the supply chain and how that trickles down into two users of those open source projects   so if we look at the time it takes these projects to apply security relevant patches the median time is about six months which is already not great and it gets even worse if you look sort of it the right of the speaker at the 95th percentile we see that 5% of projects take three and a half years or more   Ago by a security relevant patch and these are not projects that just like never like they did eventually apply it just took them three and a half years to get there  
  21. Do these projects stay up to date in general? (The projects with strong MTTR) They are many more updates to perform in general, than vulnerabilities to correct.
  22. Most projects stay secure by staying up-to-date. 55% have MTTR and MTTU within 20% of each other. Only 15% maintain better than average MTTR with worse than average MTTU.
  23. actually we do see a correlation between update behavior in general and update behavior for security relevant updates of how quickly projects applying for developing updates versus how quickly we apply security Roman updates and we see we see a reasonable correlation here there's a point six correlation coefficient between two and you certainly see projects that fall on one side or the other in a bit better about security or for whatever reason that end up performing better on on security   but if you if you dig into the date a little bit more we see that 55% of project have an empty tea are in an empty Tu that are within 20% of each other so they're sort of close to this line and and if you look for projects that manage to stay up-to-date from a security perspective while not updating at dependencies in general so they they do very good and remediating vulnerabilities but don't keep the rest of their dependencies up to date   so small population only 15% of projects end up having that exhibiting odd behavior so stay secure by staying up to date as a common behavior is just stay up-to-date in general and as a consequence so that was a second hypothesis that we entered into this research with and
  24. Only 15% maintain better than average MTTR with worse than average MTTU.
  25. actually we do see a correlation between update behavior in general and update behavior for security relevant updates of how quickly projects applying for developing updates versus how quickly we apply security Roman updates and we see we see a reasonable correlation here there's a point six correlation coefficient between two and you certainly see projects that fall on one side or the other in a bit better about security or for whatever reason that end up performing better on on security   but if you if you dig into the date a little bit more we see that 55% of project have an empty tea are in an empty Tu that are within 20% of each other so they're sort of close to this line and and if you look for projects that manage to stay up-to-date from a security perspective while not updating at dependencies in general so they they do very good and remediating vulnerabilities but don't keep the rest of their dependencies up to date   so small population only 15% of projects end up having that exhibiting odd behavior so stay secure by staying up to date as a common behavior is just stay up-to-date in general and as a consequence so that was a second hypothesis that we entered into this research with and
  26. Only 15% maintain better than average MTTR with worse than average MTTU.
  27. we found some data to validate that another hypothesis that we came in with was that projects with fewer dependencies will stay up-to-date better and intuitively this seems to make sense   if you only have two or three dependencies it should be pretty easy to keep them up-to-date with the latest version certainly easier than if you have 10 or 15   in fact we actually found the opposite so components with more dependents he's actually had better update hygiene they would save more on top of their dependency version updates to statistically significant levels so it actually the reason this occurs is because components with more dependents he's also tends I have larger development teams   if you look at the large development teams just having a larger development team associated with a faster in TTU rate as a faster release frequency so you can see here is a plot of number of dependencies is increasing as you go to the right size of development team is increasing as you go up is a smooth plot so you can see the trend line better but there's a correlation between the size and dependency number   and again we don't know which direction it goes you need more developers to manage all these dependencies or maybe every developer brings his own favorite to Hennessy and you end up with like for unit testing libraries
  28. Gene: so curious and surprising: more developers cause more dependencies? Or the number of dependencies create so much work that you need more deveopers? We don’t know…
  29. we really wanted to look into this one because so many people use popularity as a proxy for security everyone else is using it so it must be a good project they must have security must be useful we have investigated hypothesis for was that more popular projects will be better about staying up State and we really wanted to look into this one because so many people use popularity as a proxy for security right everyone else is using it so it must be a good project they must have security must be useful we have investigated hypothesis for was that more popular projects will be better about staying up State and we really wanted to look into this one because so many people use popularity as a proxy for security right everyone else is using it so it must be a good project they must have security must be useful   Data for this so first of all there's plenty of popular components with 4 update hiking but there's always those outliers right but more interestingly we don't see any sort of correlation between these two attributes and even if you look at the most popular projects you say okay I'm just going to look at the top 10% by popularity those are not statistically better with respect to update Behavior than the general population   so I took you take one thing away from this talk don't choose your components just based on popularity
  30. so we have the difference of populations in different colors you can see the exemplars over here at the lab so this is the plot of popularity vs. update   I'm So at the left you see products releasing frequently and they tend to be more popular in the exemplars in particular are more generally more popular and release frequently   and then another thing to note is what I said before about hypothesis for not all popular projects or exemplary by so that you can see through the prevalence the big spread of red dots across their right in and some very popular projects that have very poor update I had
  31. Gene: surprising: when I pick OSS components, I pick number of stars and forks. And apparently, this is not as an effective of a heuristic as I thought. And I think this is problematic.
  32. Gene: surprising: when I pick OSS components, I pick number of stars and forks. And apparently, this is not as an effective of a heuristic as I thought. And I think this is problematic.
  33. so we have the difference of populations in different colors you can see the exemplars over here at the lab so this is the plot of popularity vs. update   I'm So at the left you see products releasing frequently and they tend to be more popular in the exemplars in particular are more generally more popular and release frequently   and then another thing to note is what I said before about hypothesis for not all popular projects or exemplary by so that you can see through the prevalence the big spread of red dots across their right in and some very popular projects that have very poor update I had
  34. Gene: color commentary: it was so interesting to find this “OSS Industrial Complex”: large teams, with around 10 active developers committing constantly, it’s probably part of their day job. Exemplars are not just the domain of these large projects – they are found in both small and large groups. It’s not about the size of the team; it’s about the values and culture of the team. (It’s not about whether you have CI; it’s about why you use it and how you use it)
  35. so we have the difference of populations in different colors you can see the exemplars over here at the lab so this is the plot of popularity vs. update   I'm So at the left you see products releasing frequently and they tend to be more popular in the exemplars in particular are more generally more popular and release frequently   and then another thing to note is what I said before about hypothesis for not all popular projects or exemplary by so that you can see through the prevalence the big spread of red dots across their right in and some very popular projects that have very poor update I had
  36. actually we do see a correlation between update behavior in general and update behavior for security relevant updates of how quickly projects applying for developing updates versus how quickly we apply security Roman updates and we see we see a reasonable correlation here there's a point six correlation coefficient between two and you certainly see projects that fall on one side or the other in a bit better about security or for whatever reason that end up performing better on on security   but if you if you dig into the date a little bit more we see that 55% of project have an empty tea are in an empty Tu that are within 20% of each other so they're sort of close to this line and and if you look for projects that manage to stay up-to-date from a security perspective while not updating at dependencies in general so they they do very good and remediating vulnerabilities but don't keep the rest of their dependencies up to date   so small population only 15% of projects end up having that exhibiting odd behavior so stay secure by staying up to date as a common behavior is just stay up-to-date in general and as a consequence so that was a second hypothesis that we entered into this research with and
  37. It was this extensive use of open source in high performing DevOps teams that led to an intersection of research that I had been leading for five years in the SSC report, And research that Dr. Steven Magill, Gene Kim, Bruce Mayhew, Gazi Mahmud, and I embarked upon a year ago You see, Gene Kim shared the Three Ways of DevOps inside The Phoenix Project, with the first way being “Emphasize performance of the entire system and never pass a defect downstream.”
  38. For organizations who tamed their supply chains, the rewards were impressive: use of known vulnerable component releases was reduced by 55%.
  39. In particular: why people don’t update more often, or what are the conditions that allow updating to happen quickly, painlessly and effecgtively