SlideShare a Scribd company logo
1 of 32
Download to read offline
Bits of Evidence What We Actually Know About Software Development, and Why We Believe It’s True Greg Wilson http://third-bit.com Feb 2010
Once Upon a Time... Seven Years’ War (actually 1754-63) Britain lost 1,512 sailors to enemy action... ...and almost 100,000 to scurvy
Oh, the Irony James Lind (1716-94) 1747: (possibly) the first-ever controlled medical experiment No-one paid attention until a proper Englishman repeated the experiment in 1794... ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
It Took a While to Catch On 1950: Hill & Doll publish a case-control study  comparing smokers with non-smokers 1951: start the British Doctors Study (which runs until 2001)
What They Discovered #1: Smoking causes lung cancer “ ...what happens ‘on average’ is of no help when one is faced with a specific patient...” #2: Many people would rather fail than change
Like Water on Stone 1992: Sackett coins the term “ evidence-based medicine” Randomized double-blind trials are accepted as the gold standard for medical research The Cochrane Collaboration (http://www.cochrane.org/) now archives results from hundreds of medical studies
So Where Are We? “ [Using domain-specific languages] leads to two primary benefits. The first, and simplest, is improved programmer productivity... The second...is...communication with domain experts.” –  Martin Fowler (IEEE Software, July/August 2009)
Say Again? One of the smartest guys in our industry... ...made two substantive claims... ...in an academic journal... ...without a single citation Please note: I’m not disagreeing with his claims —I just want to point out that even the best of us aren’t doing what we expect the makers of acne creams to do.
Um, No “ Debate still continues about how valuable DSLs are in practice. I believe that debate is hampered because not enough people know how to develop DSLs effectively.” I think debate is hampered by  low standards for proof The good news is, things have started to improve
The Times They Are A-Changin’ Growing emphasis on empirical studies in software engineering research since the mid-1990s Papers describing new tools or practices routinely include results from some kind of field study Yes, many are flawed or incomplete, but standards are constantly improving
My Favorite Little Result Aranda & Easterbrook (2005): “Anchoring and Adjustment in Software Estimation” “ How long do you think it will take to make a change to this program?” Control Group: “ I’d like to give an estimate for this project myself, but I admit I have no experience estimating. We’ll wait for your calculations for an estimate.” Group A:  “I admit I have no experience with software projects, but I guess this will take about 2 months to finish. ” Group B:  “...I guess this will take about 20 months... ”
Results The anchor mattered more than experience, how formal the estimation method was, or anything else. Q: Are agile projects similarly afflicted, just on a shorter and more rapid cycle? Group A (lowball) 5.1 months Control Group 7.8 months Group B (highball) 15.4 months
Most Frequently Misquoted Sackman, Erikson, and Grant (1968): “Exploratory experimental studies comparing online and offline programming performance.” Or 10, or 40, or 100, or whatever other large number pops into the head of someone who can’t be bothered to look up the reference... The best programmers are up to 28 times more productive than the worst.
Let’s Pick That Apart ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
So What  Do  We Know? ,[object Object],[object Object],[object Object],I’m not going to tell you Instead, I’d like you to look at the work of Lutz Prechelt Productivity and reliability depend on the length of the program's text, independent of language level.
A Classic Result... Boehm et al (1975): “Some Experience with Automated Aids to the Design of Large-Scale Reliable Software.” ...and many, many more since ,[object Object],[object Object],time number  /  cost
...Which Explains a Lot Pessimists: “If we tackle the hump in the error injection curve, fewer bugs will get to the expensive part of the fixing curve.” Optimists: “If we do lots of short iterations, the total cost of fixing bugs will go down.”
The Real Reason I Care A: I've always believed that there are just fundamental differences between the sexes... B: What data are you basing that opinion on? A: It's more of an unrefuted hypothesis based on personal observation. I have read a few studies on the topic and I found them unconvincing... B: Which studies were those? A: [no reply]
What Real Scientists Do ,[object Object],[object Object],[object Object],[object Object],Ceci & Williams (eds):  Why Aren’t More Women in Science? Top Researchers Debate the Evidence Informed  debate on nature vs. nurture
Greatest Hits ,[object Object],[object Object],[object Object],FIXME: add gratuitous images to liven up these slides.
Greatest Hits (cont.) ,[object Object],[object Object],Gratuitous image. Shouldn’t our development practices  be built around these facts?
More Than Numbers ,[object Object],[object Object],More gratuitous images.
Another Personal Favorite Conway’s Law: A system reflects the organizational structure that built it. Meant as a joke Turns out to be true (Herbsleb et al 1999)
But Wait, There’s More! Nagappan et al (2007) & Bird et al (2009): Physical distance doesn’t affect post-release fault rates Distance in the organizational chart does No, really — shouldn’t our development practices  be built around these facts?
Two Steps Forward... ,[object Object],[object Object],“ Progress” sometimes means saying, “Oops.” El Emam et al (2001): “The Confounding Effect of Class Size on the Validity of Object-Oriented Metrics” Can code metrics predict post-release fault rates? We thought so, but then...
Folk Medicine for Software Systematizing and synthesizing colloquial practice has been very productive in other disciplines…
How Do We Get There? 2007 2008 – 2009
The Book Without a Name Wanted to call the next one  Beautiful Evidence , but Edward Tufte got there first “ What we know and why we think it’s true” (By the way, his book is really good) Knowledge transfer A better textbook Change the debate
A Lot Of Editing In My Future Jorge Aranda Tom Ball Victor Basili Andrew Begel Christian Bird Barry Boehm Marcelo Cataldo Steven Clarke Jason Cohen Rob DeLine Khaled El Emam Hakan Erdogmus Michael Godfrey Mark Guzdial Jo Hannay Ahmed Hassan Israel Herraiz Kim Herzig Barbara Kitchenham Andrew Ko Lucas Layman Steve McConnell Audris Mockus Gail Murphy Nachi Nagappan Tom Ostrand Dewayne Perry Marian Petre Lutz Prechelt Rahul Premraj Dieter Rombach Forrest Shull Beth Simon Janice Singer Diomidis Spinellis Neil Thomas Walter Tichy Burak Turhan Gina Venolia Elaine Weyuker Laurie Williams Andreas Zeller Tom Zimmermann
The Hopeful Result
The  Real  Reason It Matters
Thank you, and good luck

More Related Content

What's hot

Testing Strategy in Micro Frontend architecture
Testing Strategy in Micro Frontend architectureTesting Strategy in Micro Frontend architecture
Testing Strategy in Micro Frontend architecturevodQA
 
Efficient API delivery with APIOps
Efficient API delivery with APIOpsEfficient API delivery with APIOps
Efficient API delivery with APIOpsSven Bernhardt
 
Developing a Testing Strategy for DevOps Success
Developing a Testing Strategy for DevOps SuccessDeveloping a Testing Strategy for DevOps Success
Developing a Testing Strategy for DevOps SuccessDevOps.com
 
Can you trust your APIs - White Paper on API Testing
Can you trust your APIs - White Paper on API TestingCan you trust your APIs - White Paper on API Testing
Can you trust your APIs - White Paper on API TestingSwetha Sridharan
 
Z api overview
Z api overviewZ api overview
Z api overviewbamadhu
 
Microservices Testing Strategies JUnit Cucumber Mockito Pact
Microservices Testing Strategies JUnit Cucumber Mockito PactMicroservices Testing Strategies JUnit Cucumber Mockito Pact
Microservices Testing Strategies JUnit Cucumber Mockito PactAraf Karsh Hamid
 
Speed Clinical Trials from Start to Close: The Path to a Modern CTMS
Speed Clinical Trials from Start to Close: The Path to a Modern CTMSSpeed Clinical Trials from Start to Close: The Path to a Modern CTMS
Speed Clinical Trials from Start to Close: The Path to a Modern CTMSVeeva Systems
 
Best practices for implementing and rolling out a memoQ server in an organiz...
Best practices for implementing and rolling out  a memoQ server in an organiz...Best practices for implementing and rolling out  a memoQ server in an organiz...
Best practices for implementing and rolling out a memoQ server in an organiz...Loctimize GmbH
 
An Introduction to AWS GovCloud (US) | AWS Public Sector Summit 2016
An Introduction to AWS GovCloud (US) | AWS Public Sector Summit 2016An Introduction to AWS GovCloud (US) | AWS Public Sector Summit 2016
An Introduction to AWS GovCloud (US) | AWS Public Sector Summit 2016Amazon Web Services
 
Introduction to Amazon Web Services
Introduction to Amazon Web ServicesIntroduction to Amazon Web Services
Introduction to Amazon Web ServicesRobert Greiner
 
Pooja shift left 1.0
Pooja shift left 1.0Pooja shift left 1.0
Pooja shift left 1.0Xebia India
 
Shift Left Testing: A New Paradigm Shift To Quality
Shift Left Testing: A New Paradigm Shift To QualityShift Left Testing: A New Paradigm Shift To Quality
Shift Left Testing: A New Paradigm Shift To QualityPooja Wandile
 
API Risk: Taking Your API Security to the Next Level
API Risk: Taking Your API Security to the Next LevelAPI Risk: Taking Your API Security to the Next Level
API Risk: Taking Your API Security to the Next LevelCA Technologies
 
Quality Assurance/Testing Overview & Capability Deck
Quality Assurance/Testing Overview & Capability DeckQuality Assurance/Testing Overview & Capability Deck
Quality Assurance/Testing Overview & Capability DeckSowmak Bardhan
 
Low Code Capabilities of Digital Product Design Platforms
Low Code Capabilities of Digital Product Design PlatformsLow Code Capabilities of Digital Product Design Platforms
Low Code Capabilities of Digital Product Design PlatformsJohnMcGuigan10
 
Multiplatform App Architecture
Multiplatform App ArchitectureMultiplatform App Architecture
Multiplatform App ArchitectureEdwin Van Schaick
 
How We Do DevOps at Walmart: OneOps OSS Application Lifecycle Management Plat...
How We Do DevOps at Walmart: OneOps OSS Application Lifecycle Management Plat...How We Do DevOps at Walmart: OneOps OSS Application Lifecycle Management Plat...
How We Do DevOps at Walmart: OneOps OSS Application Lifecycle Management Plat...WalmartLabs
 

What's hot (20)

Testing Strategy in Micro Frontend architecture
Testing Strategy in Micro Frontend architectureTesting Strategy in Micro Frontend architecture
Testing Strategy in Micro Frontend architecture
 
API Governance in the Enterprise
API Governance in the EnterpriseAPI Governance in the Enterprise
API Governance in the Enterprise
 
Testing
TestingTesting
Testing
 
Efficient API delivery with APIOps
Efficient API delivery with APIOpsEfficient API delivery with APIOps
Efficient API delivery with APIOps
 
Developing a Testing Strategy for DevOps Success
Developing a Testing Strategy for DevOps SuccessDeveloping a Testing Strategy for DevOps Success
Developing a Testing Strategy for DevOps Success
 
Can you trust your APIs - White Paper on API Testing
Can you trust your APIs - White Paper on API TestingCan you trust your APIs - White Paper on API Testing
Can you trust your APIs - White Paper on API Testing
 
Z api overview
Z api overviewZ api overview
Z api overview
 
Microservices Testing Strategies JUnit Cucumber Mockito Pact
Microservices Testing Strategies JUnit Cucumber Mockito PactMicroservices Testing Strategies JUnit Cucumber Mockito Pact
Microservices Testing Strategies JUnit Cucumber Mockito Pact
 
Speed Clinical Trials from Start to Close: The Path to a Modern CTMS
Speed Clinical Trials from Start to Close: The Path to a Modern CTMSSpeed Clinical Trials from Start to Close: The Path to a Modern CTMS
Speed Clinical Trials from Start to Close: The Path to a Modern CTMS
 
Best practices for implementing and rolling out a memoQ server in an organiz...
Best practices for implementing and rolling out  a memoQ server in an organiz...Best practices for implementing and rolling out  a memoQ server in an organiz...
Best practices for implementing and rolling out a memoQ server in an organiz...
 
An Introduction to AWS GovCloud (US) | AWS Public Sector Summit 2016
An Introduction to AWS GovCloud (US) | AWS Public Sector Summit 2016An Introduction to AWS GovCloud (US) | AWS Public Sector Summit 2016
An Introduction to AWS GovCloud (US) | AWS Public Sector Summit 2016
 
Iso22000 bilgilendirmeegitimi
Iso22000 bilgilendirmeegitimiIso22000 bilgilendirmeegitimi
Iso22000 bilgilendirmeegitimi
 
Introduction to Amazon Web Services
Introduction to Amazon Web ServicesIntroduction to Amazon Web Services
Introduction to Amazon Web Services
 
Pooja shift left 1.0
Pooja shift left 1.0Pooja shift left 1.0
Pooja shift left 1.0
 
Shift Left Testing: A New Paradigm Shift To Quality
Shift Left Testing: A New Paradigm Shift To QualityShift Left Testing: A New Paradigm Shift To Quality
Shift Left Testing: A New Paradigm Shift To Quality
 
API Risk: Taking Your API Security to the Next Level
API Risk: Taking Your API Security to the Next LevelAPI Risk: Taking Your API Security to the Next Level
API Risk: Taking Your API Security to the Next Level
 
Quality Assurance/Testing Overview & Capability Deck
Quality Assurance/Testing Overview & Capability DeckQuality Assurance/Testing Overview & Capability Deck
Quality Assurance/Testing Overview & Capability Deck
 
Low Code Capabilities of Digital Product Design Platforms
Low Code Capabilities of Digital Product Design PlatformsLow Code Capabilities of Digital Product Design Platforms
Low Code Capabilities of Digital Product Design Platforms
 
Multiplatform App Architecture
Multiplatform App ArchitectureMultiplatform App Architecture
Multiplatform App Architecture
 
How We Do DevOps at Walmart: OneOps OSS Application Lifecycle Management Plat...
How We Do DevOps at Walmart: OneOps OSS Application Lifecycle Management Plat...How We Do DevOps at Walmart: OneOps OSS Application Lifecycle Management Plat...
How We Do DevOps at Walmart: OneOps OSS Application Lifecycle Management Plat...
 

Similar to Bits of Evidence

Greg Wilson - We Know (but ignore) More Than We Think
Greg Wilson - We Know (but ignore) More Than We ThinkGreg Wilson - We Know (but ignore) More Than We Think
Greg Wilson - We Know (but ignore) More Than We Think#DevTO
 
Why Do Computational Scientists Trust Their So
Why Do Computational Scientists Trust Their SoWhy Do Computational Scientists Trust Their So
Why Do Computational Scientists Trust Their Sojpipitone
 
Possible Essay Questions On Romeo And Juliet
Possible Essay Questions On Romeo And JulietPossible Essay Questions On Romeo And Juliet
Possible Essay Questions On Romeo And JulietJamie Jackson
 
Text, Tags and Thumbnails: Latest Trends in Bioscience Literature Search
Text, Tags and Thumbnails:Latest Trends in Bioscience Literature SearchText, Tags and Thumbnails:Latest Trends in Bioscience Literature Search
Text, Tags and Thumbnails: Latest Trends in Bioscience Literature Searchmarti_hearst
 
Usability Evaluation Considered Harmful (Some of the Time)
Usability Evaluation Considered Harmful (Some of the Time)Usability Evaluation Considered Harmful (Some of the Time)
Usability Evaluation Considered Harmful (Some of the Time)Saul Greenberg
 
Building Interactive Systems for Social Good [Job Talk]
Building Interactive Systems for Social Good [Job Talk]Building Interactive Systems for Social Good [Job Talk]
Building Interactive Systems for Social Good [Job Talk]Matthew Louis Mauriello
 
Software estimation is crap
Software estimation is crapSoftware estimation is crap
Software estimation is crapIan Garrison
 
Examples Of University Application Essay
Examples Of University Application EssayExamples Of University Application Essay
Examples Of University Application EssaySue Jones
 
Validation and mechanism: exploring the limits of evaluation
Validation and mechanism: exploring the limits of evaluationValidation and mechanism: exploring the limits of evaluation
Validation and mechanism: exploring the limits of evaluationAlan Dix
 
Into AB experiments
Into AB experimentsInto AB experiments
Into AB experimentsDeven
 
Becoming a software testing expert
Becoming a software testing expertBecoming a software testing expert
Becoming a software testing expertmoosix
 
Annotated Bibliography .Guidelines Annotated Bibliograph.docx
Annotated Bibliography  .Guidelines Annotated Bibliograph.docxAnnotated Bibliography  .Guidelines Annotated Bibliograph.docx
Annotated Bibliography .Guidelines Annotated Bibliograph.docxjustine1simpson78276
 
Effectiveness of Bug Bounty Program in preventing security v
Effectiveness of Bug Bounty Program in preventing security vEffectiveness of Bug Bounty Program in preventing security v
Effectiveness of Bug Bounty Program in preventing security vEvonCanales257
 
Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013
Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013
Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013TEST Huddle
 
Bug debug keynote - Present problems and future solutions
Bug debug keynote - Present problems and future solutionsBug debug keynote - Present problems and future solutions
Bug debug keynote - Present problems and future solutionsRIA RUI Society
 
Synthesisunit4 121023171541-phpapp02
Synthesisunit4 121023171541-phpapp02Synthesisunit4 121023171541-phpapp02
Synthesisunit4 121023171541-phpapp02Prince Anwar Ali
 

Similar to Bits of Evidence (20)

Greg Wilson - We Know (but ignore) More Than We Think
Greg Wilson - We Know (but ignore) More Than We ThinkGreg Wilson - We Know (but ignore) More Than We Think
Greg Wilson - We Know (but ignore) More Than We Think
 
1325 keynote kohavi
1325 keynote kohavi1325 keynote kohavi
1325 keynote kohavi
 
Why Do Computational Scientists Trust Their So
Why Do Computational Scientists Trust Their SoWhy Do Computational Scientists Trust Their So
Why Do Computational Scientists Trust Their So
 
2014 abic-talk
2014 abic-talk2014 abic-talk
2014 abic-talk
 
Agile numbers
Agile numbersAgile numbers
Agile numbers
 
Possible Essay Questions On Romeo And Juliet
Possible Essay Questions On Romeo And JulietPossible Essay Questions On Romeo And Juliet
Possible Essay Questions On Romeo And Juliet
 
Text, Tags and Thumbnails: Latest Trends in Bioscience Literature Search
Text, Tags and Thumbnails:Latest Trends in Bioscience Literature SearchText, Tags and Thumbnails:Latest Trends in Bioscience Literature Search
Text, Tags and Thumbnails: Latest Trends in Bioscience Literature Search
 
Usability Evaluation Considered Harmful (Some of the Time)
Usability Evaluation Considered Harmful (Some of the Time)Usability Evaluation Considered Harmful (Some of the Time)
Usability Evaluation Considered Harmful (Some of the Time)
 
Building Interactive Systems for Social Good [Job Talk]
Building Interactive Systems for Social Good [Job Talk]Building Interactive Systems for Social Good [Job Talk]
Building Interactive Systems for Social Good [Job Talk]
 
Software estimation is crap
Software estimation is crapSoftware estimation is crap
Software estimation is crap
 
Examples Of University Application Essay
Examples Of University Application EssayExamples Of University Application Essay
Examples Of University Application Essay
 
Validation and mechanism: exploring the limits of evaluation
Validation and mechanism: exploring the limits of evaluationValidation and mechanism: exploring the limits of evaluation
Validation and mechanism: exploring the limits of evaluation
 
TesTrek Notes
TesTrek NotesTesTrek Notes
TesTrek Notes
 
Into AB experiments
Into AB experimentsInto AB experiments
Into AB experiments
 
Becoming a software testing expert
Becoming a software testing expertBecoming a software testing expert
Becoming a software testing expert
 
Annotated Bibliography .Guidelines Annotated Bibliograph.docx
Annotated Bibliography  .Guidelines Annotated Bibliograph.docxAnnotated Bibliography  .Guidelines Annotated Bibliograph.docx
Annotated Bibliography .Guidelines Annotated Bibliograph.docx
 
Effectiveness of Bug Bounty Program in preventing security v
Effectiveness of Bug Bounty Program in preventing security vEffectiveness of Bug Bounty Program in preventing security v
Effectiveness of Bug Bounty Program in preventing security v
 
Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013
Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013
Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013
 
Bug debug keynote - Present problems and future solutions
Bug debug keynote - Present problems and future solutionsBug debug keynote - Present problems and future solutions
Bug debug keynote - Present problems and future solutions
 
Synthesisunit4 121023171541-phpapp02
Synthesisunit4 121023171541-phpapp02Synthesisunit4 121023171541-phpapp02
Synthesisunit4 121023171541-phpapp02
 

Recently uploaded

MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotesMuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotesManik S Magar
 
Generative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdfGenerative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdfIngrid Airi González
 
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesHow to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesThousandEyes
 
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...Wes McKinney
 
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...Jeffrey Haguewood
 
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...amber724300
 
A Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software DevelopersA Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software DevelopersNicole Novielli
 
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...Nikki Chapple
 
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better StrongerModern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better Strongerpanagenda
 
4. Cobus Valentine- Cybersecurity Threats and Solutions for the Public Sector
4. Cobus Valentine- Cybersecurity Threats and Solutions for the Public Sector4. Cobus Valentine- Cybersecurity Threats and Solutions for the Public Sector
4. Cobus Valentine- Cybersecurity Threats and Solutions for the Public Sectoritnewsafrica
 
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)Mark Simos
 
Potential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsPotential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsRavi Sanghani
 
Landscape Catalogue 2024 Australia-1.pdf
Landscape Catalogue 2024 Australia-1.pdfLandscape Catalogue 2024 Australia-1.pdf
Landscape Catalogue 2024 Australia-1.pdfAarwolf Industries LLC
 
UiPath Community: Communication Mining from Zero to Hero
UiPath Community: Communication Mining from Zero to HeroUiPath Community: Communication Mining from Zero to Hero
UiPath Community: Communication Mining from Zero to HeroUiPathCommunity
 
Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Farhan Tariq
 
Decarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a realityDecarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a realityIES VE
 
A Glance At The Java Performance Toolbox
A Glance At The Java Performance ToolboxA Glance At The Java Performance Toolbox
A Glance At The Java Performance ToolboxAna-Maria Mihalceanu
 
Zeshan Sattar- Assessing the skill requirements and industry expectations for...
Zeshan Sattar- Assessing the skill requirements and industry expectations for...Zeshan Sattar- Assessing the skill requirements and industry expectations for...
Zeshan Sattar- Assessing the skill requirements and industry expectations for...itnewsafrica
 
Digital Tools & AI in Career Development
Digital Tools & AI in Career DevelopmentDigital Tools & AI in Career Development
Digital Tools & AI in Career DevelopmentMahmoud Rabie
 
Generative AI - Gitex v1Generative AI - Gitex v1.pptx
Generative AI - Gitex v1Generative AI - Gitex v1.pptxGenerative AI - Gitex v1Generative AI - Gitex v1.pptx
Generative AI - Gitex v1Generative AI - Gitex v1.pptxfnnc6jmgwh
 

Recently uploaded (20)

MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotesMuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
 
Generative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdfGenerative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdf
 
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesHow to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
 
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
 
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...
 
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
 
A Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software DevelopersA Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software Developers
 
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
 
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better StrongerModern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
 
4. Cobus Valentine- Cybersecurity Threats and Solutions for the Public Sector
4. Cobus Valentine- Cybersecurity Threats and Solutions for the Public Sector4. Cobus Valentine- Cybersecurity Threats and Solutions for the Public Sector
4. Cobus Valentine- Cybersecurity Threats and Solutions for the Public Sector
 
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)
 
Potential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsPotential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and Insights
 
Landscape Catalogue 2024 Australia-1.pdf
Landscape Catalogue 2024 Australia-1.pdfLandscape Catalogue 2024 Australia-1.pdf
Landscape Catalogue 2024 Australia-1.pdf
 
UiPath Community: Communication Mining from Zero to Hero
UiPath Community: Communication Mining from Zero to HeroUiPath Community: Communication Mining from Zero to Hero
UiPath Community: Communication Mining from Zero to Hero
 
Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...
 
Decarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a realityDecarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a reality
 
A Glance At The Java Performance Toolbox
A Glance At The Java Performance ToolboxA Glance At The Java Performance Toolbox
A Glance At The Java Performance Toolbox
 
Zeshan Sattar- Assessing the skill requirements and industry expectations for...
Zeshan Sattar- Assessing the skill requirements and industry expectations for...Zeshan Sattar- Assessing the skill requirements and industry expectations for...
Zeshan Sattar- Assessing the skill requirements and industry expectations for...
 
Digital Tools & AI in Career Development
Digital Tools & AI in Career DevelopmentDigital Tools & AI in Career Development
Digital Tools & AI in Career Development
 
Generative AI - Gitex v1Generative AI - Gitex v1.pptx
Generative AI - Gitex v1Generative AI - Gitex v1.pptxGenerative AI - Gitex v1Generative AI - Gitex v1.pptx
Generative AI - Gitex v1Generative AI - Gitex v1.pptx
 

Bits of Evidence

  • 1. Bits of Evidence What We Actually Know About Software Development, and Why We Believe It’s True Greg Wilson http://third-bit.com Feb 2010
  • 2. Once Upon a Time... Seven Years’ War (actually 1754-63) Britain lost 1,512 sailors to enemy action... ...and almost 100,000 to scurvy
  • 3.
  • 4. It Took a While to Catch On 1950: Hill & Doll publish a case-control study comparing smokers with non-smokers 1951: start the British Doctors Study (which runs until 2001)
  • 5. What They Discovered #1: Smoking causes lung cancer “ ...what happens ‘on average’ is of no help when one is faced with a specific patient...” #2: Many people would rather fail than change
  • 6. Like Water on Stone 1992: Sackett coins the term “ evidence-based medicine” Randomized double-blind trials are accepted as the gold standard for medical research The Cochrane Collaboration (http://www.cochrane.org/) now archives results from hundreds of medical studies
  • 7. So Where Are We? “ [Using domain-specific languages] leads to two primary benefits. The first, and simplest, is improved programmer productivity... The second...is...communication with domain experts.” – Martin Fowler (IEEE Software, July/August 2009)
  • 8. Say Again? One of the smartest guys in our industry... ...made two substantive claims... ...in an academic journal... ...without a single citation Please note: I’m not disagreeing with his claims —I just want to point out that even the best of us aren’t doing what we expect the makers of acne creams to do.
  • 9. Um, No “ Debate still continues about how valuable DSLs are in practice. I believe that debate is hampered because not enough people know how to develop DSLs effectively.” I think debate is hampered by low standards for proof The good news is, things have started to improve
  • 10. The Times They Are A-Changin’ Growing emphasis on empirical studies in software engineering research since the mid-1990s Papers describing new tools or practices routinely include results from some kind of field study Yes, many are flawed or incomplete, but standards are constantly improving
  • 11. My Favorite Little Result Aranda & Easterbrook (2005): “Anchoring and Adjustment in Software Estimation” “ How long do you think it will take to make a change to this program?” Control Group: “ I’d like to give an estimate for this project myself, but I admit I have no experience estimating. We’ll wait for your calculations for an estimate.” Group A: “I admit I have no experience with software projects, but I guess this will take about 2 months to finish. ” Group B: “...I guess this will take about 20 months... ”
  • 12. Results The anchor mattered more than experience, how formal the estimation method was, or anything else. Q: Are agile projects similarly afflicted, just on a shorter and more rapid cycle? Group A (lowball) 5.1 months Control Group 7.8 months Group B (highball) 15.4 months
  • 13. Most Frequently Misquoted Sackman, Erikson, and Grant (1968): “Exploratory experimental studies comparing online and offline programming performance.” Or 10, or 40, or 100, or whatever other large number pops into the head of someone who can’t be bothered to look up the reference... The best programmers are up to 28 times more productive than the worst.
  • 14.
  • 15.
  • 16.
  • 17. ...Which Explains a Lot Pessimists: “If we tackle the hump in the error injection curve, fewer bugs will get to the expensive part of the fixing curve.” Optimists: “If we do lots of short iterations, the total cost of fixing bugs will go down.”
  • 18. The Real Reason I Care A: I've always believed that there are just fundamental differences between the sexes... B: What data are you basing that opinion on? A: It's more of an unrefuted hypothesis based on personal observation. I have read a few studies on the topic and I found them unconvincing... B: Which studies were those? A: [no reply]
  • 19.
  • 20.
  • 21.
  • 22.
  • 23. Another Personal Favorite Conway’s Law: A system reflects the organizational structure that built it. Meant as a joke Turns out to be true (Herbsleb et al 1999)
  • 24. But Wait, There’s More! Nagappan et al (2007) & Bird et al (2009): Physical distance doesn’t affect post-release fault rates Distance in the organizational chart does No, really — shouldn’t our development practices be built around these facts?
  • 25.
  • 26. Folk Medicine for Software Systematizing and synthesizing colloquial practice has been very productive in other disciplines…
  • 27. How Do We Get There? 2007 2008 – 2009
  • 28. The Book Without a Name Wanted to call the next one Beautiful Evidence , but Edward Tufte got there first “ What we know and why we think it’s true” (By the way, his book is really good) Knowledge transfer A better textbook Change the debate
  • 29. A Lot Of Editing In My Future Jorge Aranda Tom Ball Victor Basili Andrew Begel Christian Bird Barry Boehm Marcelo Cataldo Steven Clarke Jason Cohen Rob DeLine Khaled El Emam Hakan Erdogmus Michael Godfrey Mark Guzdial Jo Hannay Ahmed Hassan Israel Herraiz Kim Herzig Barbara Kitchenham Andrew Ko Lucas Layman Steve McConnell Audris Mockus Gail Murphy Nachi Nagappan Tom Ostrand Dewayne Perry Marian Petre Lutz Prechelt Rahul Premraj Dieter Rombach Forrest Shull Beth Simon Janice Singer Diomidis Spinellis Neil Thomas Walter Tichy Burak Turhan Gina Venolia Elaine Weyuker Laurie Williams Andreas Zeller Tom Zimmermann
  • 31. The Real Reason It Matters
  • 32. Thank you, and good luck