SlideShare a Scribd company logo
1 of 22
Download to read offline
BT1	
Session	
6/9/16	10:00	AM	
	
	
	
	
	
	
Experiments:	The	Good,	the	Bad,	and	
the	Beautiful	
	
Presented	by:	
	
Linda	Rising	
Independent	Consultant	
	
	
Brought	to	you	by:		
		
	
	
	
	
350	Corporate	Way,	Suite	400,	Orange	Park,	FL	32073		
888---268---8770	··	904---278---0524	-	info@techwell.com	-	http://www.techwell.com/
Linda	Rising	
Independent	Consultant	
	
Linda	Rising	is	an	internationally-known	presenter	on	topics	of	agile	
development,	patterns,	retrospectives,	the	change	process,	and	the	connection	
between	the	latest	neuroscience	and	software	development.	An	independent	
consultant	from	Tennessee,	Linda	(lindarising.org)	has	a	background	in	university	
teaching	and	work	in	industries	of	telecommunications,	avionics,	and	strategic	
weapons	systems.	She	has	authored	numerous	articles	and	published	several	
books:	Design	Patterns	in	Communications,	The	Pattern	Almanac	2000,	A	
Patterns	Handbook,	Fearless	Change:	Patterns	for	Introducing	New	Ideas	(with	
coauthor	Mary	Lynn	Manns),	and	her	latest	More	Fearless	Change	(also	with	
coauthor	Mary	Lynn	Manns).
5/30/16	
1	
Experiments: the Good,
the Bad, and the
Beautiful
Linda Rising
www.lindarising.org
linda@lindarising.org
@RisingLinda
A short survey
How many are doing some flavor of Agile?
How did your organization decide to do
that?
How many looked at the randomized,
controlled studies that provided evidence
that Agile was better than your current
process?
5/30/16	
2	
Not much science
n  The short history of software is not
progress based on scientific experiments.
n  Instead we jump on the latest
bandwagon because we hear a good
story.
n  These are not even really case studies.
n  The plural of anecdote is not data ☺!
Aren’t we experimenting?
n  “Experiment” for most organizations really
means “try.”
n  No clear hypothesis, just an idea to try.
n  No randomization. Usually those who
participate are enthusiastic believers.
n  No control group, just the memory of the way
things were.
n  No analysis, perhaps some easy-to-measure
attributes or good “feelings.”
5/30/16	
3	
How about A/B Tests?
n  A/B tests could be scientific, but usually
aren’t.
n  No clear hypothesis, just a couple of
different ways to do something.
n  Results don’t build toward a theory or
real learning.
n  Results are context-sensitive and may not
apply elsewhere.
How about saying “trial”?
We don’t have resources for one, let
alone repeated, experiments that
good science requires.
Just “trying” is our best hope.
5/30/16	
4	
Industrial age workers
Schools created in that
image
5/30/16	
5	
We’re natural scientists!
Our educational system?
Science education focuses on “what” to think
about, that is, content, not “how” to think.
Problems in school are solvable. Problems in
the “real world” often have no solution.
We’re taught to be linear thinkers—to follow
pre-established procedures and plans—in
a nonlinear world.
It’s not always a joyous experience "!
5/30/16	
6	
Agile contributions
Failure and learning are important.
Shifting from following a checklist to
stopping after a short iteration to get
feedback.
Agile can help move us to a somewhat more
scientific approach.
Each iteration can be framed as a small trial,
an empirical, incremental approach.
Why do trials?
n  Answer: To “prove” something
n  Problem: One trial “proves” little,
especially if treatment is not randomized
or controlled.
n  “Once and done” doesn’t work. We need
lots and lots of trials. The process never
ends.
5/30/16	
7	
Even scientists are biased
n  Drug trials are now “double-blind” because it
was discovered that if researchers and doctors
knew which patients were getting “real”
treatment, that would change the outcome.
n  Even scientists suffer from confirmation bias.
I wouldn’t believe that – even if it were
true! -- Anonymous reviewer of scientific paper
n  We do not see things as they are—we see them
as we are -- Talmudic saying
What CAN we do?
n  Many small, simple, fast, frugal trials.
n  Vary contexts, number of participants,
degree of enthusiasm, kind of project.
n  Re-test. Keep learning.
5/30/16	
8	
Why Small?
n  Huge experiments leave no room for
failure. Use small, cheap trials that
barely register if they don’t work
out.
n  Keep asking, “Can we make it even
smaller?”
Why Simple?
n  Everyone feels safe to try something
that might bring benefit.
n  Encourage each other to try!
n  Keep asking, “Can we make it even
simpler?”
5/30/16	
9	
Why Cheap?
n  Counter the sunk-cost fallacy. It’s
surprising how little an investment it
takes to get us to avoid “wasting” that
effort.
n  Keep asking, “Can we make it even
cheaper?”
Why Time-Box?
n  Begin with the end in mind - Stephen
Covey
n  Encourages openness and a way around
confirmation bias: “Let’s try having the
stand-up at 10 am instead of 8 am for the
NEXT TWO WEEKS and see if
attendance is better and we get more
done in the morning.”
5/30/16	
10	
Why Faster?
Establish a standard of fast, frequent, and inexpensive
experimentation. Assume that many of your experiments
will fail. One of the most common phrases you’ll hear at
Menlo is “Let’s run the experiment.” We are apt to say that
at least once a day. We don’t count experiments and we
don’t track success/failure rates, but if we did, we would
look for success and failure rates to be about even. If the
percentage of failures started dropping, we’d become
concerned that fear had crept into the room and that people
weren’t taking enough risks.
Rich Sheridan, Co-founder and CEO Menlo Innovations
Why do trials?
n  Answer: To show “those people”
n  Problem: Research shows that
evidence/data is not convincing but
only serves to bolster our own
beliefs (confirmation bias)
5/30/16	
11	
Better ways to convince
n  Include others in trials -- encourage
sharing
n  Use patterns from Fearless Change
n  Involve Everyone – don’t hide – use the
trial to draw others in – watch your
language – more “us” and less “them”
n  Hometown Story – share results and
encourage others to tell their stories
Why do trials?
n  Answer: To find a solution to a
problem.
n  Problem: Intervention in a
complex system tends to create
unanticipated and often
undesirable outcomes.
5/30/16	
12	
Change in a complex system
n  Probe, Sense, Respond
n  “On Understanding Software
Agility: A Social Complexity Point
Of View” Joseph Pelrine
n  Stop looking for answers.
n  Discover ideas for more trials.
Uncertainty
n  We don’t know and we don’t
know what we don’t know (thank
you, Donald Rumsfeld)
n  Experiments involve risk,
uncertainty, and failure—no
wonder we don’t do them ☺!
n  Prepare to be surprised ☺!
5/30/16	
13	
Alexander Fleming said,
“That’s funny!”
…and we weren’t even trying
for that!
“Thank God it’s Open
Friday,” Corinna Baldauf at
Sipgate
5/30/16	
14	
Why do trials?
n  Answer: To convince
management
n  This makes sense! BUT have
your ducks in a row.
Presentations to management
n  Easy to follow, easy to understand, easy to share, easy
to believe.
n  No extraneous complications or technical jargon.
n  Good ideas can crash and burn because more thought
wasn’t put into the presentation.
n  Less emphasis on details. More on communicating
strategic value. Less about failure. More about
learning.
n  Executives should ask, “Why haven’t we thought of
this in that way before?”
n  The Innovator’s Hypothesis, Michael Schrage
5/30/16	
15	
Benefits of trials
You kill off the HiPPOs. (Highest Paid Person’s
Opinion.) Testing is a sure way to get to the
bottom of a decision without relying on anyone’s
gut instinct. At Shutterstock, if a senior executive
has an idea in a meeting, the response is simply
“Let’s test it.”
Wyatt Jenkins, VP of Product
Confirmation bias
Our tendency to search for, interpret, favor, and recall (yes,
it affects memory) information to confirm our beliefs.
Charles Darwin said that whenever he ran into something
that contradicted a notion he cherished, he wrote down the
new finding within 30 minutes. Otherwise his mind would
start to work to reject the discordant information, much as
the body rejects transplants. Man’s natural inclination is to
cling to his beliefs, particularly if they are reinforced by
recent experience. – Warren Buffet
5/30/16	
16	
Believing is seeing
n  We begin our scientific study with the
result we want to see and then the
confirmation bias kicks in, so it's
difficult to be objective about the
experiment.
n  Once we have a belief, we only see the
information that will confirm that belief.
We stop seeing what we don’t want to
see.
Cognitive dissonance
n  It’s difficult for us to hold two
disconfirming ideas at the same time.
n  To truly test each hypothesis, we have to
be open to showing that we might be
wrong.
5/30/16	
17	
This bias is challenging!
…because the scientific method is designed to
create dissonance…one of the reasons science is so
difficult—because scientists are humans, and
scientists don’t like it when their predictions are
disconfirmed.
I wish for every student that something they deeply
hold to be true is shown to be wrong. Once you’ve
had that experience, then you get it; then you get
what science is about. - Lawrence Krauss
Some help for biases
n  Talk out loud and use words like ‘rational,’
‘scientific’ and ‘experiment.’ Say, “Most people
want to overcome their biases.”
n  Write -- on paper, white board, flip chart
n  Diversity – include skeptics – listen to all
contributors
n  Be aware and alert for bias -- ask questions
n  Slow down. Take a break. Get enough sleep.
5/30/16	
18	
Correlation is not causation
Honolulu Heart Program: 8,004 men studied over 30 years,
examined relationship between coffee intake and the
incidence of Parkinson's. Men who drank the most coffee
were least likely to get Parkinson's. Men who did not drink
coffee were 5 times more likely to exhibit symptoms of
Parkinson's than men who drank more than 28 oz of coffee
each day.
Now we know that there's a genetic connection between
liking coffee and risk for Parkinson's. It's not that coffee
prevents the disease. It's that not liking coffee means that
you are at risk for the disease (no one is sure why).
In a disagreement
Instead of arguing and taking up
valuable meeting time, ask…
“What small, fast, frugal trial would
help us answer this question?”
5/30/16	
19	
Other benefits
If you are careful in framing your
hypothesis and in designing your
experiments, you will get better over time.
This is another great result – the
experimenters themselves and an
environment of replacing argument and
heated discussion with ‘experiments.’ A
culture of experimentation ☺!
Reading suggestions
The WHY Axis, Uri Gneezy & John List
Innovator’s Hypothesis, Michael Schrage
Little Bets, Peter Sims
Fearless Change and More Fearless Change, Manns & Rising
Anything Dan Ariely has written
Thinking Fast and Slow, Daniel Kahneman
Mistakes Were Made (But Not by Me), Carol Tavris & Elliot
Aronson
Joy, Inc., Richard Sheridan
5/30/16	
20	
More reading suggestions
Behavioral Insights Team, “Test, Learn,
Adapt”
https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/62529/TLA-1906126.pdf
Try an experiment!
n  Have fun!
n  Think like a child!
n  Embrace failure!
n  Thanks for listening…

More Related Content

What's hot

Rogues At DNI Aug 06 Part 1
Rogues At DNI Aug 06 Part 1Rogues At DNI Aug 06 Part 1
Rogues At DNI Aug 06 Part 1Gabe Mounce
 
The Science of Behavior Change
The Science of Behavior ChangeThe Science of Behavior Change
The Science of Behavior ChangeJulie Dirksen
 
Insights from psychology on lack of reproducibility
Insights from psychology on lack of reproducibilityInsights from psychology on lack of reproducibility
Insights from psychology on lack of reproducibilityDorothy Bishop
 
Short talk on 2 cognitive biases and reproducibility
Short talk on 2 cognitive biases and reproducibilityShort talk on 2 cognitive biases and reproducibility
Short talk on 2 cognitive biases and reproducibilityDorothy Bishop
 
Screwing up for fun and profit
Screwing up for fun and profitScrewing up for fun and profit
Screwing up for fun and profitOliver Hankeln
 
What I Learned By Talking to 100+ Teams About Data
What I Learned By Talking to 100+ Teams About DataWhat I Learned By Talking to 100+ Teams About Data
What I Learned By Talking to 100+ Teams About DataJohn Cutler
 
PopcornFlow: Continuous Evolution Through Ultra-Rapid Experimentation
PopcornFlow: Continuous Evolution Through Ultra-Rapid ExperimentationPopcornFlow: Continuous Evolution Through Ultra-Rapid Experimentation
PopcornFlow: Continuous Evolution Through Ultra-Rapid ExperimentationClaudio Perrone
 
Critical Thinking for Software Testers
Critical Thinking for Software TestersCritical Thinking for Software Testers
Critical Thinking for Software TestersTechWell
 
Accident investigation BY Muhammad Fahad Ansari 12IEEM14
Accident investigation BY Muhammad Fahad Ansari 12IEEM14Accident investigation BY Muhammad Fahad Ansari 12IEEM14
Accident investigation BY Muhammad Fahad Ansari 12IEEM14fahadansari131
 
CYCLES Course (2): Alignment
CYCLES Course (2): AlignmentCYCLES Course (2): Alignment
CYCLES Course (2): AlignmentBryan Cassady
 
Cognitve biases and project management
Cognitve biases  and project managementCognitve biases  and project management
Cognitve biases and project managementLuc Taesch
 
Better Startups Through Science - MIT Accelerator 2014
Better Startups Through Science -  MIT Accelerator 2014Better Startups Through Science -  MIT Accelerator 2014
Better Startups Through Science - MIT Accelerator 2014Colin Kennedy
 
CYCLES Course (1): Course Introduction
CYCLES Course (1): Course Introduction CYCLES Course (1): Course Introduction
CYCLES Course (1): Course Introduction Bryan Cassady
 
Innovation Chilling Challenge Or Childs Play
Innovation   Chilling Challenge Or Childs PlayInnovation   Chilling Challenge Or Childs Play
Innovation Chilling Challenge Or Childs Playskukula
 
The Snail Entrepreneur: The 7-year-old kid every startup should learn from
The Snail Entrepreneur: The 7-year-old kid every startup should learn fromThe Snail Entrepreneur: The 7-year-old kid every startup should learn from
The Snail Entrepreneur: The 7-year-old kid every startup should learn fromClaudio Perrone
 
Why You're a Terrible PM: Cognitive Biases in Project Management notes
Why You're a Terrible PM: Cognitive Biases in Project Management notesWhy You're a Terrible PM: Cognitive Biases in Project Management notes
Why You're a Terrible PM: Cognitive Biases in Project Management notesCarson Pierce
 
Entrepreneur Talk for Transition Year Students
Entrepreneur Talk for Transition Year StudentsEntrepreneur Talk for Transition Year Students
Entrepreneur Talk for Transition Year Studentskantanmt
 
Incident Investigation Guidelines
Incident Investigation GuidelinesIncident Investigation Guidelines
Incident Investigation Guidelinessimurares
 
What Kills Our Curiosity?
What Kills Our Curiosity?What Kills Our Curiosity?
What Kills Our Curiosity?Mike Rother
 
Product Market Fit - lessons from the hampster wheel
Product Market Fit - lessons from the hampster wheelProduct Market Fit - lessons from the hampster wheel
Product Market Fit - lessons from the hampster wheelDavid Jones
 

What's hot (20)

Rogues At DNI Aug 06 Part 1
Rogues At DNI Aug 06 Part 1Rogues At DNI Aug 06 Part 1
Rogues At DNI Aug 06 Part 1
 
The Science of Behavior Change
The Science of Behavior ChangeThe Science of Behavior Change
The Science of Behavior Change
 
Insights from psychology on lack of reproducibility
Insights from psychology on lack of reproducibilityInsights from psychology on lack of reproducibility
Insights from psychology on lack of reproducibility
 
Short talk on 2 cognitive biases and reproducibility
Short talk on 2 cognitive biases and reproducibilityShort talk on 2 cognitive biases and reproducibility
Short talk on 2 cognitive biases and reproducibility
 
Screwing up for fun and profit
Screwing up for fun and profitScrewing up for fun and profit
Screwing up for fun and profit
 
What I Learned By Talking to 100+ Teams About Data
What I Learned By Talking to 100+ Teams About DataWhat I Learned By Talking to 100+ Teams About Data
What I Learned By Talking to 100+ Teams About Data
 
PopcornFlow: Continuous Evolution Through Ultra-Rapid Experimentation
PopcornFlow: Continuous Evolution Through Ultra-Rapid ExperimentationPopcornFlow: Continuous Evolution Through Ultra-Rapid Experimentation
PopcornFlow: Continuous Evolution Through Ultra-Rapid Experimentation
 
Critical Thinking for Software Testers
Critical Thinking for Software TestersCritical Thinking for Software Testers
Critical Thinking for Software Testers
 
Accident investigation BY Muhammad Fahad Ansari 12IEEM14
Accident investigation BY Muhammad Fahad Ansari 12IEEM14Accident investigation BY Muhammad Fahad Ansari 12IEEM14
Accident investigation BY Muhammad Fahad Ansari 12IEEM14
 
CYCLES Course (2): Alignment
CYCLES Course (2): AlignmentCYCLES Course (2): Alignment
CYCLES Course (2): Alignment
 
Cognitve biases and project management
Cognitve biases  and project managementCognitve biases  and project management
Cognitve biases and project management
 
Better Startups Through Science - MIT Accelerator 2014
Better Startups Through Science -  MIT Accelerator 2014Better Startups Through Science -  MIT Accelerator 2014
Better Startups Through Science - MIT Accelerator 2014
 
CYCLES Course (1): Course Introduction
CYCLES Course (1): Course Introduction CYCLES Course (1): Course Introduction
CYCLES Course (1): Course Introduction
 
Innovation Chilling Challenge Or Childs Play
Innovation   Chilling Challenge Or Childs PlayInnovation   Chilling Challenge Or Childs Play
Innovation Chilling Challenge Or Childs Play
 
The Snail Entrepreneur: The 7-year-old kid every startup should learn from
The Snail Entrepreneur: The 7-year-old kid every startup should learn fromThe Snail Entrepreneur: The 7-year-old kid every startup should learn from
The Snail Entrepreneur: The 7-year-old kid every startup should learn from
 
Why You're a Terrible PM: Cognitive Biases in Project Management notes
Why You're a Terrible PM: Cognitive Biases in Project Management notesWhy You're a Terrible PM: Cognitive Biases in Project Management notes
Why You're a Terrible PM: Cognitive Biases in Project Management notes
 
Entrepreneur Talk for Transition Year Students
Entrepreneur Talk for Transition Year StudentsEntrepreneur Talk for Transition Year Students
Entrepreneur Talk for Transition Year Students
 
Incident Investigation Guidelines
Incident Investigation GuidelinesIncident Investigation Guidelines
Incident Investigation Guidelines
 
What Kills Our Curiosity?
What Kills Our Curiosity?What Kills Our Curiosity?
What Kills Our Curiosity?
 
Product Market Fit - lessons from the hampster wheel
Product Market Fit - lessons from the hampster wheelProduct Market Fit - lessons from the hampster wheel
Product Market Fit - lessons from the hampster wheel
 

Viewers also liked

Project Estimation: Myths, Taboos, and Inconvenient Truths
Project Estimation: Myths, Taboos, and Inconvenient TruthsProject Estimation: Myths, Taboos, and Inconvenient Truths
Project Estimation: Myths, Taboos, and Inconvenient TruthsTechWell
 
Playwriting, Imagination, and Agile Software Development … Oh My!
Playwriting, Imagination, and Agile Software Development … Oh My!Playwriting, Imagination, and Agile Software Development … Oh My!
Playwriting, Imagination, and Agile Software Development … Oh My!TechWell
 
Use Business Analysts for User Interface Design
Use Business Analysts for User Interface DesignUse Business Analysts for User Interface Design
Use Business Analysts for User Interface DesignTechWell
 
Scaling Scrum with Scrum™ (SSwS): A Universal Framework
Scaling Scrum with Scrum™ (SSwS): A Universal FrameworkScaling Scrum with Scrum™ (SSwS): A Universal Framework
Scaling Scrum with Scrum™ (SSwS): A Universal FrameworkTechWell
 
What Everyone on the Team Needs to Know about Test Automation
What Everyone on the Team Needs to Know about Test AutomationWhat Everyone on the Team Needs to Know about Test Automation
What Everyone on the Team Needs to Know about Test AutomationTechWell
 
Continuous Discovery: The Path to Learning and Growing
Continuous Discovery: The Path to Learning and GrowingContinuous Discovery: The Path to Learning and Growing
Continuous Discovery: The Path to Learning and GrowingTechWell
 
The Soft Skills of Great Software Developers
The Soft Skills of Great Software DevelopersThe Soft Skills of Great Software Developers
The Soft Skills of Great Software DevelopersTechWell
 
Identify and Exploit Behavioral Boundaries for Unit Testing
Identify and Exploit Behavioral Boundaries for Unit TestingIdentify and Exploit Behavioral Boundaries for Unit Testing
Identify and Exploit Behavioral Boundaries for Unit TestingTechWell
 
Apply Phil Jackson’s Coaching Principles to Build Better Agile Teams
Apply Phil Jackson’s Coaching Principles to Build Better Agile TeamsApply Phil Jackson’s Coaching Principles to Build Better Agile Teams
Apply Phil Jackson’s Coaching Principles to Build Better Agile TeamsTechWell
 
Don’t Make These Scrum Mistakes
Don’t Make These Scrum MistakesDon’t Make These Scrum Mistakes
Don’t Make These Scrum MistakesTechWell
 
Determining Business Value in Agile Development
Determining Business Value in Agile DevelopmentDetermining Business Value in Agile Development
Determining Business Value in Agile DevelopmentJosiah Renaudin
 
Which Agile Scaling Framework Is Best?
Which Agile Scaling Framework Is Best?Which Agile Scaling Framework Is Best?
Which Agile Scaling Framework Is Best?TechWell
 
What Hollywood Can Teach Us about Software Testing
What Hollywood Can Teach Us about Software TestingWhat Hollywood Can Teach Us about Software Testing
What Hollywood Can Teach Us about Software TestingTechWell
 
Agile Hacks: Creative Solutions for Common Agile Issues
Agile Hacks: Creative Solutions for Common Agile IssuesAgile Hacks: Creative Solutions for Common Agile Issues
Agile Hacks: Creative Solutions for Common Agile IssuesTechWell
 
Your User Stories Are Too Big: Yes, They Are!
Your User Stories Are Too Big: Yes, They Are!Your User Stories Are Too Big: Yes, They Are!
Your User Stories Are Too Big: Yes, They Are!TechWell
 
How Far Can You Go with Agile for Embedded Software?
How Far Can You Go with Agile for Embedded Software?How Far Can You Go with Agile for Embedded Software?
How Far Can You Go with Agile for Embedded Software?TechWell
 
How to Do Kick-Ass Software Development
How to Do Kick-Ass Software DevelopmentHow to Do Kick-Ass Software Development
How to Do Kick-Ass Software DevelopmentJosiah Renaudin
 
Continuous Integration as a Development Team’s Way of Life
Continuous Integration as a Development Team’s Way of LifeContinuous Integration as a Development Team’s Way of Life
Continuous Integration as a Development Team’s Way of LifeTechWell
 
White Box Testing: It’s Not Just for Developers Any More
White Box Testing: It’s Not Just for Developers Any MoreWhite Box Testing: It’s Not Just for Developers Any More
White Box Testing: It’s Not Just for Developers Any MoreTechWell
 
Use Feature Flags for Clean Deployments
Use Feature Flags for Clean DeploymentsUse Feature Flags for Clean Deployments
Use Feature Flags for Clean DeploymentsTechWell
 

Viewers also liked (20)

Project Estimation: Myths, Taboos, and Inconvenient Truths
Project Estimation: Myths, Taboos, and Inconvenient TruthsProject Estimation: Myths, Taboos, and Inconvenient Truths
Project Estimation: Myths, Taboos, and Inconvenient Truths
 
Playwriting, Imagination, and Agile Software Development … Oh My!
Playwriting, Imagination, and Agile Software Development … Oh My!Playwriting, Imagination, and Agile Software Development … Oh My!
Playwriting, Imagination, and Agile Software Development … Oh My!
 
Use Business Analysts for User Interface Design
Use Business Analysts for User Interface DesignUse Business Analysts for User Interface Design
Use Business Analysts for User Interface Design
 
Scaling Scrum with Scrum™ (SSwS): A Universal Framework
Scaling Scrum with Scrum™ (SSwS): A Universal FrameworkScaling Scrum with Scrum™ (SSwS): A Universal Framework
Scaling Scrum with Scrum™ (SSwS): A Universal Framework
 
What Everyone on the Team Needs to Know about Test Automation
What Everyone on the Team Needs to Know about Test AutomationWhat Everyone on the Team Needs to Know about Test Automation
What Everyone on the Team Needs to Know about Test Automation
 
Continuous Discovery: The Path to Learning and Growing
Continuous Discovery: The Path to Learning and GrowingContinuous Discovery: The Path to Learning and Growing
Continuous Discovery: The Path to Learning and Growing
 
The Soft Skills of Great Software Developers
The Soft Skills of Great Software DevelopersThe Soft Skills of Great Software Developers
The Soft Skills of Great Software Developers
 
Identify and Exploit Behavioral Boundaries for Unit Testing
Identify and Exploit Behavioral Boundaries for Unit TestingIdentify and Exploit Behavioral Boundaries for Unit Testing
Identify and Exploit Behavioral Boundaries for Unit Testing
 
Apply Phil Jackson’s Coaching Principles to Build Better Agile Teams
Apply Phil Jackson’s Coaching Principles to Build Better Agile TeamsApply Phil Jackson’s Coaching Principles to Build Better Agile Teams
Apply Phil Jackson’s Coaching Principles to Build Better Agile Teams
 
Don’t Make These Scrum Mistakes
Don’t Make These Scrum MistakesDon’t Make These Scrum Mistakes
Don’t Make These Scrum Mistakes
 
Determining Business Value in Agile Development
Determining Business Value in Agile DevelopmentDetermining Business Value in Agile Development
Determining Business Value in Agile Development
 
Which Agile Scaling Framework Is Best?
Which Agile Scaling Framework Is Best?Which Agile Scaling Framework Is Best?
Which Agile Scaling Framework Is Best?
 
What Hollywood Can Teach Us about Software Testing
What Hollywood Can Teach Us about Software TestingWhat Hollywood Can Teach Us about Software Testing
What Hollywood Can Teach Us about Software Testing
 
Agile Hacks: Creative Solutions for Common Agile Issues
Agile Hacks: Creative Solutions for Common Agile IssuesAgile Hacks: Creative Solutions for Common Agile Issues
Agile Hacks: Creative Solutions for Common Agile Issues
 
Your User Stories Are Too Big: Yes, They Are!
Your User Stories Are Too Big: Yes, They Are!Your User Stories Are Too Big: Yes, They Are!
Your User Stories Are Too Big: Yes, They Are!
 
How Far Can You Go with Agile for Embedded Software?
How Far Can You Go with Agile for Embedded Software?How Far Can You Go with Agile for Embedded Software?
How Far Can You Go with Agile for Embedded Software?
 
How to Do Kick-Ass Software Development
How to Do Kick-Ass Software DevelopmentHow to Do Kick-Ass Software Development
How to Do Kick-Ass Software Development
 
Continuous Integration as a Development Team’s Way of Life
Continuous Integration as a Development Team’s Way of LifeContinuous Integration as a Development Team’s Way of Life
Continuous Integration as a Development Team’s Way of Life
 
White Box Testing: It’s Not Just for Developers Any More
White Box Testing: It’s Not Just for Developers Any MoreWhite Box Testing: It’s Not Just for Developers Any More
White Box Testing: It’s Not Just for Developers Any More
 
Use Feature Flags for Clean Deployments
Use Feature Flags for Clean DeploymentsUse Feature Flags for Clean Deployments
Use Feature Flags for Clean Deployments
 

Similar to Experiments: The Good, the Bad, and the Beautiful

Beyond the agile mindset agile 2012
Beyond the agile mindset   agile 2012Beyond the agile mindset   agile 2012
Beyond the agile mindset agile 2012drewz lin
 
Psyc 321_01 what is science
Psyc 321_01 what is sciencePsyc 321_01 what is science
Psyc 321_01 what is scienceRyan Sain
 
Persuasion
PersuasionPersuasion
Persuasionadefred
 
Linda rising - the power of an agile mindset
Linda rising  - the power of an agile mindsetLinda rising  - the power of an agile mindset
Linda rising - the power of an agile mindsetMagneta AI
 
Geelong College Presentation
Geelong College PresentationGeelong College Presentation
Geelong College PresentationGeoff Brown
 
Complexity, Collaboration and Unconferencing
Complexity, Collaboration and UnconferencingComplexity, Collaboration and Unconferencing
Complexity, Collaboration and UnconferencingGeoff Brown
 
Persuasion Slideshow
Persuasion SlideshowPersuasion Slideshow
Persuasion Slideshowmrhundermark
 
Research method
Research methodResearch method
Research methodCh Irfan
 
Mindset presentation currie cluster jan 2015
Mindset presentation currie cluster jan 2015Mindset presentation currie cluster jan 2015
Mindset presentation currie cluster jan 2015curriechs
 
Y12 res meth workbook hanan
Y12 res meth workbook hananY12 res meth workbook hanan
Y12 res meth workbook hananhma1
 
83270_book_item_83270.pdf
83270_book_item_83270.pdf83270_book_item_83270.pdf
83270_book_item_83270.pdfLawiiiKkk
 
Influence strategies for practitioners - Linda Rising
Influence strategies for practitioners - Linda RisingInfluence strategies for practitioners - Linda Rising
Influence strategies for practitioners - Linda RisingAgileSparks
 
Critical Analytical ThinkingPart II Heuristics and Bias.docx
Critical Analytical ThinkingPart II Heuristics and Bias.docxCritical Analytical ThinkingPart II Heuristics and Bias.docx
Critical Analytical ThinkingPart II Heuristics and Bias.docxannettsparrow
 
Persuasion architectures: Nudging People to do the Right Thing
Persuasion architectures: Nudging People to do the Right ThingPersuasion architectures: Nudging People to do the Right Thing
Persuasion architectures: Nudging People to do the Right ThingUser Vision
 

Similar to Experiments: The Good, the Bad, and the Beautiful (20)

Beyond the agile mindset agile 2012
Beyond the agile mindset   agile 2012Beyond the agile mindset   agile 2012
Beyond the agile mindset agile 2012
 
Psyc 321_01 what is science
Psyc 321_01 what is sciencePsyc 321_01 what is science
Psyc 321_01 what is science
 
Failure is an Option - Ben Yerxa, PhD - ChemE On Demand
Failure is an Option - Ben Yerxa, PhD - ChemE On DemandFailure is an Option - Ben Yerxa, PhD - ChemE On Demand
Failure is an Option - Ben Yerxa, PhD - ChemE On Demand
 
Persuasion
PersuasionPersuasion
Persuasion
 
Linda rising - the power of an agile mindset
Linda rising  - the power of an agile mindsetLinda rising  - the power of an agile mindset
Linda rising - the power of an agile mindset
 
Geelong College Presentation
Geelong College PresentationGeelong College Presentation
Geelong College Presentation
 
Complexity, Collaboration and Unconferencing
Complexity, Collaboration and UnconferencingComplexity, Collaboration and Unconferencing
Complexity, Collaboration and Unconferencing
 
Persuasion Slideshow
Persuasion SlideshowPersuasion Slideshow
Persuasion Slideshow
 
Research method
Research methodResearch method
Research method
 
Agile mindset BY Linda Rising
Agile mindset BY Linda RisingAgile mindset BY Linda Rising
Agile mindset BY Linda Rising
 
Es estadísticas duro
Es estadísticas duroEs estadísticas duro
Es estadísticas duro
 
AUA Networking
AUA NetworkingAUA Networking
AUA Networking
 
Mindset presentation currie cluster jan 2015
Mindset presentation currie cluster jan 2015Mindset presentation currie cluster jan 2015
Mindset presentation currie cluster jan 2015
 
Y12 res meth workbook hanan
Y12 res meth workbook hananY12 res meth workbook hanan
Y12 res meth workbook hanan
 
Mental Models
Mental ModelsMental Models
Mental Models
 
83270_book_item_83270.pdf
83270_book_item_83270.pdf83270_book_item_83270.pdf
83270_book_item_83270.pdf
 
Using evidence
Using evidenceUsing evidence
Using evidence
 
Influence strategies for practitioners - Linda Rising
Influence strategies for practitioners - Linda RisingInfluence strategies for practitioners - Linda Rising
Influence strategies for practitioners - Linda Rising
 
Critical Analytical ThinkingPart II Heuristics and Bias.docx
Critical Analytical ThinkingPart II Heuristics and Bias.docxCritical Analytical ThinkingPart II Heuristics and Bias.docx
Critical Analytical ThinkingPart II Heuristics and Bias.docx
 
Persuasion architectures: Nudging People to do the Right Thing
Persuasion architectures: Nudging People to do the Right ThingPersuasion architectures: Nudging People to do the Right Thing
Persuasion architectures: Nudging People to do the Right Thing
 

More from TechWell

Failing and Recovering
Failing and RecoveringFailing and Recovering
Failing and RecoveringTechWell
 
Instill a DevOps Testing Culture in Your Team and Organization
Instill a DevOps Testing Culture in Your Team and Organization Instill a DevOps Testing Culture in Your Team and Organization
Instill a DevOps Testing Culture in Your Team and Organization TechWell
 
Test Design for Fully Automated Build Architecture
Test Design for Fully Automated Build ArchitectureTest Design for Fully Automated Build Architecture
Test Design for Fully Automated Build ArchitectureTechWell
 
System-Level Test Automation: Ensuring a Good Start
System-Level Test Automation: Ensuring a Good StartSystem-Level Test Automation: Ensuring a Good Start
System-Level Test Automation: Ensuring a Good StartTechWell
 
Build Your Mobile App Quality and Test Strategy
Build Your Mobile App Quality and Test StrategyBuild Your Mobile App Quality and Test Strategy
Build Your Mobile App Quality and Test StrategyTechWell
 
Testing Transformation: The Art and Science for Success
Testing Transformation: The Art and Science for SuccessTesting Transformation: The Art and Science for Success
Testing Transformation: The Art and Science for SuccessTechWell
 
Implement BDD with Cucumber and SpecFlow
Implement BDD with Cucumber and SpecFlowImplement BDD with Cucumber and SpecFlow
Implement BDD with Cucumber and SpecFlowTechWell
 
Develop WebDriver Automated Tests—and Keep Your Sanity
Develop WebDriver Automated Tests—and Keep Your SanityDevelop WebDriver Automated Tests—and Keep Your Sanity
Develop WebDriver Automated Tests—and Keep Your SanityTechWell
 
Eliminate Cloud Waste with a Holistic DevOps Strategy
Eliminate Cloud Waste with a Holistic DevOps StrategyEliminate Cloud Waste with a Holistic DevOps Strategy
Eliminate Cloud Waste with a Holistic DevOps StrategyTechWell
 
Transform Test Organizations for the New World of DevOps
Transform Test Organizations for the New World of DevOpsTransform Test Organizations for the New World of DevOps
Transform Test Organizations for the New World of DevOpsTechWell
 
The Fourth Constraint in Project Delivery—Leadership
The Fourth Constraint in Project Delivery—LeadershipThe Fourth Constraint in Project Delivery—Leadership
The Fourth Constraint in Project Delivery—LeadershipTechWell
 
Resolve the Contradiction of Specialists within Agile Teams
Resolve the Contradiction of Specialists within Agile TeamsResolve the Contradiction of Specialists within Agile Teams
Resolve the Contradiction of Specialists within Agile TeamsTechWell
 
Pin the Tail on the Metric: A Field-Tested Agile Game
Pin the Tail on the Metric: A Field-Tested Agile GamePin the Tail on the Metric: A Field-Tested Agile Game
Pin the Tail on the Metric: A Field-Tested Agile GameTechWell
 
Agile Performance Holarchy (APH)—A Model for Scaling Agile Teams
Agile Performance Holarchy (APH)—A Model for Scaling Agile TeamsAgile Performance Holarchy (APH)—A Model for Scaling Agile Teams
Agile Performance Holarchy (APH)—A Model for Scaling Agile TeamsTechWell
 
A Business-First Approach to DevOps Implementation
A Business-First Approach to DevOps ImplementationA Business-First Approach to DevOps Implementation
A Business-First Approach to DevOps ImplementationTechWell
 
Databases in a Continuous Integration/Delivery Process
Databases in a Continuous Integration/Delivery ProcessDatabases in a Continuous Integration/Delivery Process
Databases in a Continuous Integration/Delivery ProcessTechWell
 
Mobile Testing: What—and What Not—to Automate
Mobile Testing: What—and What Not—to AutomateMobile Testing: What—and What Not—to Automate
Mobile Testing: What—and What Not—to AutomateTechWell
 
Cultural Intelligence: A Key Skill for Success
Cultural Intelligence: A Key Skill for SuccessCultural Intelligence: A Key Skill for Success
Cultural Intelligence: A Key Skill for SuccessTechWell
 
Turn the Lights On: A Power Utility Company's Agile Transformation
Turn the Lights On: A Power Utility Company's Agile TransformationTurn the Lights On: A Power Utility Company's Agile Transformation
Turn the Lights On: A Power Utility Company's Agile TransformationTechWell
 

More from TechWell (20)

Failing and Recovering
Failing and RecoveringFailing and Recovering
Failing and Recovering
 
Instill a DevOps Testing Culture in Your Team and Organization
Instill a DevOps Testing Culture in Your Team and Organization Instill a DevOps Testing Culture in Your Team and Organization
Instill a DevOps Testing Culture in Your Team and Organization
 
Test Design for Fully Automated Build Architecture
Test Design for Fully Automated Build ArchitectureTest Design for Fully Automated Build Architecture
Test Design for Fully Automated Build Architecture
 
System-Level Test Automation: Ensuring a Good Start
System-Level Test Automation: Ensuring a Good StartSystem-Level Test Automation: Ensuring a Good Start
System-Level Test Automation: Ensuring a Good Start
 
Build Your Mobile App Quality and Test Strategy
Build Your Mobile App Quality and Test StrategyBuild Your Mobile App Quality and Test Strategy
Build Your Mobile App Quality and Test Strategy
 
Testing Transformation: The Art and Science for Success
Testing Transformation: The Art and Science for SuccessTesting Transformation: The Art and Science for Success
Testing Transformation: The Art and Science for Success
 
Implement BDD with Cucumber and SpecFlow
Implement BDD with Cucumber and SpecFlowImplement BDD with Cucumber and SpecFlow
Implement BDD with Cucumber and SpecFlow
 
Develop WebDriver Automated Tests—and Keep Your Sanity
Develop WebDriver Automated Tests—and Keep Your SanityDevelop WebDriver Automated Tests—and Keep Your Sanity
Develop WebDriver Automated Tests—and Keep Your Sanity
 
Ma 15
Ma 15Ma 15
Ma 15
 
Eliminate Cloud Waste with a Holistic DevOps Strategy
Eliminate Cloud Waste with a Holistic DevOps StrategyEliminate Cloud Waste with a Holistic DevOps Strategy
Eliminate Cloud Waste with a Holistic DevOps Strategy
 
Transform Test Organizations for the New World of DevOps
Transform Test Organizations for the New World of DevOpsTransform Test Organizations for the New World of DevOps
Transform Test Organizations for the New World of DevOps
 
The Fourth Constraint in Project Delivery—Leadership
The Fourth Constraint in Project Delivery—LeadershipThe Fourth Constraint in Project Delivery—Leadership
The Fourth Constraint in Project Delivery—Leadership
 
Resolve the Contradiction of Specialists within Agile Teams
Resolve the Contradiction of Specialists within Agile TeamsResolve the Contradiction of Specialists within Agile Teams
Resolve the Contradiction of Specialists within Agile Teams
 
Pin the Tail on the Metric: A Field-Tested Agile Game
Pin the Tail on the Metric: A Field-Tested Agile GamePin the Tail on the Metric: A Field-Tested Agile Game
Pin the Tail on the Metric: A Field-Tested Agile Game
 
Agile Performance Holarchy (APH)—A Model for Scaling Agile Teams
Agile Performance Holarchy (APH)—A Model for Scaling Agile TeamsAgile Performance Holarchy (APH)—A Model for Scaling Agile Teams
Agile Performance Holarchy (APH)—A Model for Scaling Agile Teams
 
A Business-First Approach to DevOps Implementation
A Business-First Approach to DevOps ImplementationA Business-First Approach to DevOps Implementation
A Business-First Approach to DevOps Implementation
 
Databases in a Continuous Integration/Delivery Process
Databases in a Continuous Integration/Delivery ProcessDatabases in a Continuous Integration/Delivery Process
Databases in a Continuous Integration/Delivery Process
 
Mobile Testing: What—and What Not—to Automate
Mobile Testing: What—and What Not—to AutomateMobile Testing: What—and What Not—to Automate
Mobile Testing: What—and What Not—to Automate
 
Cultural Intelligence: A Key Skill for Success
Cultural Intelligence: A Key Skill for SuccessCultural Intelligence: A Key Skill for Success
Cultural Intelligence: A Key Skill for Success
 
Turn the Lights On: A Power Utility Company's Agile Transformation
Turn the Lights On: A Power Utility Company's Agile TransformationTurn the Lights On: A Power Utility Company's Agile Transformation
Turn the Lights On: A Power Utility Company's Agile Transformation
 

Recently uploaded

Large Language Models for Test Case Evolution and Repair
Large Language Models for Test Case Evolution and RepairLarge Language Models for Test Case Evolution and Repair
Large Language Models for Test Case Evolution and RepairLionel Briand
 
SensoDat: Simulation-based Sensor Dataset of Self-driving Cars
SensoDat: Simulation-based Sensor Dataset of Self-driving CarsSensoDat: Simulation-based Sensor Dataset of Self-driving Cars
SensoDat: Simulation-based Sensor Dataset of Self-driving CarsChristian Birchler
 
Precise and Complete Requirements? An Elusive Goal
Precise and Complete Requirements? An Elusive GoalPrecise and Complete Requirements? An Elusive Goal
Precise and Complete Requirements? An Elusive GoalLionel Briand
 
What is Advanced Excel and what are some best practices for designing and cre...
What is Advanced Excel and what are some best practices for designing and cre...What is Advanced Excel and what are some best practices for designing and cre...
What is Advanced Excel and what are some best practices for designing and cre...Technogeeks
 
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...confluent
 
英国UN学位证,北安普顿大学毕业证书1:1制作
英国UN学位证,北安普顿大学毕业证书1:1制作英国UN学位证,北安普顿大学毕业证书1:1制作
英国UN学位证,北安普顿大学毕业证书1:1制作qr0udbr0
 
Call Us🔝>༒+91-9711147426⇛Call In girls karol bagh (Delhi)
Call Us🔝>༒+91-9711147426⇛Call In girls karol bagh (Delhi)Call Us🔝>༒+91-9711147426⇛Call In girls karol bagh (Delhi)
Call Us🔝>༒+91-9711147426⇛Call In girls karol bagh (Delhi)jennyeacort
 
Understanding Flamingo - DeepMind's VLM Architecture
Understanding Flamingo - DeepMind's VLM ArchitectureUnderstanding Flamingo - DeepMind's VLM Architecture
Understanding Flamingo - DeepMind's VLM Architecturerahul_net
 
20240415 [Container Plumbing Days] Usernetes Gen2 - Kubernetes in Rootless Do...
20240415 [Container Plumbing Days] Usernetes Gen2 - Kubernetes in Rootless Do...20240415 [Container Plumbing Days] Usernetes Gen2 - Kubernetes in Rootless Do...
20240415 [Container Plumbing Days] Usernetes Gen2 - Kubernetes in Rootless Do...Akihiro Suda
 
1C_PNS.pdf Philippines National standard
1C_PNS.pdf Philippines National standard1C_PNS.pdf Philippines National standard
1C_PNS.pdf Philippines National standardraffietividad53
 
Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...
Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...
Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...Angel Borroy López
 
Powering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data StreamsPowering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data StreamsSafe Software
 
VK Business Profile - provides IT solutions and Web Development
VK Business Profile - provides IT solutions and Web DevelopmentVK Business Profile - provides IT solutions and Web Development
VK Business Profile - provides IT solutions and Web Developmentvyaparkranti
 
How to submit a standout Adobe Champion Application
How to submit a standout Adobe Champion ApplicationHow to submit a standout Adobe Champion Application
How to submit a standout Adobe Champion ApplicationBradBedford3
 
Odoo 14 - eLearning Module In Odoo 14 Enterprise
Odoo 14 - eLearning Module In Odoo 14 EnterpriseOdoo 14 - eLearning Module In Odoo 14 Enterprise
Odoo 14 - eLearning Module In Odoo 14 Enterprisepreethippts
 
Sending Calendar Invites on SES and Calendarsnack.pdf
Sending Calendar Invites on SES and Calendarsnack.pdfSending Calendar Invites on SES and Calendarsnack.pdf
Sending Calendar Invites on SES and Calendarsnack.pdf31events.com
 
SuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte Germany
SuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte GermanySuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte Germany
SuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte GermanyChristoph Pohl
 
Machine Learning Software Engineering Patterns and Their Engineering
Machine Learning Software Engineering Patterns and Their EngineeringMachine Learning Software Engineering Patterns and Their Engineering
Machine Learning Software Engineering Patterns and Their EngineeringHironori Washizaki
 
Unveiling the Future: Sylius 2.0 New Features
Unveiling the Future: Sylius 2.0 New FeaturesUnveiling the Future: Sylius 2.0 New Features
Unveiling the Future: Sylius 2.0 New FeaturesŁukasz Chruściel
 
Tech Tuesday - Mastering Time Management Unlock the Power of OnePlan's Timesh...
Tech Tuesday - Mastering Time Management Unlock the Power of OnePlan's Timesh...Tech Tuesday - Mastering Time Management Unlock the Power of OnePlan's Timesh...
Tech Tuesday - Mastering Time Management Unlock the Power of OnePlan's Timesh...OnePlan Solutions
 

Recently uploaded (20)

Large Language Models for Test Case Evolution and Repair
Large Language Models for Test Case Evolution and RepairLarge Language Models for Test Case Evolution and Repair
Large Language Models for Test Case Evolution and Repair
 
SensoDat: Simulation-based Sensor Dataset of Self-driving Cars
SensoDat: Simulation-based Sensor Dataset of Self-driving CarsSensoDat: Simulation-based Sensor Dataset of Self-driving Cars
SensoDat: Simulation-based Sensor Dataset of Self-driving Cars
 
Precise and Complete Requirements? An Elusive Goal
Precise and Complete Requirements? An Elusive GoalPrecise and Complete Requirements? An Elusive Goal
Precise and Complete Requirements? An Elusive Goal
 
What is Advanced Excel and what are some best practices for designing and cre...
What is Advanced Excel and what are some best practices for designing and cre...What is Advanced Excel and what are some best practices for designing and cre...
What is Advanced Excel and what are some best practices for designing and cre...
 
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
 
英国UN学位证,北安普顿大学毕业证书1:1制作
英国UN学位证,北安普顿大学毕业证书1:1制作英国UN学位证,北安普顿大学毕业证书1:1制作
英国UN学位证,北安普顿大学毕业证书1:1制作
 
Call Us🔝>༒+91-9711147426⇛Call In girls karol bagh (Delhi)
Call Us🔝>༒+91-9711147426⇛Call In girls karol bagh (Delhi)Call Us🔝>༒+91-9711147426⇛Call In girls karol bagh (Delhi)
Call Us🔝>༒+91-9711147426⇛Call In girls karol bagh (Delhi)
 
Understanding Flamingo - DeepMind's VLM Architecture
Understanding Flamingo - DeepMind's VLM ArchitectureUnderstanding Flamingo - DeepMind's VLM Architecture
Understanding Flamingo - DeepMind's VLM Architecture
 
20240415 [Container Plumbing Days] Usernetes Gen2 - Kubernetes in Rootless Do...
20240415 [Container Plumbing Days] Usernetes Gen2 - Kubernetes in Rootless Do...20240415 [Container Plumbing Days] Usernetes Gen2 - Kubernetes in Rootless Do...
20240415 [Container Plumbing Days] Usernetes Gen2 - Kubernetes in Rootless Do...
 
1C_PNS.pdf Philippines National standard
1C_PNS.pdf Philippines National standard1C_PNS.pdf Philippines National standard
1C_PNS.pdf Philippines National standard
 
Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...
Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...
Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...
 
Powering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data StreamsPowering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data Streams
 
VK Business Profile - provides IT solutions and Web Development
VK Business Profile - provides IT solutions and Web DevelopmentVK Business Profile - provides IT solutions and Web Development
VK Business Profile - provides IT solutions and Web Development
 
How to submit a standout Adobe Champion Application
How to submit a standout Adobe Champion ApplicationHow to submit a standout Adobe Champion Application
How to submit a standout Adobe Champion Application
 
Odoo 14 - eLearning Module In Odoo 14 Enterprise
Odoo 14 - eLearning Module In Odoo 14 EnterpriseOdoo 14 - eLearning Module In Odoo 14 Enterprise
Odoo 14 - eLearning Module In Odoo 14 Enterprise
 
Sending Calendar Invites on SES and Calendarsnack.pdf
Sending Calendar Invites on SES and Calendarsnack.pdfSending Calendar Invites on SES and Calendarsnack.pdf
Sending Calendar Invites on SES and Calendarsnack.pdf
 
SuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte Germany
SuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte GermanySuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte Germany
SuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte Germany
 
Machine Learning Software Engineering Patterns and Their Engineering
Machine Learning Software Engineering Patterns and Their EngineeringMachine Learning Software Engineering Patterns and Their Engineering
Machine Learning Software Engineering Patterns and Their Engineering
 
Unveiling the Future: Sylius 2.0 New Features
Unveiling the Future: Sylius 2.0 New FeaturesUnveiling the Future: Sylius 2.0 New Features
Unveiling the Future: Sylius 2.0 New Features
 
Tech Tuesday - Mastering Time Management Unlock the Power of OnePlan's Timesh...
Tech Tuesday - Mastering Time Management Unlock the Power of OnePlan's Timesh...Tech Tuesday - Mastering Time Management Unlock the Power of OnePlan's Timesh...
Tech Tuesday - Mastering Time Management Unlock the Power of OnePlan's Timesh...
 

Experiments: The Good, the Bad, and the Beautiful

  • 3. 5/30/16 1 Experiments: the Good, the Bad, and the Beautiful Linda Rising www.lindarising.org linda@lindarising.org @RisingLinda A short survey How many are doing some flavor of Agile? How did your organization decide to do that? How many looked at the randomized, controlled studies that provided evidence that Agile was better than your current process?
  • 4. 5/30/16 2 Not much science n  The short history of software is not progress based on scientific experiments. n  Instead we jump on the latest bandwagon because we hear a good story. n  These are not even really case studies. n  The plural of anecdote is not data ☺! Aren’t we experimenting? n  “Experiment” for most organizations really means “try.” n  No clear hypothesis, just an idea to try. n  No randomization. Usually those who participate are enthusiastic believers. n  No control group, just the memory of the way things were. n  No analysis, perhaps some easy-to-measure attributes or good “feelings.”
  • 5. 5/30/16 3 How about A/B Tests? n  A/B tests could be scientific, but usually aren’t. n  No clear hypothesis, just a couple of different ways to do something. n  Results don’t build toward a theory or real learning. n  Results are context-sensitive and may not apply elsewhere. How about saying “trial”? We don’t have resources for one, let alone repeated, experiments that good science requires. Just “trying” is our best hope.
  • 7. 5/30/16 5 We’re natural scientists! Our educational system? Science education focuses on “what” to think about, that is, content, not “how” to think. Problems in school are solvable. Problems in the “real world” often have no solution. We’re taught to be linear thinkers—to follow pre-established procedures and plans—in a nonlinear world. It’s not always a joyous experience "!
  • 8. 5/30/16 6 Agile contributions Failure and learning are important. Shifting from following a checklist to stopping after a short iteration to get feedback. Agile can help move us to a somewhat more scientific approach. Each iteration can be framed as a small trial, an empirical, incremental approach. Why do trials? n  Answer: To “prove” something n  Problem: One trial “proves” little, especially if treatment is not randomized or controlled. n  “Once and done” doesn’t work. We need lots and lots of trials. The process never ends.
  • 9. 5/30/16 7 Even scientists are biased n  Drug trials are now “double-blind” because it was discovered that if researchers and doctors knew which patients were getting “real” treatment, that would change the outcome. n  Even scientists suffer from confirmation bias. I wouldn’t believe that – even if it were true! -- Anonymous reviewer of scientific paper n  We do not see things as they are—we see them as we are -- Talmudic saying What CAN we do? n  Many small, simple, fast, frugal trials. n  Vary contexts, number of participants, degree of enthusiasm, kind of project. n  Re-test. Keep learning.
  • 10. 5/30/16 8 Why Small? n  Huge experiments leave no room for failure. Use small, cheap trials that barely register if they don’t work out. n  Keep asking, “Can we make it even smaller?” Why Simple? n  Everyone feels safe to try something that might bring benefit. n  Encourage each other to try! n  Keep asking, “Can we make it even simpler?”
  • 11. 5/30/16 9 Why Cheap? n  Counter the sunk-cost fallacy. It’s surprising how little an investment it takes to get us to avoid “wasting” that effort. n  Keep asking, “Can we make it even cheaper?” Why Time-Box? n  Begin with the end in mind - Stephen Covey n  Encourages openness and a way around confirmation bias: “Let’s try having the stand-up at 10 am instead of 8 am for the NEXT TWO WEEKS and see if attendance is better and we get more done in the morning.”
  • 12. 5/30/16 10 Why Faster? Establish a standard of fast, frequent, and inexpensive experimentation. Assume that many of your experiments will fail. One of the most common phrases you’ll hear at Menlo is “Let’s run the experiment.” We are apt to say that at least once a day. We don’t count experiments and we don’t track success/failure rates, but if we did, we would look for success and failure rates to be about even. If the percentage of failures started dropping, we’d become concerned that fear had crept into the room and that people weren’t taking enough risks. Rich Sheridan, Co-founder and CEO Menlo Innovations Why do trials? n  Answer: To show “those people” n  Problem: Research shows that evidence/data is not convincing but only serves to bolster our own beliefs (confirmation bias)
  • 13. 5/30/16 11 Better ways to convince n  Include others in trials -- encourage sharing n  Use patterns from Fearless Change n  Involve Everyone – don’t hide – use the trial to draw others in – watch your language – more “us” and less “them” n  Hometown Story – share results and encourage others to tell their stories Why do trials? n  Answer: To find a solution to a problem. n  Problem: Intervention in a complex system tends to create unanticipated and often undesirable outcomes.
  • 14. 5/30/16 12 Change in a complex system n  Probe, Sense, Respond n  “On Understanding Software Agility: A Social Complexity Point Of View” Joseph Pelrine n  Stop looking for answers. n  Discover ideas for more trials. Uncertainty n  We don’t know and we don’t know what we don’t know (thank you, Donald Rumsfeld) n  Experiments involve risk, uncertainty, and failure—no wonder we don’t do them ☺! n  Prepare to be surprised ☺!
  • 15. 5/30/16 13 Alexander Fleming said, “That’s funny!” …and we weren’t even trying for that! “Thank God it’s Open Friday,” Corinna Baldauf at Sipgate
  • 16. 5/30/16 14 Why do trials? n  Answer: To convince management n  This makes sense! BUT have your ducks in a row. Presentations to management n  Easy to follow, easy to understand, easy to share, easy to believe. n  No extraneous complications or technical jargon. n  Good ideas can crash and burn because more thought wasn’t put into the presentation. n  Less emphasis on details. More on communicating strategic value. Less about failure. More about learning. n  Executives should ask, “Why haven’t we thought of this in that way before?” n  The Innovator’s Hypothesis, Michael Schrage
  • 17. 5/30/16 15 Benefits of trials You kill off the HiPPOs. (Highest Paid Person’s Opinion.) Testing is a sure way to get to the bottom of a decision without relying on anyone’s gut instinct. At Shutterstock, if a senior executive has an idea in a meeting, the response is simply “Let’s test it.” Wyatt Jenkins, VP of Product Confirmation bias Our tendency to search for, interpret, favor, and recall (yes, it affects memory) information to confirm our beliefs. Charles Darwin said that whenever he ran into something that contradicted a notion he cherished, he wrote down the new finding within 30 minutes. Otherwise his mind would start to work to reject the discordant information, much as the body rejects transplants. Man’s natural inclination is to cling to his beliefs, particularly if they are reinforced by recent experience. – Warren Buffet
  • 18. 5/30/16 16 Believing is seeing n  We begin our scientific study with the result we want to see and then the confirmation bias kicks in, so it's difficult to be objective about the experiment. n  Once we have a belief, we only see the information that will confirm that belief. We stop seeing what we don’t want to see. Cognitive dissonance n  It’s difficult for us to hold two disconfirming ideas at the same time. n  To truly test each hypothesis, we have to be open to showing that we might be wrong.
  • 19. 5/30/16 17 This bias is challenging! …because the scientific method is designed to create dissonance…one of the reasons science is so difficult—because scientists are humans, and scientists don’t like it when their predictions are disconfirmed. I wish for every student that something they deeply hold to be true is shown to be wrong. Once you’ve had that experience, then you get it; then you get what science is about. - Lawrence Krauss Some help for biases n  Talk out loud and use words like ‘rational,’ ‘scientific’ and ‘experiment.’ Say, “Most people want to overcome their biases.” n  Write -- on paper, white board, flip chart n  Diversity – include skeptics – listen to all contributors n  Be aware and alert for bias -- ask questions n  Slow down. Take a break. Get enough sleep.
  • 20. 5/30/16 18 Correlation is not causation Honolulu Heart Program: 8,004 men studied over 30 years, examined relationship between coffee intake and the incidence of Parkinson's. Men who drank the most coffee were least likely to get Parkinson's. Men who did not drink coffee were 5 times more likely to exhibit symptoms of Parkinson's than men who drank more than 28 oz of coffee each day. Now we know that there's a genetic connection between liking coffee and risk for Parkinson's. It's not that coffee prevents the disease. It's that not liking coffee means that you are at risk for the disease (no one is sure why). In a disagreement Instead of arguing and taking up valuable meeting time, ask… “What small, fast, frugal trial would help us answer this question?”
  • 21. 5/30/16 19 Other benefits If you are careful in framing your hypothesis and in designing your experiments, you will get better over time. This is another great result – the experimenters themselves and an environment of replacing argument and heated discussion with ‘experiments.’ A culture of experimentation ☺! Reading suggestions The WHY Axis, Uri Gneezy & John List Innovator’s Hypothesis, Michael Schrage Little Bets, Peter Sims Fearless Change and More Fearless Change, Manns & Rising Anything Dan Ariely has written Thinking Fast and Slow, Daniel Kahneman Mistakes Were Made (But Not by Me), Carol Tavris & Elliot Aronson Joy, Inc., Richard Sheridan
  • 22. 5/30/16 20 More reading suggestions Behavioral Insights Team, “Test, Learn, Adapt” https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/62529/TLA-1906126.pdf Try an experiment! n  Have fun! n  Think like a child! n  Embrace failure! n  Thanks for listening…