It’s one week after your product’s launch, and everyone is happy. After all, for the first time in years, your product development exceeded expectations. Coding was completed on time with very few defects. Suddenly, the report of a major usability and security flaw destroys the euphoria and sends everything into chaos. Unfortunately, this is not uncommon in our industry. So, how can we mitigate such things from happening? As he shares stories about the complex domain of product delivery, Ray Arell introduces a framework with associated emergent practices that enable you to better guide your product to success. He presents an overview of the Cynefin model, a description of complicated and complex systems, and discusses how to use it to establish an effective testing strategy. Ray describes how to identify key patterns of product usage to establish a robust defect-prevention system that reduces product development costs. Lastly, Ray describes how to interview customers to identify key quality expectations, ensuring that your testing focuses on producing the highest value for your customers.

1.
e
Presented by:
Ray rell
Brought to you by:
340 Corporate Way, Suite Orange Park, FL 32073
888‐2
K3
Keynot
4/9/2014
8:30 AM
“The Art of Complex System Testing”
A
Intel
300,
68‐8770 ∙ 904‐278‐0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

2. Ray Arell
Intel
As director of Intel Emergent Systems and Coaching, Ray Arell is a transformative
leader in the adoption of agile, lean, and complex system methods inside Intel. Ray’s
group is currently coaching a community of practice of more than ten-thousand
people who are moving to a continuous value delivery culture. Prior to this role, he
spent several decades as both an engineer and engineering manager of teams focused
on CPU, chipset, graphics, wireless, and software development. Ray is a popular
speaker at events worldwide and coauthor of Change-Based Test Management:
Improving the Software Validation Process.

3. 2/26/2014
1
THE ART OF COMPLEX SYSTEM THE ART OF COMPLEX SYSTEM TESTINGTESTING
Ray Arell, Director
Intel Emergent Systems and Coaching
Copyright © 2014 Intel Corporation. All rights reserved. *Other names and brands may be claimed as the property of others.

4. 2/26/2014
2
Source: http://en.wikipedia.o

5. 2/26/2014
3

6. 2/26/2014
4
Source: Flickr.com

7. 2/26/2014
5
nts
Many
Chaotic
Requireme
Fe
Complex
Simple
Complicated
Technology Far from
certainty
Close to
certainty
Few
KnowledgeHigh Low
Simple
Source: Flickr.com

8. 2/26/2014
6

9. 2/26/2014
7
Source: http://www.calresco.org/lucas/cas.htm
You can't connect the dots looking
forward; you can only connect them
looking backwards—Steve Jobs

10. 2/26/2014
8
We manage the emergence of beneficial coherence within
attractors, within boundaries—Dave Snowden
Voltage
Frequency

11. 2/26/2014
9
Source: http://www.mdofpc.com

12. 2/26/2014
10
CYNEFIN FRAMEWORK
Complicated
C E
Complex
C E Dave Snowden
Simple
C E
good practiceemergent practice
Chaotic
probe-sense-respond sense-analyze-respond
C E
Disorder
Dave Snowden
Disorder
best practice
Cause Effect
novel practice
act-sense-respond sense-categorize-respond
C E
Source: cognitive‐edge.com

13. 2/26/2014
11
Source: www.thephonetown.com; with a fish m
Complex Complicated
c
a
Disorder
b
c
SimpleChaos
Source: Cynefin by David Snowden
http://cognitive‐edge.com/

14. 2/26/2014
12
D i d St t
dge
Desired State
Tyranny of the
expert
Knowle
PROBEPROBE--SENSESENSE--RESPONDRESPOND
emergent practiceemergent practice

15. 2/26/2014
13

16. 2/26/2014
14
Source: Ylvis; The Fox (What Does The Fox Say?)
“It takes a thousand voices to tell a single story”
- Native American proverb
Source: Competition among memes in a world with limited attention; L. Weng, A. Flammini, A. Vespignani & F. Menczer

17. 2/26/2014
15
Prompting Question
Story Capture
Act on Signals
Stimulate/Dampen
Patterns
Self‐Signification
Discussion Visualization
Deepen Insight
Source: cognitive‐edge.com
SELF‐SIGNIFICATION IS THE KEY
Poor Performance Exciting
This story is about: The product talked about is:
l b l b l
g
d bl
N/AN/A
Poor Reliability Poor Usability
Quality
Expected Desirable
Qualities
More info: cognitive‐edge.com

18. 2/26/2014
16
Poor Performance
N/A
Poor Reliability Poor Usability
Quality
More info: cognitive‐edge.com

19. 2/26/2014
17
EXPANDING OUR TEST METHODS TO ADDRESS
THE COMPLEX SPACE
Sources: right http://www.ece.lsu.edu; left Paul Mathews
Complicated Complexto
C E C E
FINAL THOUGHTS
• We need to start to develop new test methods
to address complex products
• Testing the rules is fine but it needs to be with
real world complex stimulus
• High satisfaction can only be understood if we
listen closer to the stories our customers are
telling

20. 2/26/2014
18
ABSTRACT
It’s one week after your product’s launch, and everyone is happy. After all, for the
first time in years your product development exceeded expectations Coding wasfirst time in years, your product development exceeded expectations. Coding was
completed on time with very few defects. Suddenly, the report of a major usability
and security flaw destroys the euphoria and sends everything into chaos.
Unfortunately, this is not uncommon in our industry. So, how can we mitigate such
things from happening? As he shares stories about the complex domain of product
delivery, Ray Arell introduces a framework with associated emergent practices that
enable you to better guide your product to success. He presents an overview of
the Cynefin model, a description of complicated and complex systems, and
discusses how to use it to establish an effective testing strategy. Ray describes how
to identify key patterns of product usage to establish a robust defect‐prevention
system that reduces product development costs. Lastly, Ray describes how to
interview customers to identify key quality expectations, ensuring that your testing
focuses on producing the highest value for your customers.

21. 2/26/2014
19
As director of Intel Emergent Systems and Coaching, Ray Arell is a g y g, y
transformative leader in the adoption of agile, lean, and complex
system methods inside Intel. Ray’s group is currently coaching a
community of practice of more than ten‐thousand people who are
moving to a continuous value delivery culture. Prior to this role, he
spent several decades as both an engineer and engineering
manager of teams focused on CPU, chipset, graphics, wireless, and
software development. Ray is a popular speaker at events
worldwide and coauthor of Change‐Based Test Management:
Improving the Software Validation Process.