The Art of Complex System Testing


Published on

It’s one week after your product’s launch, and everyone is happy. After all, for the first time in years, your product development exceeded expectations. Coding was completed on time with very few defects. Suddenly, the report of a major usability and security flaw destroys the euphoria and sends everything into chaos. Unfortunately, this is not uncommon in our industry. So, how can we mitigate such things from happening? As he shares stories about the complex domain of product delivery, Ray Arell introduces a framework with associated emergent practices that enable you to better guide your product to success. He presents an overview of the Cynefin model, a description of complicated and complex systems, and discusses how to use it to establish an effective testing strategy. Ray describes how to identify key patterns of product usage to establish a robust defect-prevention system that reduces product development costs. Lastly, Ray describes how to interview customers to identify key quality expectations, ensuring that your testing focuses on producing the highest value for your customers.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

The Art of Complex System Testing

  1. 1.       e        Presented by:  Ray  rell  Brought to you by:      340 Corporate Way, Suite   Orange Park, FL 32073  888‐2 K3  Keynot 4/9/2014 8:30 AM          “The Art of Complex System Testing”      A Intel                    300, 68‐8770 ∙ 904‐278‐0524 ∙ ∙ 
  2. 2. Ray Arell Intel   As director of Intel Emergent Systems and Coaching, Ray Arell is a transformative leader in the adoption of agile, lean, and complex system methods inside Intel. Ray’s group is currently coaching a community of practice of more than ten-thousand people who are moving to a continuous value delivery culture. Prior to this role, he spent several decades as both an engineer and engineering manager of teams focused on CPU, chipset, graphics, wireless, and software development. Ray is a popular speaker at events worldwide and coauthor of Change-Based Test Management: Improving the Software Validation Process.  
  3. 3. 2/26/2014 1 THE ART OF COMPLEX SYSTEM THE ART OF COMPLEX SYSTEM TESTINGTESTING Ray Arell, Director  Intel Emergent Systems and Coaching  Copyright © 2014 Intel Corporation. All rights reserved. *Other names and brands may be claimed as the property of others.  
  4. 4. 2/26/2014 2 Source: http://en.wikipedia.o
  5. 5. 2/26/2014 3
  6. 6. 2/26/2014 4 Source: 
  7. 7. 2/26/2014 5 nts Many Chaotic Requireme Fe Complex Simple Complicated Technology Far from certainty Close to certainty Few KnowledgeHigh Low Simple Source: 
  8. 8. 2/26/2014 6
  9. 9. 2/26/2014 7 Source: You can't connect the dots looking  forward; you can only connect them  looking backwards—Steve Jobs
  10. 10. 2/26/2014 8 We manage the emergence of beneficial coherence within  attractors, within boundaries—Dave Snowden Voltage Frequency 
  11. 11. 2/26/2014 9 Source:
  12. 12. 2/26/2014 10 CYNEFIN FRAMEWORK Complicated C E Complex C E Dave Snowden Simple C E good practiceemergent practice Chaotic probe-sense-respond sense-analyze-respond C E Disorder Dave Snowden Disorder best practice Cause Effect novel practice act-sense-respond sense-categorize-respond C E Source: cognitive‐ 
  13. 13. 2/26/2014 11 Source:; with a fish m Complex Complicated c a Disorder b c SimpleChaos Source: Cynefin by David Snowden http://cognitive‐
  14. 14. 2/26/2014 12 D i d St t dge  Desired State Tyranny of the expert Knowle PROBEPROBE--SENSESENSE--RESPONDRESPOND emergent practiceemergent practice
  15. 15. 2/26/2014 13
  16. 16. 2/26/2014 14 Source: Ylvis; The Fox (What Does The Fox Say?) “It takes a thousand voices to tell a single story” - Native American proverb Source: Competition among memes in a world with limited attention; L. Weng, A. Flammini, A. Vespignani & F. Menczer
  17. 17. 2/26/2014 15 Prompting Question Story Capture  Act on Signals Stimulate/Dampen  Patterns Self‐Signification Discussion Visualization Deepen Insight Source: cognitive‐  SELF‐SIGNIFICATION IS THE KEY Poor Performance Exciting  This story is about:  The product talked about is:  l b l b l g d bl N/AN/A Poor Reliability Poor Usability Quality Expected Desirable Qualities  More info: cognitive‐ 
  18. 18. 2/26/2014 16 Poor Performance N/A Poor Reliability Poor Usability Quality More info: cognitive‐ 
  19. 19. 2/26/2014 17 EXPANDING OUR TEST METHODS TO ADDRESS  THE COMPLEX SPACE  Sources: right; left Paul Mathews  Complicated  Complexto C E C E FINAL THOUGHTS • We need to start to develop new test methods  to address complex products • Testing the rules is fine but it needs to be with  real world complex stimulus  • High satisfaction can only be understood if we  listen closer to the stories our customers are  telling
  20. 20. 2/26/2014 18 ABSTRACT It’s one week after your product’s launch, and everyone is happy. After all, for the  first time in years your product development exceeded expectations Coding wasfirst time in years, your product development exceeded expectations. Coding was  completed on time with very few defects. Suddenly, the report of a major usability  and security flaw destroys the euphoria and sends everything into chaos.  Unfortunately, this is not uncommon in our industry. So, how can we mitigate such  things from happening? As he shares stories about the complex domain of product  delivery, Ray Arell introduces a framework with associated emergent practices that  enable you to better guide your product to success. He presents an overview of  the Cynefin model, a description of complicated and complex systems, and  discusses how to use it to establish an effective testing strategy. Ray describes how  to identify key patterns of product usage to establish a robust defect‐prevention  system that reduces product development costs. Lastly, Ray describes how to  interview customers to identify key quality expectations, ensuring that your testing  focuses on producing the highest value for your customers.
  21. 21. 2/26/2014 19 As director of Intel Emergent Systems and Coaching, Ray Arell is a g y g, y transformative leader in the adoption of agile, lean, and complex  system methods inside Intel. Ray’s group is currently coaching a  community of practice of more than ten‐thousand people who are  moving to a continuous value delivery culture. Prior to this role, he  spent several decades as both an engineer and engineering  manager of teams focused on CPU, chipset, graphics, wireless, and  software development. Ray is a popular speaker at events  worldwide and coauthor of Change‐Based Test Management:  Improving the Software Validation Process.