Model-based Testing: Today And Tomorrow


Published on

Keynote, ETSI Model-Based Testing User Conference. Tallinn, Estonia September 27, 2012.
High-level discussion of model-based testing and trends driving software/system reliability. Explains how emergent behavior in complex systems ("dragon kings") causes catastrophic failures. My Multi-dimensional testing strategy can reveal this hard to find bugs/failure modes, but this requires a better approach to model-based testing. Overview: Is software eating the world? Bugs, Black Swans, Dragon Kings. Multi-dimensional Testing. Challenges.

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Model-based Testing: Today And Tomorrow

  1. 1. MODEL-BASED TESTING:TODAY AND TOMORROWRobert V. BinderETSI Model-Based Testing User ConferenceTallinn, EstoniaSeptember 27, 2012© 2012, Robert V. Binder. All Rights Reserved
  2. 2. 2Overview• Is software eating the world?• Bugs• Black Swans• Dragon Kings• Multi-dimensional Testing• Challenges
  3. 3. 3“Software is Eating the World”-Marc Andreessen Wall Street Journal, Aug 20 2011
  4. 4. 4Internet Trends. Mary Meeker, D10 Conference. May 2012
  5. 5. 5Internet Trends. Mary Meeker, D10 Conference. May 2012
  6. 6. 6Internet Trends. Mary Meeker, D10 Conference. May 2012
  7. 7. 7Embedded and Connected 2015 Connected Device Landscape: The Expanding Computing Continuum. David Perlmutter, Intel. 2009
  8. 8. 8 IT DisruptionGartner Top Predictions for 2012: Control Slips Away, Daryl Plummer. March 2012
  9. 9. 9Internet Trends. Mary Meeker, D10 Conference. May 2012
  10. 10. 10Is Software really eating the world?Methods/Tools since Y2K• 2nd gen IDE• MBT, MDD• Aspect OP• Open Source• UML 2.0• Cont. Integration• Web 2.0• Agile• ISO 12027• LAMP Stack• Many protocols• HTML 5• iOS, Android• Big Data• Software Def Netwks Relative rank from ~2M data points
  11. 11. 11Is Software really eating the world? Engineering Change to Accelerate Product and Systems Development. Christopher Rommel, VDC Research. August 2012
  12. 12. 12 Hardware is eating the worldEverything Computes & Connects Software Methods and Tools • 10x to 100x • ?X Productivity • ten years • ?X Quality • 100x to 1000x • Last ten years? • twenty years • Last twenty years? • Cheaper, better, faster • Cheaper, better, faster • Disrupt or die • “Choose two”
  13. 13. 13Old bugs, new bugs• No fundamental or • End of the uni-stack transformative • Multi-stack, mobile, improvements in connected software • Inherently unstable• Bugs still as likely, • New failure modes more code What do you think will happen?
  14. 14. 14Old Bugs• Incorrect output • Bohr Bugs• Missing feature  Predictable, repeatable• Abnormal end • Mandel Bugs  Obscure, complex trigger• Missed deadline • Heisen Bugs  Disappears if you look The Jargon File
  15. 15. 15Bohr Bugs
  16. 16. 16Mandel Bugs
  17. 17. 17Heisen Bugs
  18. 18. 18New Bugs• Power Law Models• Black Swans• Dragon Kings
  19. 19. 19 Power Law Models• Many natural phenomena• “Fat Tails” Dragon-Kings, Black Swans and the Prediction of Crises. Didier Sornette
  20. 20. 20Reliability Growth The Lognormal Distribution of Software Failure Rates: Application to Software Reliability Growth Modeling. Robert E. Mullen
  21. 21. 21 Black Swans• Very rare• Surprise• Cannot predict• Power law outliers
  22. 22. 22Software Black Swans• Total eclipse • Once every 360 years• Multi-cycle race conditions in hard realtime systems
  23. 23. 23Dragon Kings• Extreme Outlier  Beyond fat tail• Latent Positive Feedback• “Phase Shift”  Earthquakes  Material Fractures  Epileptic Seizures  Market Crashes
  24. 24. 24Dragon KingsSornett calls these exceptional events dragonkings “to stress that we deal with a completelydifferent kind of animal, beyond the normal,whose extraordinary characteristics [have]profound significance.”
  25. 25. 25Latent Positive Feedback = Dragon King
  26. 26. 26New Failure Modes:Amazon Cloud Outage• Amazon web service Elastic • ELB reaction: try to allocate Compute Cloud crashes more larger servers. (power failure) • Backlog in the “control plane”• The Elastic Load Balancing results. (ELB) system frantically tries to assign workloads to • Demand from customers in available servers. unaffected availability zones continues• As Amazon’s cloud rebooted, “a large number of ELBs • System swamped and came up in a state which crashes again. triggered a bug we hadn’t seen before”
  27. 27. 27What kind of failure was this?• Why didn’t test find it?• What would we have to do to find these kind of bugs?
  28. 28. 28Hic SuntDracones
  29. 29. 29Here be Dragons Latent positive feedback  External disruptors  Malicious attack target Partial degradation  Provoked panic Low testability  Multi-stack  Field-infeasible  Uncontrollable inputs
  30. 30. 30Looking under the Model-based Lampost?Logic errors and incorrectassumptions are inverselyproportional to a path’sexecution probability. Boris Beizer• 9 in 10 MBT testers do functionality only• Climbs a tree, not the twigs• Typically no load test
  31. 31. 31Multi-Dimensional TestingTest suite interleaves three kinds of variationBehavior Interfaces• Realistic variation • Drive all interfaces• Markov event/data • Disrupt resources• Include • Fat finger Load • Abuse cases • Vary input rate wrt time • Sneak paths • Panics Not possible without full automation
  32. 32. 32 Interleave Behavior, Load, Interfaces 3500.00010000000 3000.0001000000 2500.000 100000 X 2000.000 10000 Events Per Second 1500.000 1000 1000.000 100 500.000 10 0.000 1 -5000 0 5000 10000 15000 20000 25000Use Case A B C D E F G H I J K L -500.000 Time (seconds) Simulator OracleX = Adapter Splainer Adapter Adapter Comparator Adapter Test Verdict Run Reports SUT
  33. 33. 33Hunting the Dragon King Common Black Dragon Bugs Swans Kings Fragile 1 0 0 Massively Manual 2 0 0 Test Driven Dev 3 1 0 Exploratory 3 1 0 Crowdsourced 2 0 0 Functional MBT 5 3 1Multi-dimensional MBT 5 4 3• 0 You must be joking • 3 One Sigma• 1 Write if you get work • 4 Michael Jordan free throw (82%)• 2 Even a broken clock is right… • 5 Sargent Preston
  34. 34. 34What have we learned?• Information Technology is eating the world• Common bug rates will not decline• More new systems will have dragon kings• Functional/model coverage MBT will not find dragon kings• Multi-dimensional MBT can find dragon kings• Opportunity for test engineering to lead
  35. 35. 35May you live in interesting times …Challenges for the MBT CommunityTool Providers MBT Test Engineers • Start shipping multi- • Learn to lead dimensional • Start hunting your solutions dragon kings
  36. 36. 36Thank you !
  37. 37. 37Content and Image SourcesFirst Generation of Re-Imagination Reliability GrowthAndroid Phone Adoption has ramped even faster R. E. Mullen, “The Lognormal Distribution of Software Failure Rates:Despite Tremendous Ramp so Far, Smartphone User Adoption… Application to Software Reliability Growth Modeling,” InternationalMagnitude of Upcoming Change Will be Stunning… Symposium on Software Reliability Engineering, 1998, p 134.Internet Trends. Mary Meeker, D10 Conference May 2012 Black Swans The Black Swan: the Impact of the Highly Improbable. NassimEmbedded and Connected Taleb.2015 Connected Device Landscape: The Expanding Continuum. David Perlmutter, Intel. 2009 image.php?image=8248&picture=black-swan&large=1 Software Black SwansIT Disruption Top Predictions for 2012: Control Slips Away, DarylPlummer. March 2012 Dragon Kings Software really eating the world?Engineering Change to Accelerate Product and Systems Latent Positive Feedback = Dragon KingDevelopment. Christopher Rommel, VDC Research. August Amazon OutageMandel Bug Hidden bugs that made Amazon Web Service outage worse. Cnet News, July 3, 2012. Bug made-amazon-web-service-outage-worse/ Hic Sunt Dracones Law ModelsDragon-Kings, Black Swans and the Prediction of Crises. Looking under the Model-based Lampost?Didier Sornette. International Journal of Terraspace Science and 2009