Model-based Testing: Today And Tomorrow

  • 479 views
Uploaded on

Keynote, ETSI Model-Based Testing User Conference. Tallinn, Estonia September 27, 2012. …

Keynote, ETSI Model-Based Testing User Conference. Tallinn, Estonia September 27, 2012.
High-level discussion of model-based testing and trends driving software/system reliability. Explains how emergent behavior in complex systems ("dragon kings") causes catastrophic failures. My Multi-dimensional testing strategy can reveal this hard to find bugs/failure modes, but this requires a better approach to model-based testing. Overview: Is software eating the world? Bugs, Black Swans, Dragon Kings. Multi-dimensional Testing. Challenges.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
479
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
15
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. MODEL-BASED TESTING:TODAY AND TOMORROWRobert V. BinderETSI Model-Based Testing User ConferenceTallinn, EstoniaSeptember 27, 2012© 2012, Robert V. Binder. All Rights Reserved
  • 2. 2Overview• Is software eating the world?• Bugs• Black Swans• Dragon Kings• Multi-dimensional Testing• Challenges
  • 3. 3“Software is Eating the World”-Marc Andreessen Wall Street Journal, Aug 20 2011
  • 4. 4Internet Trends. Mary Meeker, D10 Conference. May 2012
  • 5. 5Internet Trends. Mary Meeker, D10 Conference. May 2012
  • 6. 6Internet Trends. Mary Meeker, D10 Conference. May 2012
  • 7. 7Embedded and Connected 2015 Connected Device Landscape: The Expanding Computing Continuum. David Perlmutter, Intel. 2009
  • 8. 8 IT DisruptionGartner Top Predictions for 2012: Control Slips Away, Daryl Plummer. March 2012
  • 9. 9Internet Trends. Mary Meeker, D10 Conference. May 2012
  • 10. 10Is Software really eating the world?Methods/Tools since Y2K• 2nd gen IDE• MBT, MDD• Aspect OP• Open Source• UML 2.0• Cont. Integration• Web 2.0• Agile• ISO 12027• LAMP Stack• Many protocols• HTML 5• iOS, Android• Big Data http://langpop.com/• Software Def Netwks Relative rank from ~2M data points
  • 11. 11Is Software really eating the world? Engineering Change to Accelerate Product and Systems Development. Christopher Rommel, VDC Research. August 2012
  • 12. 12 Hardware is eating the worldEverything Computes & Connects Software Methods and Tools • 10x to 100x • ?X Productivity • ten years • ?X Quality • 100x to 1000x • Last ten years? • twenty years • Last twenty years? • Cheaper, better, faster • Cheaper, better, faster • Disrupt or die • “Choose two”
  • 13. 13Old bugs, new bugs• No fundamental or • End of the uni-stack transformative • Multi-stack, mobile, improvements in connected software • Inherently unstable• Bugs still as likely, • New failure modes more code What do you think will happen?
  • 14. 14Old Bugs• Incorrect output • Bohr Bugs• Missing feature  Predictable, repeatable• Abnormal end • Mandel Bugs  Obscure, complex trigger• Missed deadline • Heisen Bugs  Disappears if you look The Jargon File http://www.catb.org/jargon
  • 15. 15Bohr Bugs
  • 16. 16Mandel Bugs
  • 17. 17Heisen Bugs
  • 18. 18New Bugs• Power Law Models• Black Swans• Dragon Kings
  • 19. 19 Power Law Models• Many natural phenomena• “Fat Tails” Dragon-Kings, Black Swans and the Prediction of Crises. Didier Sornette http://www.youtube.com/watch?v=FlTSbzOvKZI
  • 20. 20Reliability Growth The Lognormal Distribution of Software Failure Rates: Application to Software Reliability Growth Modeling. Robert E. Mullen
  • 21. 21 Black Swans• Very rare• Surprise• Cannot predict• Power law outliers
  • 22. 22Software Black Swans• Total eclipse • Once every 360 years• Multi-cycle race conditions in hard realtime systems
  • 23. 23Dragon Kings• Extreme Outlier  Beyond fat tail• Latent Positive Feedback• “Phase Shift”  Earthquakes  Material Fractures  Epileptic Seizures  Market Crashes
  • 24. 24Dragon KingsSornett calls these exceptional events dragonkings “to stress that we deal with a completelydifferent kind of animal, beyond the normal,whose extraordinary characteristics [have]profound significance.”
  • 25. 25Latent Positive Feedback = Dragon King
  • 26. 26New Failure Modes:Amazon Cloud Outage• Amazon web service Elastic • ELB reaction: try to allocate Compute Cloud crashes more larger servers. (power failure) • Backlog in the “control plane”• The Elastic Load Balancing results. (ELB) system frantically tries to assign workloads to • Demand from customers in available servers. unaffected availability zones continues• As Amazon’s cloud rebooted, “a large number of ELBs • System swamped and came up in a state which crashes again. triggered a bug we hadn’t seen before”
  • 27. 27What kind of failure was this?• Why didn’t test find it?• What would we have to do to find these kind of bugs?
  • 28. 28Hic SuntDracones
  • 29. 29Here be Dragons Latent positive feedback  External disruptors  Malicious attack target Partial degradation  Provoked panic Low testability  Multi-stack  Field-infeasible  Uncontrollable inputs
  • 30. 30Looking under the Model-based Lampost?Logic errors and incorrectassumptions are inverselyproportional to a path’sexecution probability. Boris Beizer• 9 in 10 MBT testers do functionality only• Climbs a tree, not the twigs• Typically no load test
  • 31. 31Multi-Dimensional TestingTest suite interleaves three kinds of variationBehavior Interfaces• Realistic variation • Drive all interfaces• Markov event/data • Disrupt resources• Include • Fat finger Load • Abuse cases • Vary input rate wrt time • Sneak paths • Panics Not possible without full automation
  • 32. 32 Interleave Behavior, Load, Interfaces 3500.00010000000 3000.0001000000 2500.000 100000 X 2000.000 10000 Events Per Second 1500.000 1000 1000.000 100 500.000 10 0.000 1 -5000 0 5000 10000 15000 20000 25000Use Case A B C D E F G H I J K L -500.000 Time (seconds) Simulator OracleX = Adapter Splainer Adapter Adapter Comparator Adapter Test Verdict Run Reports SUT
  • 33. 33Hunting the Dragon King Common Black Dragon Bugs Swans Kings Fragile 1 0 0 Massively Manual 2 0 0 Test Driven Dev 3 1 0 Exploratory 3 1 0 Crowdsourced 2 0 0 Functional MBT 5 3 1Multi-dimensional MBT 5 4 3• 0 You must be joking • 3 One Sigma• 1 Write if you get work • 4 Michael Jordan free throw (82%)• 2 Even a broken clock is right… • 5 Sargent Preston
  • 34. 34What have we learned?• Information Technology is eating the world• Common bug rates will not decline• More new systems will have dragon kings• Functional/model coverage MBT will not find dragon kings• Multi-dimensional MBT can find dragon kings• Opportunity for test engineering to lead
  • 35. 35May you live in interesting times …Challenges for the MBT CommunityTool Providers MBT Test Engineers • Start shipping multi- • Learn to lead dimensional • Start hunting your solutions dragon kings
  • 36. 36Thank you !rvbinder@gmail.comwww.robertvbinder.com
  • 37. 37Content and Image SourcesFirst Generation of Re-Imagination Reliability GrowthAndroid Phone Adoption has ramped even faster R. E. Mullen, “The Lognormal Distribution of Software Failure Rates:Despite Tremendous Ramp so Far, Smartphone User Adoption… Application to Software Reliability Growth Modeling,” InternationalMagnitude of Upcoming Change Will be Stunning… Symposium on Software Reliability Engineering, 1998, p 134.Internet Trends. Mary Meeker, D10 Conference May 2012http://kpcb.com/insights/2012-internet-trends Black Swans The Black Swan: the Impact of the Highly Improbable. NassimEmbedded and Connected Taleb.2015 Connected Device Landscape: The Expanding http://www.publicdomainpictures.net/view-Computing Continuum. David Perlmutter, Intel. 2009 image.php?image=8248&picture=black-swan&large=1http://www.intel.com/pressroom/kits/events/idffall_2009/video.htm Software Black SwansIT Disruption http://apod.nasa.gov/apod/ap060404.htmlGartner Top Predictions for 2012: Control Slips Away, DarylPlummer. March 2012 Dragon Kings http://wallpaper4me.com/wallpaper/Dragon-King-of-The-Land/Is Software really eating the world?Engineering Change to Accelerate Product and Systems Latent Positive Feedback = Dragon KingDevelopment. Christopher Rommel, VDC Research. August http://www.everythingselectric.com/forum/index.php?topic=244.02012 Amazon OutageMandel Bug Hidden bugs that made Amazon Web Service outage worse. Cnethttp://www.incamouflage.com/2009/05/bug-camouflage/ News, July 3, 2012. http://news.cnet.com/8301-1009_3-57465761-83/hidden-bugs-that-Heisen Bug made-amazon-web-service-outage-worse/http://www.adriennehedger.com/2009/07/june-bug-situation.htmlhttp://openclipart.org/detail/145735/cartoon-fly-by-printerkiller Hic Sunt Dracones http://tomlytle.com/Power Law ModelsDragon-Kings, Black Swans and the Prediction of Crises. Looking under the Model-based Lampost?Didier Sornette. International Journal of Terraspace Science and http://stconsultant.blogspot.com/2010/11/old-saying.htmlEngineering 2009http://www.youtube.com/watch?v=FlTSbzOvKZI