Mobile Reliability Challenges

386 views

Published on

Keynote, ISSRE-13, St. Malo, France, November 4, 2004.
Outline: 21st Century IT Trends, Mobile Technology Crisis, Test Effectiveness Levels, Level 4 Case Study, Reliability Arithmetic, Test Performance Envelope.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
386
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
8
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Mobile Reliability Challenges

  1. 1. ™ mVerify A Million Users in a Box ®Mobile Reliability Challenges Robert V. Binder ISSRE 2004 November 2, 2004 www.mVerify.com
  2. 2. Overview 21st Century IT Trends Mobile Technology Crisis Test Effectiveness Levels Level 4 Case Study Reliability Arithmetic Test Performance Envelope Conclusion © 2004 mVerify Corporation 2
  3. 3. The New IT Reality Last 25 years  Next 25 years  Information driven society  User population 100x  New and better ways to do  New things old things  Persistent partial attention  Focused, skilled interaction  Mobile, ubiquitous  Tethered Very low visibility, very high failure impact © 2004 mVerify Corporation 3
  4. 4. The New IT Reality: Ubiquity Cheap fat pipes everywhere  Optical backbone + wireless MAN, LAN, PAN  WiMax: 802.16, 802.20  Ad-hoc Mobile networks Cycles and storage  Moores law  Watts per MIPS No more shrink-wrap  Application service provider/subscription model  Web services, ultra large databases  Grid computing Converging user device form factor  Cell phone + PDA + pager + Pocket PC + ...  D-2-B Interfaces – 5 years? About 10x every five years! © 2004 mVerify Corporation 4
  5. 5. Robert’s Afternoon Robert’s Afternoon: Seamless Mobility courtesy Motorola, Inc. This shows Motorola’s high-level vision.It is not a product roadmap or indicative of any specific product/service offering. © 2004 mVerify Corporation 5
  6. 6. The Unchanged IT Reality: Software The Next Big Thing  Extreme Programming ?  Aspect-oriented languages ?  Model-driven Architecture ? Still no Silver Bullet  Subtractive component reliability  Design limited to human ability and organization  Low-fidelity test suites aren’t effective Bug barrier: 5/KLOC (pre-test) any language, any process © 2004 mVerify Corporation 6
  7. 7. Mobile Technology Challenges Testing wired apps difficult and expensive  20% to 50% of all software development $ on testing  Available test automation technology 10+ years old  Annual cost to U.S. of inadequate testing: $56 billion Testing mobile apps much harder:  Connectivity: “Can you hear me now?”  Mobility: location-based services  Scalability: at least 10x web user population and mobile  Security: always on, always hackable  PLUS assure functionality, performance, and integration Mobile App Fault Space Much Bigger © 2004 mVerify Corporation 7
  8. 8. A True CrisisCurrent software technology CANNOTachieve reliable mobile apps © 2004 mVerify Corporation 8
  9. 9. What Can Testing Do? Test Performance  Effectiveness (reliability/quality increase)  Efficiency (average cost per test) Levels  1: Testing by poking around  2: Manual Testing  3: Automated Test Script  4: Model-based  5: Full Test Automation Each Level 10x Improvement © 2004 mVerify Corporation 9
  10. 10. Level 1: Testing by Poking Around Manual “Exploratory” Testing•Low Coverage•Not Repeatable•Can’t Scale•Inconsistent System Under Test © 2004 mVerify Corporation 10
  11. 11. Level 2: Manual Testing Test SetupManual ManualTest Design/ Test InputGeneration •1 test per hour •Not repeatable Test Results System Under Test Evaluation © 2004 mVerify Corporation 11
  12. 12. Level 3: Automated Test Script Test SetupManual Test ScriptTest Design/ ProgrammingGeneration •10+ tests per hour •Repeatable •High change cost Test Results System Under Test Evaluation © 2004 mVerify Corporation 12
  13. 13. Level 4: Automated Model-based Test Setup Model-based Test Design/ Automatic Generation Test Execution•1000+ tests per hour•High fidelity•Case study Test Results System Under Test Evaluation © 2004 mVerify Corporation 13
  14. 14. Level 4 Case Study Leading financial market  3 million transactions per hour  15 billion dollars per day 650 KLOC Java, Distributed Services … System Test Process & Environment  Automated, Model-based  Executable operational profile  Simulator generates realistic unique test suites 3 years, version 1.0 live Q4 2001 1,000 to 750,000 unique tests per day © 2004 mVerify Corporation 14
  15. 15. Model-based TestingExtended Use CaseMode MachineInvariant Boundaries Stealth Requirements Engineering © 2004 mVerify Corporation 15
  16. 16. Simulator Discrete event simulation Prolog implementation (50 KLOC)  Rule inversion Load Profile  Time domain variation  Orthogonal to operational profile Each event assigned a "port" and submit time © 2004 mVerify Corporation 16
  17. 17. Automated Run Evaluation Oracle accepts output of simulator About 500 unique rules Verification  Splainer – result/rule backtracking tool  Rule/Run coverage analyzer Comparator  Extract transaction log  Post run database state  End-to-end invariant © 2004 mVerify Corporation 17
  18. 18. Results Revealed about 1,500 bugs over two years  5% showstoppers Five person team, huge productivity increase Achieved proven high reliability  Last pre-release test run: 500,000 events in two hours, no failures detected  No production failures © 2004 mVerify Corporation 18
  19. 19. Level 5: Total Automation Automated Test SetupModel-based AutomaticTest Design/ TestGeneration Execution•10,000 TPH•Oracle Problem Automated System Under Test Test Results Evaluation © 2004 mVerify Corporation 19
  20. 20. AMATE: Level 5 for Mobile Apps Advanced Mobile Application Test Environment NIST/ATP funded R&D Highly realistic end-to-end mobile testing  Generate and control  Signal variation related to mobility  User behavior related to mobility  Traffic related to mobility  Model-based Trial Use Q2 2005 © 2004 mVerify Corporation 20
  21. 21. Reliability Arithmetic Reliability: probability of non-failure MTTR: mean time to  recover, repair, restart … Availability: percent up-time  Availability = 1 / 1 + (MTTR  Reliability)  99.999% availability = 5 min downtime per year  “Five nines” © 2004 mVerify Corporation 21
  22. 22. Some Reliability Data Points Reliability Availability, (Failures/million hours) 6 min MTTRNT 4.0 Desktop 82,000 0.999000000Windows 2K Server 36,013 0.999640000Common Light Bulb 1,000 0.999990000Stepstone OO Framework 5 0.999999500Telelabs Digital Cross Connect 3 0.999999842 © 2004 mVerify Corporation 22
  23. 23. Test Automation EnvelopeReliability (Effectiveness) 5 Nines L4 Level 4 Nines Case 5 Study AMATE 3 Nines L3 2 Nines Scripting L2 1 Nine Manual 1 10 100 1,000 10,000 Productivity: Tests/Hour (Efficiency) © 2004 mVerify Corporation 23
  24. 24. The Mobile Reliability Challenge Mobile App Fault Space Bigger  Connectivity: “Can you hear me now?”  Mobility: location-based services  Scalability: large and mobile  Security: always on, always hackable  PLUS assure functionality, performance, and integration What can be done?  Test performance envelope the same  Test budget the same © 2004 mVerify Corporation 24
  25. 25. Scenario: Manual Testing Level 2 manual 5 9s testing 4 9s Mobile app 3 9s fault space 2 9s 10x bigger 1 9s Reliability slips 1 10 100 1,000 10,000 to Level 1 © 2004 mVerify Corporation 26
  26. 26. Scenario: Improve Efficiency 10x L2 Manual improves 10x to 5 9s L3 Scripted 4 9s L3 Scripted 3 9s improves 10x to L4 Model-based 2 9s Expect same 1 9s average reliability 1 10 100 1,000 10,000 © 2004 mVerify Corporation 27
  27. 27. Scenario: Mobile 5 Nines Increase L4 5 9s efficiency 10x  Realistic mobile environment 4 9s  Realistic loading  Realistic 3 9s functional profile  Model-based and automated 2 9s AMATE 1 9s 1 10 100 1,000 10,000 Expect mobile app 5 Nines © 2004 mVerify Corporation 28
  28. 28. ConclusionModel-Based mobile testing CANachieve reliable mobile apps © 2004 mVerify Corporation 29
  29. 29. Q&A © 2004 mVerify Corporation 30

×