Achieving Very High Reliability for Ubiquitous Information Technology

436 views

Published on

Invited talk, November 26, 2001. Center for Embedded Software Systems. Aalborg, Denmark. Overview of reliability challenges in mobile systems.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
436
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Achieving Very High Reliability for Ubiquitous Information Technology

  1. 1. Achieving Very High Reliability forUbiquitous Information Technology Center for Embedded Software Systems Aalborg, Denmark November 26, 2001 Robert V. Binder Mobile Systems Verification Corporation www.MobileSystemsVerification.com
  2. 2. Overview• Vision of very high IT reliability• The new IT reality and its challenges• Strategy for very high reliability © 2001 Mobile Systems Verification Corporation 2
  3. 3. The Vision• Test engineering = system engineering• Testing is fully automated, high-fidelity• Very high reliability no longer exotic © 2001 Mobile Systems Verification Corporation 3
  4. 4. Reliability Arithmetic• Reliability: probability of non-failure• MTTR: mean time to – recover, repair, restart …• Availability: percent up-time – Availability = 1 / 1 + (MTTR  Reliability) – 99.999% availability = 5 min per year – “Five nines” © 2001 Mobile Systems Verification Corporation 4
  5. 5. Some Reliability Data Points Reliability, Failures per Availability, million hours 6 min MTTRNT 4.0 Desktop 82000.0000 0.999000000Windows 2000 Server Family 36012.9647 0.999640000Horseback riding injury 2860.0000 0.999971401Electric light bulb 1000.0000 0.999990000Motorcycle riding injury 143.0000 0.999998570Stepstone OO Framework 5.0000 0.999999950Residential fire 4.1100 0.999999959Telecom switch 3.0000 0.999999970Auto theft 1.0800 0.999999989Auto accident fatality 0.6680 0.999999993Commercial airline fatality 0.5000 0.999999995 © 2001 Mobile Systems Verification Corporation 5
  6. 6. The New IT Reality: Reliability Sells!• 372 Google hits on “five nines” “press release” – „AT&T Reports Precedent-Setting Five Nines Performance On Its Market-Leading Frame Relay Network‟ – „Five Nines: Windows 2000 Servers Deliver Near-Perfect Reliability for Businesses‟• Customer retention critical• “War of the nines” coming soon © 2001 Mobile Systems Verification Corporation 6
  7. 7. The New IT Reality: Usage• Last 25 years – New and better ways to do old things – 1/6th world using cell phones – Web -- thin client, fat server• Next 25 years – Mobile connectivity, new things – Persistent partial attention You ain’t seen nothin’ yet © 2001 Mobile Systems Verification Corporation 7
  8. 8. The New IT Reality: Hardware• A few indicators – Moores law – Watts per MIPS – Primary, secondary storage capacity – Wireless networks: personal, local, metro – 2.5/3 G cellular build-out, adaptive antenna – Optical data rates, fiber build-out – Application-specific instruction processors © 2001 Mobile Systems Verification Corporation 8
  9. 9. The New IT Reality: Convergence• Telecom, datacom converging• Optical backbone, wireless subnet – Broadband to the hand• Remote services, very large databases• Application service providers – Executable streaming – Thin client, fat pipe © 2001 Mobile Systems Verification Corporation 9
  10. 10. The New IT Reality: Mobility• Converging user device form factor – Cell phone, PDA, pager, Pocket PC• Applications – Location-based – Peer-to-peer – Personal agents• “Digital” interface receding – Wearable (neck tie, jewelry … ) – Voice recognition © 2001 Mobile Systems Verification Corporation 10
  11. 11. The New IT Reality: Mobility• Wireless vs. wired connection – On (off), Connected (not), In-range (out) – May reconnect with any base station – Base stations typically not homogenous in large networks – Security, privacy can‟t be achieved physically Behavior, load, noise, and data link strongly related to user itinerary © 2001 Mobile Systems Verification Corporation 11
  12. 12. The Unchanged IT Reality: Software• Still no silver bullet – Bug barrier  5 per KLOC (pre-test) any language, any process – Subtractive component reliability – Design limited to human bandwidth – Low-fidelity test suites aren‟t effective – Hand crafted test suites can‟t scale © 2001 Mobile Systems Verification Corporation 12
  13. 13. The Engineering Challenge• The new IT reality – User population grows 100-1000x – Most users will be mobile – Infrastructure uncontrollable, unstable – Increasing feature complexity – Increasing visibility and impact – “Five nines” will be required Existing software technology can’t achieve high reliability for ubiquitous IT © 2001 Mobile Systems Verification Corporation 13
  14. 14. The Strategy• The new IT reality – IT capability/capacity growing rapidly – Software capability/capacity static• End-to-end approach can achieve very high reliability – Best practices at component scope – Full test automation at system scope © 2001 Mobile Systems Verification Corporation 14
  15. 15. Testing Process• Test process – Design/generation, setup – Execution, evaluation• Levels – Testing by Poking Around – Manual Testing – Automated Test Script – Automated Generation/Agent – Full Test Automation © 2001 Mobile Systems Verification Corporation 15
  16. 16. Testing by Poking Around Test Execution System Under Test © 2001 Mobile Systems Verification Corporation 16
  17. 17. Manual Testing Test SetupTest Design/Generation Test Execution Test Results System Under Test Evaluation © 2001 Mobile Systems Verification Corporation 17
  18. 18. Automated Test Script Test SetupTest Design/Generation Test Execution Test Results System Under Test Evaluation © 2001 Mobile Systems Verification Corporation 18
  19. 19. Automated Generation/Agent Test SetupTest Design/Generation Test Execution Test Results System Under Test Evaluation © 2001 Mobile Systems Verification Corporation 19
  20. 20. Full Test Automation Test SetupTest Design/Generation Test Execution Test Results System Under Test Evaluation © 2001 Mobile Systems Verification Corporation 20
  21. 21. Test Effectiveness Tests/Hour Effectiveness ScopeTesting by Poking Around 100 Very low 100Manual Testing 1 Low 1,000Automated Test Script 10 Low, repeatable 10,000Automated Generation/Agent 10,000 Medium 1,000,000Full Test Automation 1,000,000 High 100,000,000 Full Automation achieves high reliability promise of profile-based testing © 2001 Mobile Systems Verification Corporation 21
  22. 22. Full Automation Case Study• E-commerce/securities market over private tcp/ip net• 3 million transactions per hour• 15 billion dollars per day• 3 years, version 1.0 live 4Q 2001 © 2001 Mobile Systems Verification Corporation 22
  23. 23. Full Automation Case Study• Rational unified process• About 90 use-cases, 600 KLOC• Java (services and GUI), some XML• Oracle DBMS• Many legacy interfaces• CORBA/IDL distributed object model• HA Sun server farm © 2001 Mobile Systems Verification Corporation 23
  24. 24. Full Automation Case Study• Full Automated Testing – Extended Use-case and Mode Machine – Rule-based simulation generates test load – Test agents (COTS + custom) – Automate pass/no pass evaluation• Estimated reliability at five nines – 1,000,000 events in four hours – No critical failures © 2001 Mobile Systems Verification Corporation 24
  25. 25. Lessons Learned• Automatable test requirements are better than design requirements• User-based simulation very effective test approach• Partial automated evaluation necessary Full automation can achieve high reliability for the new IT reality © 2001 Mobile Systems Verification Corporation 25
  26. 26. Conclusion• Great challenges and opportunities• High reliability can be achieved• Very high reliability need not be exotic © 2001 Mobile Systems Verification Corporation 26

×