Your SlideShare is downloading. ×
Achieving Very High Reliability for Ubiquitous Information Technology
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Achieving Very High Reliability for Ubiquitous Information Technology

180
views

Published on

Invited talk, November 26, 2001. Center for Embedded Software Systems. Aalborg, Denmark. Overview of reliability challenges in mobile systems.

Invited talk, November 26, 2001. Center for Embedded Software Systems. Aalborg, Denmark. Overview of reliability challenges in mobile systems.

Published in: Technology

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
180
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Achieving Very High Reliability forUbiquitous Information Technology Center for Embedded Software Systems Aalborg, Denmark November 26, 2001 Robert V. Binder Mobile Systems Verification Corporation www.MobileSystemsVerification.com
  • 2. Overview• Vision of very high IT reliability• The new IT reality and its challenges• Strategy for very high reliability © 2001 Mobile Systems Verification Corporation 2
  • 3. The Vision• Test engineering = system engineering• Testing is fully automated, high-fidelity• Very high reliability no longer exotic © 2001 Mobile Systems Verification Corporation 3
  • 4. Reliability Arithmetic• Reliability: probability of non-failure• MTTR: mean time to – recover, repair, restart …• Availability: percent up-time – Availability = 1 / 1 + (MTTR  Reliability) – 99.999% availability = 5 min per year – “Five nines” © 2001 Mobile Systems Verification Corporation 4
  • 5. Some Reliability Data Points Reliability, Failures per Availability, million hours 6 min MTTRNT 4.0 Desktop 82000.0000 0.999000000Windows 2000 Server Family 36012.9647 0.999640000Horseback riding injury 2860.0000 0.999971401Electric light bulb 1000.0000 0.999990000Motorcycle riding injury 143.0000 0.999998570Stepstone OO Framework 5.0000 0.999999950Residential fire 4.1100 0.999999959Telecom switch 3.0000 0.999999970Auto theft 1.0800 0.999999989Auto accident fatality 0.6680 0.999999993Commercial airline fatality 0.5000 0.999999995 © 2001 Mobile Systems Verification Corporation 5
  • 6. The New IT Reality: Reliability Sells!• 372 Google hits on “five nines” “press release” – „AT&T Reports Precedent-Setting Five Nines Performance On Its Market-Leading Frame Relay Network‟ – „Five Nines: Windows 2000 Servers Deliver Near-Perfect Reliability for Businesses‟• Customer retention critical• “War of the nines” coming soon © 2001 Mobile Systems Verification Corporation 6
  • 7. The New IT Reality: Usage• Last 25 years – New and better ways to do old things – 1/6th world using cell phones – Web -- thin client, fat server• Next 25 years – Mobile connectivity, new things – Persistent partial attention You ain’t seen nothin’ yet © 2001 Mobile Systems Verification Corporation 7
  • 8. The New IT Reality: Hardware• A few indicators – Moores law – Watts per MIPS – Primary, secondary storage capacity – Wireless networks: personal, local, metro – 2.5/3 G cellular build-out, adaptive antenna – Optical data rates, fiber build-out – Application-specific instruction processors © 2001 Mobile Systems Verification Corporation 8
  • 9. The New IT Reality: Convergence• Telecom, datacom converging• Optical backbone, wireless subnet – Broadband to the hand• Remote services, very large databases• Application service providers – Executable streaming – Thin client, fat pipe © 2001 Mobile Systems Verification Corporation 9
  • 10. The New IT Reality: Mobility• Converging user device form factor – Cell phone, PDA, pager, Pocket PC• Applications – Location-based – Peer-to-peer – Personal agents• “Digital” interface receding – Wearable (neck tie, jewelry … ) – Voice recognition © 2001 Mobile Systems Verification Corporation 10
  • 11. The New IT Reality: Mobility• Wireless vs. wired connection – On (off), Connected (not), In-range (out) – May reconnect with any base station – Base stations typically not homogenous in large networks – Security, privacy can‟t be achieved physically Behavior, load, noise, and data link strongly related to user itinerary © 2001 Mobile Systems Verification Corporation 11
  • 12. The Unchanged IT Reality: Software• Still no silver bullet – Bug barrier  5 per KLOC (pre-test) any language, any process – Subtractive component reliability – Design limited to human bandwidth – Low-fidelity test suites aren‟t effective – Hand crafted test suites can‟t scale © 2001 Mobile Systems Verification Corporation 12
  • 13. The Engineering Challenge• The new IT reality – User population grows 100-1000x – Most users will be mobile – Infrastructure uncontrollable, unstable – Increasing feature complexity – Increasing visibility and impact – “Five nines” will be required Existing software technology can’t achieve high reliability for ubiquitous IT © 2001 Mobile Systems Verification Corporation 13
  • 14. The Strategy• The new IT reality – IT capability/capacity growing rapidly – Software capability/capacity static• End-to-end approach can achieve very high reliability – Best practices at component scope – Full test automation at system scope © 2001 Mobile Systems Verification Corporation 14
  • 15. Testing Process• Test process – Design/generation, setup – Execution, evaluation• Levels – Testing by Poking Around – Manual Testing – Automated Test Script – Automated Generation/Agent – Full Test Automation © 2001 Mobile Systems Verification Corporation 15
  • 16. Testing by Poking Around Test Execution System Under Test © 2001 Mobile Systems Verification Corporation 16
  • 17. Manual Testing Test SetupTest Design/Generation Test Execution Test Results System Under Test Evaluation © 2001 Mobile Systems Verification Corporation 17
  • 18. Automated Test Script Test SetupTest Design/Generation Test Execution Test Results System Under Test Evaluation © 2001 Mobile Systems Verification Corporation 18
  • 19. Automated Generation/Agent Test SetupTest Design/Generation Test Execution Test Results System Under Test Evaluation © 2001 Mobile Systems Verification Corporation 19
  • 20. Full Test Automation Test SetupTest Design/Generation Test Execution Test Results System Under Test Evaluation © 2001 Mobile Systems Verification Corporation 20
  • 21. Test Effectiveness Tests/Hour Effectiveness ScopeTesting by Poking Around 100 Very low 100Manual Testing 1 Low 1,000Automated Test Script 10 Low, repeatable 10,000Automated Generation/Agent 10,000 Medium 1,000,000Full Test Automation 1,000,000 High 100,000,000 Full Automation achieves high reliability promise of profile-based testing © 2001 Mobile Systems Verification Corporation 21
  • 22. Full Automation Case Study• E-commerce/securities market over private tcp/ip net• 3 million transactions per hour• 15 billion dollars per day• 3 years, version 1.0 live 4Q 2001 © 2001 Mobile Systems Verification Corporation 22
  • 23. Full Automation Case Study• Rational unified process• About 90 use-cases, 600 KLOC• Java (services and GUI), some XML• Oracle DBMS• Many legacy interfaces• CORBA/IDL distributed object model• HA Sun server farm © 2001 Mobile Systems Verification Corporation 23
  • 24. Full Automation Case Study• Full Automated Testing – Extended Use-case and Mode Machine – Rule-based simulation generates test load – Test agents (COTS + custom) – Automate pass/no pass evaluation• Estimated reliability at five nines – 1,000,000 events in four hours – No critical failures © 2001 Mobile Systems Verification Corporation 24
  • 25. Lessons Learned• Automatable test requirements are better than design requirements• User-based simulation very effective test approach• Partial automated evaluation necessary Full automation can achieve high reliability for the new IT reality © 2001 Mobile Systems Verification Corporation 25
  • 26. Conclusion• Great challenges and opportunities• High reliability can be achieved• Very high reliability need not be exotic © 2001 Mobile Systems Verification Corporation 26

×