Assurance for      Cloud Computing                      Robert V. BinderChicago Quality Assurance Association             ...
Overview   Weather Report   Role-Layer Model   Cloud Assurance Challenges   Microsoft Protocol Validation   Spec Expl...
   Weather Report   Role-Layer Model   Cloud Assurance Challenges   Microsoft Protocol Validation   Spec Explorer Fly...
Weather Report   What is Cloud Computing?       SOA, SaaS, Outsourced Infrastructure, Hosted        Apps, Web Services, ...
Distant Rumblings                                            Yes           Does your organization                         ...
Distant Rumblings                                                                                  6-12          When will...
Falling Pressure http://www.gartner.com/it/page.jsp?id=1124212, © Gartner Inc. Used by permission.                        ...
Excess Water Vapor? Why Larry Ellison hates Cloud computing http://www.youtube.com/watch?v=8UYa6gQC14o                    ...
Forecast: Partly Cloudy   Despite the hype, fundamental changes       Broader market, lower barriers           Incremen...
   Weather Report   Role-Layer Model   Cloud Assurance Challenges   Microsoft Protocol Validation   Spec Explorer Fly...
Role-Layer ModelLayers                                               Roles         Ownership                     © 2010 Ro...
Role-Layer Model   Layers       Endpoint, Network, Apps, Data, Hardware   Roles – distinct economic entities       End...
Classic IT             © 2010 Robert V. Binder   13
Classic IT, c. 1970              © 2010 Robert V. Binder   14
Service Bureau, c.1980             © 2010 Robert V. Binder   15
Client/Server, c.1995              © 2010 Robert V. Binder   16
E-Commerce ASP, c. 2002            © 2010 Robert V. Binder   17
SalesForce.com, c. 2004             © 2010 Robert V. Binder   18
Outsourced Room, c. 2009            © 2010 Robert V. Binder   19
   Weather Report   Role-Layer Model   Cloud Assurance Challenges   Microsoft Protocol Validation   Spec Explorer Fly...
What Should be Assured?   Contract between layer owners       Technical specification, not legal T&C   Confirm       W...
Assurance Concerns   SLAs       Availability, Reliability, Performance       Standby Capacity/Scalability   Security  ...
Trust, but Verify                            SLA                   Features   SecurityEnd User – Integrator   Confirm     ...
   Weather Report   Role-Layer Model   Cloud Assurance Challenges   Microsoft Protocol Validation   Spec Explorer Fly...
Microsoft Protocol Validation   Relevance for cloud assurance?       Huge win for model-based testing       Open techno...
Open Specifications Initiative   EU/US DOJ Decree       Microsoft must publish server side details       Over 250 proto...
What is a Protocol?   “Remote Desktop Protocol: Audio Output    Virtual Channel Extension”   Defines Messages   Defines...
What is a Protocol?   Remote Desktop Protocol: Audio Output    Virtual Channel Extension   Example message requirement  ...
Published TDs on MSDNMSDN Protocol Libraryhttp://msdn.microsoft.com/en-us/library/cc216513(PROT.10).aspx                  ...
Validation Approach   How to validate TDs?     Is the TD sufficient for interoperability?           Scrutinize       I...
Validation Approach   Extract requirements from TD   Analyze/model protocol contract       Data Structures       Metho...
Validation Process   Develop Technical Document   Study Phase       Start requirements extraction       Define high le...
Requirements Traceability   TD parsed to extract line-item requirements   100s to 1000s per technical document   Log re...
Typical Test Configuration              Tester Endpoint                                                                   ...
Productivity     100s of third party                                               Avg Hours Per Requirement      develop...
Quality   Produced    ~10,000    “Technical    Document    Issues”   Most TDI’s    identified before    tests run    Gri...
   Weather Report   Role-Layer Model   Cloud Assurance Challenges   Microsoft Protocol Validation   Spec Explorer Fly...
Spec Explorer   Model-based Testing Tool     Extension to Visual Studio    Model APIs as contracts     Guarded update r...
Testing with Spec Explorer   Analyze system under test, create a model    program, representing the entire SUT   Define ...
Netmon   Developed to support                                    Spec    protocol testing                                ...
   Weather Report   Role-Layer Model   Cloud Assurance Challenges   Microsoft Protocol Validation   Spec Explorer Fly...
Testing Cloud Contracts   Obtain or develop API requirements       Amazon Simple Storage Service       Google App Engin...
Amazon Simple Storage Spec Example of a cloud protocol http://docs.amazonwebservices.com/AmazonS3/latest/                 ...
Performance and Security   Working on Spec Explorer approach for       Performance testing       Reliability testing   ...
Implications for IT Assurance   Use the cloud (on-demand capacity) for testing   Assess testability of SLA and API contr...
Conclusions   Despite hype, cloud    computing means significant    changes for IT and IT    Assurance   Layers, Ownersh...
Thank You!rvbinder@gmail.com       312 404 5341
Upcoming SlideShare
Loading in …5
×

Assurance for Cloud Computing

306 views

Published on

Invited Talk, Chicago Quality Assurance Association. January, 27 2010, Chicago.
Overview of cloud computing - new challenges for achieving high reliability

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
306
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
8
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Assurance for Cloud Computing

  1. 1. Assurance for Cloud Computing Robert V. BinderChicago Quality Assurance Association January 27, 2010
  2. 2. Overview Weather Report Role-Layer Model Cloud Assurance Challenges Microsoft Protocol Validation Spec Explorer Flyby Implications for IT Assurance © 2010 Robert V. Binder 2
  3. 3.  Weather Report Role-Layer Model Cloud Assurance Challenges Microsoft Protocol Validation Spec Explorer Flyby Implications for IT Assurance © 2010 Robert V. Binder 3
  4. 4. Weather Report What is Cloud Computing?  SOA, SaaS, Outsourced Infrastructure, Hosted Apps, Web Services, Virtualization, Grid … Economic drivers  Rent versus Buy versus Make  No operational responsibilities  Scalable  “No Software” © 2010 Robert V. Binder 4
  5. 5. Distant Rumblings Yes Does your organization 34% currently have a cloud computing strategy? No66% 6-12 Months 12-24 4% Months Don’t 8% When will you Know 36 + 18% implement Months 8% cloud computing? No Plan 62% The Future of Cloud Computing, MPS Partners. www.mpspartners.com. © 2010 Robert V. Binder 5
  6. 6. Distant Rumblings 6-12 When will you adopt Months 12 + 3% Months cloud storage services? 5% Will Not No Plan 46% 46%Business Users Are Not Ready For Cloud Storage, Forrester. http://www.networkworld.com/news/2010/012510-cloud-storage-hype.html © 2010 Robert V. Binder 6
  7. 7. Falling Pressure http://www.gartner.com/it/page.jsp?id=1124212, © Gartner Inc. Used by permission. © 2010 Robert V. Binder 7
  8. 8. Excess Water Vapor? Why Larry Ellison hates Cloud computing http://www.youtube.com/watch?v=8UYa6gQC14o © 2010 Robert V. Binder 8
  9. 9. Forecast: Partly Cloudy Despite the hype, fundamental changes  Broader market, lower barriers  Incremental IT capacity  General purpose apps  Trends  Apps/cycles/storage more like electric power  Outsourcing to reduce cost and risk  Decentralization, reliance on commons Assurance challenges  Contract-oriented  Ongoing, pre and post release © 2010 Robert V. Binder 9
  10. 10.  Weather Report Role-Layer Model Cloud Assurance Challenges Microsoft Protocol Validation Spec Explorer Flyby Implications for IT Assurance © 2010 Robert V. Binder 10
  11. 11. Role-Layer ModelLayers Roles Ownership © 2010 Robert V. Binder 11
  12. 12. Role-Layer Model Layers  Endpoint, Network, Apps, Data, Hardware Roles – distinct economic entities  End User  Integrator  Provider  Operator = Provider + Integrator Ownership  Share, Rent, Buy, Build © 2010 Robert V. Binder 12
  13. 13. Classic IT © 2010 Robert V. Binder 13
  14. 14. Classic IT, c. 1970 © 2010 Robert V. Binder 14
  15. 15. Service Bureau, c.1980 © 2010 Robert V. Binder 15
  16. 16. Client/Server, c.1995 © 2010 Robert V. Binder 16
  17. 17. E-Commerce ASP, c. 2002 © 2010 Robert V. Binder 17
  18. 18. SalesForce.com, c. 2004 © 2010 Robert V. Binder 18
  19. 19. Outsourced Room, c. 2009 © 2010 Robert V. Binder 19
  20. 20.  Weather Report Role-Layer Model Cloud Assurance Challenges Microsoft Protocol Validation Spec Explorer Flyby Implications for IT Assurance © 2010 Robert V. Binder 20
  21. 21. What Should be Assured? Contract between layer owners  Technical specification, not legal T&C Confirm  We’re getting what we expect  What we expect is adequate Certify  We can meet our commitments  Individual  Collective (scalable) © 2010 Robert V. Binder 21
  22. 22. Assurance Concerns SLAs  Availability, Reliability, Performance  Standby Capacity/Scalability Security  Authentication  Access Control Features  Correct response  Data integrity Effectiveness/Acceptance  ROI, votes, sales © 2010 Robert V. Binder 22
  23. 23. Trust, but Verify SLA Features SecurityEnd User – Integrator Confirm Confirm ConfirmIntegrator – End User Certify Certify CertifyIntegrator – Provider Confirm Confirm ConfirmProvider – Integrator Certify Certify Certify © 2010 Robert V. Binder 23
  24. 24.  Weather Report Role-Layer Model Cloud Assurance Challenges Microsoft Protocol Validation Spec Explorer Flyby Implications for IT Assurance © 2010 Robert V. Binder 24
  25. 25. Microsoft Protocol Validation Relevance for cloud assurance?  Huge win for model-based testing  Open technology  Supports contract-based assurance  Protocol validation approach good fit for cloud assurance challenges © 2010 Robert V. Binder 25
  26. 26. Open Specifications Initiative EU/US DOJ Decree  Microsoft must publish server side details  Over 250 protocols (APIs) Publish protocols as “Technical Documents” (TD)  Open Specifications Initiative www.microsoft.com/openspecifications  One TD for each server-side API/service  Strict and detailed standards for TD content Goal: interoperability. TDs must be sufficient for 3rd party to replace or use target API © 2010 Robert V. Binder 26
  27. 27. What is a Protocol? “Remote Desktop Protocol: Audio Output Virtual Channel Extension” Defines Messages Defines Behavior © 2010 Robert V. Binder 27
  28. 28. What is a Protocol? Remote Desktop Protocol: Audio Output Virtual Channel Extension Example message requirement The Server Audio Formats and Version PDU is a PDU used by the server to send version information and a list of supported audio formats to the client. This PDU MUST be sent using static virtual channels. Example behavioral requirement The server may send the Training PDU at any time and during any sequence, not just during the initialization sequence. The only prerequisite are that version exchange MUST have occurred and that if the client and server are both at least version 6, the server MUST have received a Quality Mode PDU. © 2010 Robert V. Binder 28
  29. 29. Published TDs on MSDNMSDN Protocol Libraryhttp://msdn.microsoft.com/en-us/library/cc216513(PROT.10).aspx © 2010 Robert V. Binder 29
  30. 30. Validation Approach How to validate TDs?  Is the TD sufficient for interoperability?  Scrutinize  Is the TD accurate?  Develop test for each requirement  Only check over-the-wire data Protocol Engineering Team  Reviewers – independent 3rd parties  Testers – 3rd party contractor  Microsoft development teams write TDs  Reviewers and Testers Scrutinize TDs  Testers develop Test Suites from the TD © 2010 Robert V. Binder 30
  31. 31. Validation Approach Extract requirements from TD Analyze/model protocol contract  Data Structures  Method Behavior  API Behavior Develop adapters to parse/check messages Execute test suite Write “TD Issue” for any anomaly © 2010 Robert V. Binder 31
  32. 32. Validation Process Develop Technical Document Study Phase  Start requirements extraction  Define high level test approach Plan Phase  Define test model  Define test harness Design Phase  Implement model and drivers Final Phase  Run test suites, analyze traces TDs released to MSDN after passing Final © 2010 Robert V. Binder 32
  33. 33. Requirements Traceability TD parsed to extract line-item requirements 100s to 1000s per technical document Log requirement record when pass conditions met  R 562: “The server must return 404 in the reply code when the target URL cannot be found.” Contracts.Requires(ReplyEnabled(replyCode) && badURL(true)); if replyCode == 404) { log(562, “Received 404”) } } © 2010 Robert V. Binder 33
  34. 34. Typical Test Configuration Tester Endpoint SUT Control Test Suite (optional) Adapters Tested Endpoint Transport TransportGrieskamp, Kicillof, Stobie, Braberman. Model-Based Quality Assurance of Protocol Documentation: Tools and Methodology. ICST 2009. © 2010 Robert V. Binder 34
  35. 35. Productivity 100s of third party Avg Hours Per Requirement developers trained Task in modeling and TD review 1.1 test development Requirement gathering 0.8 On average, MBT Model authoring 0.5 takes 42% less Traditional test coding 0.6 time than hand- Adapter development 1.2 coding Test case execution 0.6 Final adjustments 0.3 Total, all phases 5.1 Grieskamp et al. Op cit.. © 2010 Robert V. Binder 35
  36. 36. Quality Produced ~10,000 “Technical Document Issues” Most TDI’s identified before tests run Grieskamp et al. Op cit.. © 2010 Robert V. Binder 36
  37. 37.  Weather Report Role-Layer Model Cloud Assurance Challenges Microsoft Protocol Validation Spec Explorer Flyby Implications for IT Assurance © 2010 Robert V. Binder 37
  38. 38. Spec Explorer Model-based Testing Tool  Extension to Visual Studio Model APIs as contracts  Guarded update rules for modeled state  Define accepting states Generates call sequences and data bindings that “explore” model  Finds update sequences that lead to accepting state Guards and accepters are the “oracle” Info and free download http://msdn.microsoft.com/en-us/devlabs/ee692301.aspx © 2010 Robert V. Binder 38
  39. 39. Testing with Spec Explorer Analyze system under test, create a model program, representing the entire SUT Define parameters for test generation (“cord file”) Define behavioral subsets (use cases/scenarios) “machine” Explore the model and the machine Generate test cases Run tests © 2010 Robert V. Binder 39
  40. 40. Netmon Developed to support Spec protocol testing Explorer Similar to Wireshark Parses all published Test Objects Microsoft protocols Easy to add new parsers Adapters SUT API supports automated testing SUT  Returns parsed messages Netmon Endpoint Info and free download http://blogs.technet.com/netmon/ © 2010 Robert V. Binder 40
  41. 41.  Weather Report Role-Layer Model Cloud Assurance Challenges Microsoft Protocol Validation Spec Explorer Flyby Implications for IT Assurance © 2010 Robert V. Binder 41
  42. 42. Testing Cloud Contracts Obtain or develop API requirements  Amazon Simple Storage Service  Google App Engine  Many others … Create model program for the contract  One method for each message  One method for each reply  Variables represent behavioral constraints  Scenarios © 2010 Robert V. Binder 42
  43. 43. Amazon Simple Storage Spec Example of a cloud protocol http://docs.amazonwebservices.com/AmazonS3/latest/ © 2010 Robert V. Binder 43
  44. 44. Performance and Security Working on Spec Explorer approach for  Performance testing  Reliability testing  Security assessment © 2010 Robert V. Binder 44
  45. 45. Implications for IT Assurance Use the cloud (on-demand capacity) for testing Assess testability of SLA and API contracts early Mission-critical apps still need in-house expertise Integrators will do less traditional testing Integrators have a double role: confirm/certify Ongoing monitoring necessary Favor providers that can demo contracts © 2010 Robert V. Binder 45
  46. 46. Conclusions Despite hype, cloud computing means significant changes for IT and IT Assurance Layers, Ownership, Roles Confirm or Certify Microsoft success shows how to test cloud contracts Contract testing viable approach for Providers and Integrators © 2010 Robert V. Binder 46
  47. 47. Thank You!rvbinder@gmail.com 312 404 5341

×