Keeping Score on Testing


Published on

Keeping Score on Testing

Published in: Business, Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Keeping Score on Testing

  1. 1. Keeping Score on Testing Marjorie Windelberg, Ph.D. SecureIT Conference March 2006 Copyright Marjorie Windelberg 2006. This work is the intellectual property of the author. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the authors. To disseminate otherwise or to republish requires written permission from the author.
  2. 2. Overview <ul><li>Information security and business continuity </li></ul><ul><li>Management foundations </li></ul><ul><li>Reasons to test </li></ul><ul><li>Dimensions of testing </li></ul><ul><li>Planning for tests, with 2 examples </li></ul><ul><li>Conducting tests </li></ul><ul><li>After the test </li></ul>
  3. 3. What to Protect? <ul><li>Mission critical functions </li></ul><ul><li>Information assets </li></ul><ul><li>Financial interests </li></ul><ul><li>Reputation </li></ul><ul><li>People </li></ul>
  4. 4. Planning for Incidents <ul><li>Requirements for </li></ul><ul><ul><li>Security </li></ul></ul><ul><ul><li>Business Continuity </li></ul></ul><ul><li>Strategies </li></ul><ul><li>Resources </li></ul><ul><li>Roles and responsibilities </li></ul>
  5. 5. Why Test? <ul><li>Determine soundness of policies underlying the plan </li></ul><ul><li>Enhance compliance </li></ul><ul><li>Evaluate feasibility of strategies </li></ul><ul><li>Assess state of readiness </li></ul>
  6. 6. Why Test? <ul><li>Check plans for flaws </li></ul><ul><ul><li>Errors and omissions, including level of specificity </li></ul></ul><ul><li>Assess quality (accurate, complete, realistic?) </li></ul><ul><ul><li>Of policies and procedures </li></ul></ul><ul><ul><li>Of staff skills </li></ul></ul><ul><ul><li>Of resources </li></ul></ul><ul><li>Determine impact of incidents on systems and processes, uncover additional vulnerabilities </li></ul>
  7. 7. Why Test? <ul><li>Prepare and train involved groups </li></ul><ul><ul><li>Understand the plan </li></ul></ul><ul><ul><li>Rehearse the procedures </li></ul></ul><ul><ul><li>Develop cooperation / teamwork </li></ul></ul><ul><ul><li>Share information </li></ul></ul><ul><li>Correct shortcomings </li></ul><ul><li>Update the plan </li></ul>
  8. 8. Dimensions of Testing <ul><li>Prevention </li></ul><ul><li>Detection </li></ul><ul><li>Notification and Escalation </li></ul><ul><li>Status Reporting </li></ul><ul><li>Impact Assessment </li></ul><ul><li>Response </li></ul><ul><li>Resumption </li></ul><ul><li>Recovery </li></ul><ul><li>Restoration </li></ul>
  9. 9. Dimensions of Testing <ul><li>Announced or unannounced </li></ul><ul><li>Reviews and audits </li></ul><ul><li>Tabletop, simulation, drill </li></ul><ul><li>Which incidents to test? </li></ul><ul><ul><li>Minor ones are highly probable </li></ul></ul><ul><ul><li>Major ones are less likely </li></ul></ul>
  10. 10. Dimensions of Testing <ul><li>Exercise procedures, systems </li></ul><ul><ul><li>Never been tested </li></ul></ul><ul><ul><li>Not tested recently </li></ul></ul><ul><ul><li>Found deficient in previous tests </li></ul></ul><ul><li>Testing procedures </li></ul><ul><ul><li>Individual, multiple, interrelated </li></ul></ul><ul><ul><li>With or without external groups </li></ul></ul>
  11. 11. Planning Participation: Internal <ul><li>Technical staff and management </li></ul><ul><li>Organizational managers </li></ul><ul><li>Other units </li></ul><ul><ul><li>Public relations </li></ul></ul><ul><ul><li>Legal counsel </li></ul></ul><ul><ul><li>Human resources </li></ul></ul><ul><ul><li>Facilities </li></ul></ul><ul><ul><li>End users and others </li></ul></ul>
  12. 12. Planning Participation: External <ul><li>Law enforcement, public safety </li></ul><ul><li>Information security organizations </li></ul><ul><li>ISPs </li></ul><ul><li>Interconnected partners </li></ul><ul><li>Vendors and suppliers </li></ul>
  13. 13. Planning the Tests <ul><li>Set objectives, scope, tasks to cover </li></ul><ul><li>Set performance measures </li></ul><ul><ul><li>Time, quality </li></ul></ul><ul><li>Designate participants and observers </li></ul><ul><li>Prepare documentation </li></ul><ul><ul><li>Procedures, inventories, diagrams, plans, policies </li></ul></ul><ul><li>Arrange resources </li></ul><ul><li>Schedule time for review afterwards </li></ul>
  14. 14. Example 1: IT Security
  15. 15. Example 1: IT Security
  16. 16. Example 1: IT Security
  17. 17. Example 2: Physical Disaster
  18. 18. Example 2: Physical Disaster
  19. 19. Example 2: Physical Disaster
  20. 20. The Test <ul><li>Observation Log </li></ul><ul><ul><li>Grade completion of tasks </li></ul></ul><ul><ul><li>Time to accomplish tasks </li></ul></ul><ul><ul><li>Other notes </li></ul></ul>
  21. 21. Example 1: IT Security
  22. 22. Example 2: Physical Disaster
  23. 23. After the Test <ul><li>Review </li></ul><ul><ul><li>Discuss observation log </li></ul></ul><ul><ul><li>Add participants’ insights </li></ul></ul><ul><ul><li>Document issues </li></ul></ul><ul><li>Report </li></ul><ul><ul><li>Assess test results </li></ul></ul><ul><ul><li>Make recommendations and list action items </li></ul></ul><ul><li>Update plans </li></ul><ul><li>Schedule future test </li></ul>
  24. 24. Example 1: IT Security
  25. 25. Example 2: Physical Disaster
  26. 26. Marjorie Windelberg, Ph.D. [email_address] <ul><li>If you don’t keep score, how do you know if you passed the test? </li></ul>