What is a Testing Framework? “Tools of the trade”; what are they? Test Phases, Test Stages, Testing Activities, Verification Methods, a Testing Workflow, Status Reporting, Test Types Discussions: What do you bring to the table? What does the Client bring to the table? Why are you there? How do you adapt & why do should you?

1. A Test Framework What do you bring to the table on an assignment? Presentation by Ronald Kohlman June 2010 http://au.linkedin.com/pub/ronald-kohlman/0/5bb/428 © Copyright 2010 Ronald Kohlman - all rights reserved

2. Contents What is a Testing Framework? “Tools of the trade”; what are they? Test Phases, Test Stages, Testing Activities, Verification Methods, a Testing Workflow, Status Reporting, Test Types Discussions: What do you bring to the table? What does the Client bring to the table? Why are you there? How do you adapt & why do should you? 2

3. This Presentation This presentation is for you, as Testing Practice Leaders, with the aim to generate healthy discussion and participation by you The ideas presented here are focussed on the high level attributes for a testing framework 3

4. This Presentation This presentation is for you, as Testing Practice Leaders, with the aim to generate healthy discussion and participation by you The ideas presented here are focussed on the high level attributes for a testing framework Who has set up a Testing Framework before? 4

5. What is a Testing Framework? It is not just a set of templates A process structure that encapsulates the testing activity A Methodology; that includes standards, policies, procedures and techniques Contains management processes for managing the testing effort Requires: Consistency, Common approach, Adaptability, Depth & Variety of Experience 5

6. What is a Testing Framework? It is not just a set of templates A process structure that encapsulates the testing activity A Methodology; that includes standards, policies, procedures and techniques Contains management processes for managing the testing effort Requires: Consistency, Common approach, Adaptability, Depth & Variety of Experience The purpose of testing is just as much to prove that something doesn’t work, as it is to prove that it does work. 6

7. What is a Testing Framework? What other aspects do you think make up a Testing Framework? It is not just a set of templates A process structure that encapsulates the testing activity A Methodology; that includes standards, policies, procedures and techniques Contains management processes for managing the testing effort Requires: Consistency, Common approach, Adaptability, Depth & Variety of Experience The purpose of testing is just as much to prove that something doesn’t work, as it is to prove that it does work. 7

9. Planning

10. Preparation

11. Requirements

12. Assets

13. Test Cases

14. Test Data

15. Environment

16. Execution

17. Reporting of Results & MetricsTemplates Strategy Master Test Plan Detailed Test Plan RTMs Others? Test Tools QC, QTP Excel Others? Estimation tools & Processes 8

19. Planning

20. Preparation

21. Requirements

22. Assets

23. Test Cases

24. Test Data

25. Environment

26. Execution

27. Reporting of Results & MetricsTemplates Strategy Master Test Plan Detailed Test Plan RTMs Others? Test Tools QC, QTP Excel Others? Estimation tools & Processes What’s in your “Tool Chest”? 9

29. Systems Test

30. Performance & Volume Test

31. Systems Integration Test

32. User Acceptance Test

33. Pre-Production Test10

35. Systems Test

36. Performance & Volume Test

37. Systems Integration Test

38. User Acceptance Test

39. Pre-Production TestThere are a number of “other” Test Phases or Sub-phases that could be intersected with these primary ones 11

41. Systems Test

42. Performance & Volume Test

43. Systems Integration Test

44. User Acceptance Test

45. Pre-Production TestWhat others do you know of & when are they executed? There are a number of “other” Test Phases or Sub-phases that could be intersected with these primary ones 12

47. Planning

48. Preparation

49. Execution

50. Reporting (across the three stages)13

52. Planning

53. Preparation

54. Execution

55. Reporting (across the three stages)Planning 14

57. Planning

58. Preparation

59. Execution

60. Reporting (across the three stages)Planning Reporting 15

62. Planning

63. Preparation

64. Execution

65. Reporting (across the three stages)Planning Preparation Reporting 16

67. Planning

68. Preparation

69. Execution

70. Reporting (across the three stages)Planning Preparation Execution Reporting 17

72. Planning

73. Preparation

74. Execution

75. Reporting (across the three stages)Planning Preparation Execution Reporting 18

76. Testing Activities 19

77. Testing Activities Inputs to Testing Activities 20

78. Testing Activities Inputs to Testing Activities Planning 21

79. Testing Activities Inputs to Testing Activities Planning Preparation 22

80. Testing Activities Inputs to Testing Activities Planning Preparation Execution 23

81. Testing Activities Inputs to Testing Activities Planning Preparation Execution Reporting 24

82. Verification Methods “It’s not a technology project so we don’t need to do any testing!” All parts of the solution incorporate features or characteristics that are not testable in a technical sense, but which must be verified using an alternative method. The objectives of the Verification process include using established criteria to conduct verification of the physical architecture (including software and interfaces) from the lowest level up to the total system. There are 5 potential methods of verification that you can be engaged in to verify the solution The most common verification method understood and used is “TEST” DEMONSTRATION ANALYSIS INSPECTION TEST EXTENSION 25

83. A Typical Testing Workflow A typical workflow for testing activities and deliverables You may add or vary these basics 26

84. Management of Status Reporting Some generic metrics reporting requirements These vary depending upon the Testing Stage you are in for any particular Test Phase 27

86. Static

87. Dynamic

88. Regression

89. Static

90. Dynamic28

92. Functional, Non-Functional

93. Interoperability, Capacity, Sociability, Capability

94. Load, Volume, Performance, Soak, Stress

95. Security & Penetration

96. Usability, Accessibility

97. Sanity, Smoke, Break, Shake-out

98. White Box, Black Box

100. Static

101. Dynamic

102. Regression

103. Static

104. Dynamic29

106. Functional, Non-Functional

107. Interoperability, Capacity, Sociability, Capability

108. Load, Volume, Performance, Soak, Stress

109. Security & Penetration

110. Usability, Accessibility

111. Sanity, Smoke, Break, Shake-out

112. White Box, Black Box

114. Static

115. Dynamic

116. Regression

117. Static

118. DynamicThis list is not exhaustive… 30

120. Functional, Non-Functional

121. Interoperability, Capacity, Sociability, Capability

122. Load, Volume, Performance, Soak, Stress

123. Security & Penetration

124. Usability, Accessibility

125. Sanity, Smoke, Break, Shake-out

126. White Box, Black Box

128. Static

129. Dynamic

130. Regression

131. Static

132. DynamicThis list is not exhaustive… 31

133. What do you bring to the table? What works for you? What doesn’t work you? What is the one thing that you would suggest as “a must have” in your tool kit? Where do you go to source information to help you? Own knowledge base? Colleagues? Training? Organisation? Internet? Books? Papers? Conferences? What knowledge do you pass on (what do you leave behind)? How do you approach the work? 32

134. What does the Organisation bring to the table? Their own processes, culture, ideas, practices and templates Their own problems, issues, and perceptions Organisation Maturity Ability to accept your inputs, or not Inability to clarify scope Inability to do it the “right way” Inability to understand testing (“Education”) Wanting to “cut corners” & squeeze time 33

135. What does the Organisation bring to the table? Their own processes, culture, ideas, practices and templates Their own problems, issues, and perceptions Organisation Maturity Ability to accept your inputs, or not Inability to clarify scope Inability to do it the “right way” Inability to understand testing (“Education”) Wanting to “cut corners” & squeeze time What do you look for? 34

136. What does the Organisation bring to the table? Their own processes, culture, ideas, practices and templates Their own problems, issues, and perceptions Organisation Maturity Ability to accept your inputs, or not Inability to clarify scope Inability to do it the “right way” Inability to understand testing (“Education”) Wanting to “cut corners” & squeeze time What do you look for? How do you know you have all the information you need? 35

137. Why are you there? To assist the organisation in driving the delivery of their testing requirements Help them to achieve their objectives Provide them with your testing practice experience and knowledge Provide guidance in the best approach for testing their “solution” Provide a “Framework” & Standardised approach 36

138. How do you adapt & why should you? Ask yourself:How do you adapt and why should you? 37

139. Thank You… How do you adapt and why do you need to? Thank you for your attention. 38