Testing for 20 months – how costeffective is that?Session #3102801 December, 2011Farid VaswaniUniversity of AucklandTwitter: @FVaswani
Experience o 12 years IT experience in Development & Testing o Conference Speaker o Techie / Grumpy / Unsatisfied TesterResponsibilities • Testing Manager, University of Auckland o Team of 8 - 20 Testers and Test Engineers • Associate Board Member, ANZ Testing Board o Enhancing Career Opportunities for Testing Professionals
o 40,000 studentso 5,000 staffo 8 Facultieso 2 large-scale research instituteso 4 main campuses
o PeopleSoft CSo PeopleSoft HRo PeopleSoft FINo Databaseso WebLogic Servero RightNow (CRM)
o Oracle supporto SOA architectureo Real-time updateso Identity & Access Managemento Usabilityo Hardware virtualisationo Performanceo Students o http://www.youtube.com/user/MassInTheAss#p/a/f/5/CsMfvIEUFkk
Project FOCUSFocus On Changing University Services
o Budget: eight-figure $ amounto 63 business process, to be rewritteno 500+ staff affectedo 40,000+ student recordso 400,000 rows in one of the tableso 10+ years of data to be migratedo Project team size: 50+ FTEo + UAT testers
Programme Manager Process Manager Data Conversion Manager Development Manager Integration Lead EA Lead Security Lead Testing Manager
1. Testing Types2. Testing Responsibilities3. Test Team4. Testing Approach5. Testing Schedule6. Testing Tools7. Test Reporting
o Data Conversion Testingo Process Testingo Enhancement Testingo Integration Testingo Performance Testingo Security Testingo Usabilityo Compatibilityo User Acceptance Testingo Go-live
Process Testing Manager DSS Manager Manager Process Team Test Team Reporting Team (15) (6) (4) DC Tester Security Tester (2)Data Conversion Security Manager Manager
Process Testing Manager DSS Manager Manager • Functional Process Team • Integration Reporting Team (12) • Performance (5) • Automation • Compatibility • Security Scans DC Tester Security TesterData Conversion Security Manager Manager
Tip #1:Way in to otherteams’ time is through their managers.
Data Conversion TestingCC v7.6 International -> CS v9.0 Asia-Pac o CC v7.6 -> CC v8.0 -> CS v9.0 o Vanilla upgraded by Oracle lab o Customisations, converted (implemented) in-house
Data Conversion TestingTesting o Field mappings o Field counts o Record counts o GUI testing o EPR-CS9 reconciliation
Data Conversion Testing Tip #2:Data Conversion testers are good functional testers during their down time.
Data Conversion Testing Tip #3: Never underestimate the value of manual GUI checking after DC.
Process Testing10 Mega processes, 63 sub-processes o Admissions o Enrolment o Manage Campus Community o Financial Services o Financial Support o Teach Learn Evaluate o Transition Student o Advisement o Self Service o Reporting
Process TestingTesting o 6 Process Leads + 9 team members o Written Test Cases o Exploratory Scenarios o Mostly positive testing for staff based functionality
Process Testing Tip #4: Attend Process workshops; all of them, if possible.
Enhancement TestingScope + Testing o 54 enhancements o PeopleSoft only o Split between QA & Process team o Written Test Cases o Exploratory Scenarios o Some negative testing included
Enhancement Testing Tip #5: Good to keep a tap on business testers when writing test scripts.
Integration Testing*1* Interfaces o WLI o Data Extracts o Third party o Studylink (student loans) o National Student Index o Payment Gateway o Technical Testers responsible for these
Integration Testing*2* Process Integration o Student lifecycle o Various start & end points o Multiple processes o Multiple systems
Integration Testing*2* Process Integration o Scenario 1 (e.g.) o Admission > Enrolment > Fees > Exam > Completion > Graduation o Domestic student o NCEA o NSI o Undergrad o Student loan o 1st year student
Integration Testing*2* Process Integration o Scenario 4 (e.g.) o Admission > Enrolment > Fees > Visa Processing o International student o Post grad o 120 points o Paying full fees
Integration Testing*2* Process Integration o 7 scenarios o 45 (out of 63) sub-processes o 10 external systems o 240 transitions o One person responsible for these
Integration Testing Tip #6: Book some extra time slots for testing with external-party, in advance.
Performance TestingChallenge CS9 AFA WLI • Staff • Student • World using it • Batch • EOI • Relatively new • Transactions • EPR • Real-time • Reports • Self Service Infrastructure
Performance TestingTesting: Hybrid approach o 6 batch processes: manual o eg: Generate Invoice, Advisement, Transcript o Student Self-Service: automated o Staff transactions o Manual and automated data generation scripts o 14 reports, manual o AFA, automated o 9 WLI interfaces, scripted
Performance Testing Tip #7: Pre-Prod is NOT Test
SecurityTesting o Application Penetration Tests o User roles o Staff vs Students o Positive & Negative o Infrastructure Scan o Security certificates o Split URLs o https://student.auckland.ac.nz/ o https://student.admin.auckland.ac.nz/
Security Tip #8:Make an effort to understand security-speak
o 4 test cycles o Process & SMEs o Priority based o Partial processeso Enhancements (non-PS) o Agile, continuous testingo Non-functional o Performance – last 3-4 months o Security – 3 scans
o HP Quality Centre o Test case managemento Jmeter o Performanceo soapUI o Web – services o Functional & loado HermesJMS o WLI queueso AppScano MS-Word, MS-Excelo Google Docso Confluence wikio Jira
Defects Raised per Week250 Low200150 Medium100 High 50 0 Week Ending
Defect Status 2500 2000 Opened 1500 Cumulative Total 1000 Closed Cumulative 500 Nett Open 0 Reopened Week Ending 2500 Defects Forecast Actual 2000 ExpectedCumulative Defects 1500 1000 500 0 -500 Week Ending
Tip #0:Start Early. Test Early. Deliver on time.