Software Testing: Legacy
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Software Testing: Legacy

on

  • 1,406 views

 

Statistics

Views

Total Views
1,406
Views on SlideShare
1,405
Embed Views
1

Actions

Likes
0
Downloads
53
Comments
0

1 Embed 1

http://www.slideshare.net 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Software Testing: Legacy Presentation Transcript

  • 1. Software Testing: Legacy & Future 2005 Josh Soldati Head Of Testing Services, North America
  • 2. Testing No Longer The Ugly Step-Child “Once the backroom boys, the ‘invisibles’ of the software industry are striking back. Testing has assumed strategic importance in all software services companies today. Customers and users have become tired of paying top dollar for software, which either does not work, or if it does, then not according to expectations.” - Business World India* *Emphasis added. Cognizant Confidential 2
  • 3. Software Testing: The Legacy Code & Rollout & Inception Requirements Design Testing Unit Test Deployment The Legacy The Impact • Testing teams involved too late in the lifecycle, limiting ability to provide accurate Testing considered a discrete step and comprehensive test coverage in the lifecycle – the last step. • Testing effort not adequately estimated, impacting delivery schedule and/or budget No formal processes or metrics • Limited ability to measure/improve either established for software testing. efficiency or quality of testing effort • Developers test their own code Testing and software quality • Top talent moves out of software quality assurance not considered roles specialized skills or career path. • Unqualified resources perform and manage testing activities Cognizant Confidential 3
  • 4. The Ultimate Cost: Late Defect Detection* Requirements Design Code Unit Test Test Execution Production Find/Fix defects during Requirements = x to 2x = $75 to $150** Find/Fix defects during Design = 5x = $375** Find/Fix defects during Coding = 10x = $750** Find/Fix defects during Unit Testing = 20x = $1500** Find/Fix defects during Test Execution = 50x = $3750** Find/Fix defects during Production = 200x = $15000** *Source:Gartner. ** Example cost figures assume x = 1 person hour, at a cost of $75/hr. Cognizant Confidential 4
  • 5. Breaking Away From The Legacy: Cognizant’s Holistic View Integrated Development Lifecycle Integrated Development Lifecycle Code & Test Rollout & Inception Requirements Design Unit Test Execution Deployment Integrated Testing Processes & Methodology Testing Infrastructure & Tools . Testing Operating Model Testing Operating Model Testing Organization Testing Organization Cognizant Confidential 5
  • 6. Cognizant’s Testing Framework: Critical Components Of Software Testing Process Process Methodology Methodology Infrastructure Infrastructure • •Code Promotion* • •Test Planning && Estimation • •Staging environments* Code Promotion* Test Planning Estimation Staging environments* • •Build Acceptance • •Requirements Traceability* • •Test beds Build Acceptance Requirements Traceability* Test beds • •Defect Management* • •Integration Testing • •Desktops Defect Management* Integration Testing Desktops • •Test Environment Management • •System Testing • •Networks Test Environment Management System Testing Networks • •Test Execution Reporting • •User Acceptance Testing Test Execution Reporting User Acceptance Testing • •Knowledge Management • •Performance Testing Knowledge Management Performance Testing • •Metrics Tracking • •Regression Testing Metrics Tracking Regression Testing • •Delivery Assurance • •Test Data Creation Delivery Assurance Test Data Creation • •Process Quality Initiatives • •Automation Testing Process Quality Initiatives Automation Testing • •Non-Functional Testing Non-Functional Testing Tools Tools Operating Model Operating Model Organization Organization • •Test Management • •Project Structuring • •Organization Structure Test Management Project Structuring Organization Structure • •Performance • •Staff Assignment • •Skills Mix Performance Staff Assignment Skills Mix • •Automation • •Work-load Management • •Training Programs Automation Work-load Management Training Programs • •White Box • •Organizational SLAs • •Professional Certifications White Box Organizational SLAs Professional Certifications • •Requirements Management* • •Career Path Requirements Management* Career Path • •Templates && Other Tools Templates Other Tools * Linked to overall development methodology and lifecycle. Cognizant Confidential 6
  • 7. Consolidation Of Software Testing Decentralized Software Testing Consolidated Quality Groups Project Team , Project Team , Project Team , Project Team , Project Team , Project Team , Department, or Department, or Department, or Department, or Department, or Department, or Business Unit Business Unit Business Unit Business Unit Business Unit Business Unit Evolution Development Development Development Development Development Development Team Team Team Team Team Team Testers Testers Testers Consolidated Quality Group • Testing teams aligned by project, department, or • CQG organization operates across the enterprise business unit. and supports individual teams/groups on a scheduled work queue basis. • Varied methodologies, processes, tools, infrastructure and metrics across groups. • Consistent methodologies, processes, tools, infrastructure and metrics across groups. • Limited or non-existent software testing SLAs established across the enterprise. • SLAs established and performance metrics measured throughout.. • Inefficient management of resources – people and infrastructure. • Leverage resources across teams/groups and manager work. Cognizant Confidential 7
  • 8. The Move To Of Offshore Testing § Independent Verification & Validation is now offered by all major offshore service providers. § Represents fastest growing business within Cognizant, and an increasingly larger portion of the offshore IT spend. § Many organizations have piloted offshore testing efforts over the past three years and are now embracing across the enterprise. § Key drivers: - Achieve cost savings - Leverage mature processes - Establish better controls, metrics, and SLAs - Gain access to expertise (performance testing, automation testing, test management, etc.) - Flexible resource management - Opportunity for 24X7 Testing Cognizant Confidential 8
  • 9. Managed Test Centers: The Marriage Of Onsite/Offshore With Centralized Quality Groups Integrated Governance Model CQG MTC Rationalized: Functional Performance • Processes Dedicated Resource Pool • Methodology (Core Team) • Infrastructure • Tools • Operating Model • Organization Automation Process Flexible Resource Pool (“Flex” Team) ü Standard operating model with structured SLAs ü Continuous process improvement Critical ü Optimized onsite/offshore model ü Common automation frameworks Features & ü Rigorous metrics collection and reporting ü Shared infrastructure and tools Benefits ü Common process and methodologies Cognizant Confidential 9
  • 10. Managed Test Center: An Example Scenario The Current State Scenario § An organization has a 50 person testing team, comprised of 25 FTEs and 25 contractors. § Fully loaded costs and benefits for an organization's testing team members are $100,000/yr. § Annual voluntary attrition is 10%, with the average cost of replacement cost equivalent to 3 months salary and benefits.. § The organization has implemented some automation – with about 20% of functional testing activities automated. The Future State Scenario § The organization would retain 10% of it’s internal and contract resources for testing. There would be a 10% reduction in force, with the remaining 80% of internal FTEs reassigned to other tasks within the organization. § Approximately 70% of the Managed Test Center team would form the “core” team, while an additional 30% capacity (the “flex” team) would ramp up cyclically every four months. § The onsite/offshore ratio would fluctuate between 10 – 15% onsite vs. 85 – 90% offshore depending on current release cycle demands. § The knowledge transfer and piloting phase would last three months, with a ten person team (6 onsite, 4 offshore). § By the end of the third year, the Managed Test Center will have achieved an overall 15% improvement in efficiency. § Total resource requirements would drop year-over-year based on automation and efficiency gains. Cognizant Confidential 10
  • 11. Managed Test Centers: A Three Year Labor Cost ROI Scenario Year 1 Year 2 Year 3 Total Current State Internal Testing Labor Costs Testing Team FTE $ 2,500,000.00 $ 2,500,000.00 $ 2,500,000.00 $ 7,500,000.00 Testing Contractors $ 2,016,000.00 $ 2,016,000.00 $ 2,016,000.00 $ 6,048,000.00 Voluntary Attrition Costs $ 62,500.00 $ 62,500.00 $ 62,500.00 $ 187,500.00 TOTAL CURRENT STATE COSTS $ 4,578,500.00 $ 4,578,500.00 $ 4,578,500.00 $ 13,735,500.00 MTC Future State Internal Testing Labor Costs Testing Team FTE $ 812,500.00 $ 250,000.00 $ 250,000.00 $ 1,312,500.00 Testing Contractors $ 655,200.00 $ 201,600.00 $ 201,600.00 $ 1,058,400.00 Voluntary Attrition Costs $ 20,312.50 $ 6,250.00 $ 6,250.00 $ 32,812.50 Internal Separation Costs $ 20,000.00 $ - $ - $ 20,000.00 Internal Testing Costs $ 1,508,012.50 $ 457,850.00 $ 457,850.00 $ 2,423,712.50 Cognizant Labor Costs Onsite $ 776,067.83 $ 517,208.33 $ 507,873.24 $ 1,801,149.40 Offshore $ 1,499,693.83 $ 1,398,249.22 $ 1,369,025.28 $ 4,266,968.33 Cognizant Labor Costs $ 2,275,761.66 $ 1,915,457.54 $ 1,876,898.52 $ 6,068,117.73 TOTAL MTC LABOR COSTS $ 3,783,774.16 $ 2,373,307.54 $ 2,334,748.52 $ 8,491,830.23 Labor Cost Savings MTC Cost Savings Breakdown Direct Labor Costs $ 586,644.40 $ 1,402,819.42 $ 1,429,820.71 $ 3,419,284.53 Incremental Automation Benefits $ - $ 393,724.80 $ 393,724.80 $ 787,449.60 Efficiency $ 208,081.44 $ 408,648.24 $ 420,205.97 $ 1,036,935.65 TOTAL ANNUAL SAVINGS ($) $ 794,725.84 $ 2,205,192.46 $ 2,243,751.48 $ 5,243,669.77 TOTAL ANNUAL SAVINGS (%) 17% 48% 49% 38% Cognizant Confidential 11
  • 12. Managed Test Centers: A Three Year Labor Cost ROI Scenario Managed Test Center: 3 Year ROI Scenario Average Annual Staffing Levels 60.0 50.0 40.0 Resources 30.0 20.0 10.0 0.0 Year 1 Year 2 Year 3 Client FTEs Client Contractors MTC Onsite MTC Offshore Cognizant Confidential 12
  • 13. Managed Test Center Case Study: Fortune 50 Global Conglomerate Challenge Fortune 50 conglomerate with decentralized, global IT operations sought to reduce operating costs while increasing efficiency and consistency within it’s software quality assurance organization. Executive leadership mandated continuous process improvement, but environment lacked processes or tools to manage. Solution Establish an offshore Central Quality Group to operate on deliver software testing services organization structured around internal SLAs. Results • Built a dedicated 50 person CQG to operate within enterprise and project-level SLAs. • Established well defined software testing processes; developed process road maps, estimation models, automation standards etc. • Reduced post-production defects through structured requirements analysis and full traceability to test cases. • Reduced regression test cycle times by automating test cases using Rational Robot. • Increased visibility to testing and development activities through comprehensive metrics collection and reporting. Cognizant Confidential 13
  • 14. MTC Case Study: Approach & Ramp Q1 Q2 Q3 Q4 ü Application 1 ü Application 4 ü Application 12 ü Application 20 ü Application 2 ü Application 5 ü Application 13 ü Application 21 ü Application 3 ü Application 6 ü Application 14 ü Application 22 ü Application 7 ü Application 15 ü Application 8 ü Application 16 ü Application 9 ü Application 17 ü Application 10 ü Application 18 ü Application 11 ü Application 19 Pilot Project CQG Case Study: Resource Ramp CQG Case Study: Resource Ramp • Validates and 50 tunes process 50 45 model. 45 40 40 35 • Builds experience Resources 35 Resources 30 with tools. 30 Onsite Onsite 25 25 Offshore • Provides 20 Offshore 20 opportunity to 15 15 refine metrics. 10 10 5 5 • Builds SMEs in 0 0 domain and Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 technology. Cognizant Confidential 14
  • 15. MTC Case Study: Process Model Process Model Quali ty Re port Post Launch Scale Up / Test C Archive Launch e Ca Deliverables Next (Base line & Establish Ideas / Release regression Automation) se se Analysis & UAT Automatio U T n analysis Request Information for service gathering Si n O Sign O Test evaluation Knowledge Dev’t / ff ff Concept / acquisition Develop Requirements Test execution Qu Analysis & ali ty estimation Pla Test design and n& Test development Sig Req planning ff n O uire ignO ff me nts eS Feasibility as eC / Design Us Cognizant Confidential 15
  • 16. MTC Case Study: Benefits Productivity Quality Cost • 24*7 testing model leading to • Assurance of Business • Immediate savings in reduction in test cycle time Requirements software testing costs • Automated Regression Testing • Proven and unique • Improvement in delivery of methodology for Testing Business Requirements • Quick ramp-up and ramp-down of resources • Combination of automation • Save client costs and time tools by internally managing • Perfectly synchronized onsite- resources offshore setup • Continuous Process Improvements • Flexible staffing based on • One Stop shop for providing resource load variations dedicated resources for various • High test case coverage CQG technology needs • Higher cost reduction due • SEI – CMM Level 5 and ISO to low onsite ratio • Skilled Rational automation 9001:2000 certified processes resources • Compressed time to market - early returns on IT investments • High automation gains Cognizant Confidential 16
  • 17. MTC Case Study: Automation Benefits Overall Automation Effort Savings % Effort Savings 100% 90% 44% 80% 56% 70% Automation Manual Effort Savings Key Highlights Automation Vs Manual • 56% Automation 450 400 • Rational Tools used for automation 350 • Traceability Matrix maintained Person Hours 300 250 • Module Centric Approach followed helps to 200 minimize maintenance effort 150 100 • Reusable components to be used across projects 50 • Common Standards 0 App 1 App 2 App 3 App 4 App 5 App 6 • Trained & Certified Professionals Manual test execution effort Automated script execution • Time & Cost savings during Regression testing Cognizant Confidential 17
  • 18. MTC Case Study: Sample Portfolio Metrics (Quarterly) Project Effort Schedule Defect Density Productivity Productivity Load Review Testing Variation' Variation (Defects/Hour) (TC (TC Factor Efficiency Efficiency Creation)- Execution) TCPs/Hours TCPs/Hour Application 1 0.0 -12.0% 0.470 NA 2.740 1.0 0.0 90.3% Application 2 0.0 0.0% 0.028 16.567 8.283 0.4 0.0 96.5% Application 3 0.0 0.0% 0.000 15.400 8.680 0.0 0.0 100.0% Application 4 -0.1 22.0% 0.000 18.400 11.032 0.9 0.0 100.0% Application 5 0.0 0.0% 0.000 N/A 6.120 0.0 0.0 100.0% Application 6 0.0 0.0% 0.000 N/A 8.750 0.8 0.0 N/A Application 7 0.0 0.0% 0.800 NA 0.970 0.7 0.0 15.3% Application 8 0.0 0.0% 0.014 18.000 9.815 0.4 0.0 100.0% Application 9 2.3 9.1% 0.013 7.925 20.410 0.2 0.0 92.9% Application 10 0.0 0.0% 0.003 N/A 19.732 0.7 0.0 90.9% Application 11 1.2 0.0% 0.150 N/A 9.230 1.0 0.1 90.3% Application 12 1.5 0.0% 0.152 2.243 3.180 0.3 0.1 94.0% Application 13 -0.1 -4.0% 0.043 5.538 3.512 0.8 0.1 88.9% Application 14 0.0 0.0% 0.000 NA 2.060 0.3 0.0 NA Application 15 -0.1 0.0% 0.043 9.875 3.511 0.7 0.0 100.0% Application 16 -0.3 NA 0.163 11.111 6.897 0.2 0.1 100.0% Application 17 0.0 0.0% 0.000 NA 7.000 1.0 0.0 NA Application 18 0.0 N/A 0.000 6.000 2.353 0.4 0.0 100.0% Application 19 0.2 0.0% 0.000 N/A 11.885 0.5 0.0 100.0% Application 20 0.3 0.0% 0.000 16.667 N/A 0.4 0.0 N/A Application 21 -0.2 0.0% 0.088 18.500 7.873 0.2 0.0 100.0% Application 22 0.9 N/A 0.000 N/A 3.229 0.3 0.0 100.0% Application 23 0.0 N/A 0.000 N/A 12.222 0.4 0.0 100.0% Cognizant Confidential 18
  • 19. Cognizant’s Testing Services Cognizant Confidential 19
  • 20. Cognizant’s Testing Services Cognizant Testing Services if a truly independent Cognizant Testing Services if a truly independent verification and validation service of Cognizant verification and validation service of Cognizant Technology Solutions focused exclusively on Technology Solutions focused exclusively on supporting your software testing needs. supporting your software testing needs. Functional Automation Process Performance Cognizant Confidential 20
  • 21. Cognizant Testing Services: Service Offerings Independent • Reduced costs through effective use of outsourced resources. Functional • Reduced testing cycles with predictable, repeatable results. Testing • Access to deep resource pool for peak test demands. • Greatly reduced testing cycle times – especially regression. Test • Reduced costs and effort. Automation • Supports iterative development and higher frequency of releases. Test • Higher system quality – fewer post-production defects Process • Increased predictability in test effort estimation & execution Consulting • Greater efficiency and resource utilization across the enterprise • Catch performance bottlenecks prior to production release Performance • Identify gaps in non-functional requirements definition Testing • Ensure system can grow to meet future demand Cognizant Confidential 21
  • 22. Managed Test Centers: Cognizant’s Comprehensive Testing Solution Offering Integrated Governance Model CQG MTC Cognizant Functional Performance Testing Services Capabilities Dedicated Cognizant Resource Pool • Processes (Core Team) • Methodology • Infrastructure • Tools • Operating Model Automation Process Flexible Cognizant • Organization Resource Pool (“Flex” Team) ü Standard operating model with structured SLAs ü Continuous process improvement Critical ü Optimized onsite/offshore model ü Common automation frameworks Features & ü Rigorous metrics collection and reporting ü Shared infrastructure and tools Benefits ü Common process and methodologies Cognizant Confidential 22
  • 23. Cognizant’s Testing Services Organization Is Built Upon A Robust Foundation Independent Test Automation Test Process Performance Functional Testing Consulting Testing Skilled & Experienced Professionals Proven Testing Methodology Best of Breed Tools State-Of-The-Art Infrastructure Vertical Industry Depth The Foundation Of Cognizant’s Testing Services Cognizant Confidential 23
  • 24. Skilled & Experienced Testing Professionals Cognizant’s Testing Services organization is built upon a team of over 1000 software testing professionals who are dedicated to a career in testing and QA. • Active Certifications program: • Presentations in International Testing conferences • CCP, CSTE, CSQA • Experience in Business Domain • RCP (Rational), CPS (Mercury) • Executed over 1000 person years of • Domain testing Type Of Testing Projects Tools Skilled Executed Professionals Functional 200+ Various 800+ Automation 140+ WinRunner 260+ 125+ Robot 300+ 35+ QA Run 80+ 50+ Silk Test 280+ Performance 120+ LoadRunner 140+ 60+ Silk Performer 110+ 25+ Web load 80+ 30+ Rational 60+ Cognizant Confidential 24
  • 25. Thank you 2005