Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Cost of Quality How to Save Money


Published on

EXTENT Conference October 2011 - Test Automation for Trading Systems

Presentation by Sibgatullina Taya
QA Solution Manager
HP Software CEE

Published in: Technology, Business
  • Be the first to comment

Cost of Quality How to Save Money

  1. 1. Cost of Quality How to Save moneySibgatullina TaisiyaSolution Consultant, CEE
  2. 2. Part 1: Quality assurance2 HP Application Lifecycle Management & HP Quality Center 11.00
  3. 3. Cost Of Quality3 HP Application Lifecycle Management & HP Quality Center 11.00
  4. 4. Application Lifecycle model4 HP Application Lifecycle Management & HP Quality Center 11.00
  5. 5. ALM5 HP Application Lifecycle Management & HP Quality Center 11.00 5
  6. 6. Market Leading SolutionsIDC Worldwide Automated Software Quality Revenue by Vendor in „09 “As the market revenue leader, with a 41.1% share, HP is in a position to disproportionately influence the overall ASQ market.” Others HP (41.1%) (Combined) 32.1% Empirix 1.9% Borland 2.9% Compuware 5.3% Microsoft IBM (Rational) 5.6% 11.1% Source: Melinda Ballou IDC, “Worldwide Automated Software Quality 2009-2013 Forecast Update and 2008 Vendor Shares: Evolving Quality in a Challenging Market” (Doc # 219695E) October 2009
  7. 7. HP ALM #1 AGAINNew Gartner report just published in January 20117 ©2009 HP Confidential
  8. 8. HP ALM 11.00 is a Unified PlatformDesigned for the End-to-End Application Lifecycle Application Lifecycle ManagementIntegrations into Strategy and Planning Project Planning & Tracking Enterprise Release Management Integrations into Run (Deployment, APM, ITSM) Requirements Development Quality Management Management Management (PPM, Gov) IDE, SCM, Defect Secure Requirements Requirements Build Mgmt Develop- Functional Performance Security Definition Management Integrations ment Foundation Process Standardization Extensibility Traceability (i.e. Waterfall, Agile) Reporting (Open APIs) Unified for the Core App Lifecycle, Integrated with the Complete App Lifecycle
  9. 9. HP Application Lifecycle ManagementA single platform for managing the application lifecycle Dashboard HP Application Lifecycle ManagementProject Planning Release Requirements Business Test Plan Test Lab Defect & Tracking Management Management Models Management HP Unified Functional Performance Optimization Application Business Application Services Testing Security Testing ValidationHP QuickTest HP Service HP Performance HP HP Business Process Service Test HP QAInspectProfessional Test Center LoadRunner Testing Management Test Data Management ALM Platform Process StandardizationSingle Data Repository Asset Sharing & Reuse Reporting Open APIs (REST) (i.e. Waterfall, Agile)
  10. 10. Track & measure project milestones and KPIsProject Planning & Tracking Defects High 10 Defects Med 25Requirements Test Auth 93% Defects High 45 Progress Test Exec 98% Defects Med 95 Tests Passed 89% Defects High 0Req Reviewed 92% Test Exec 78%Req Approved Tests Passed E2E Testing Defects Med 15 Development 75% 90% Test Exec 98% Progress Testing Tests Passed 92% Req Implemented 85% Progress Readiness Tests Reviewed 95% Assessment DevelopmentRequirements Test Integration User Authoring Execution E2E Testing Acceptance Test Authoring Testing Functional, Performance and Security10 HP Application Lifecycle Management & HP Quality Center 11.00
  11. 11. Requirements Management Integrated solution for defining, managing and tracking requirements– Business analyst can collaborate on application requirements without fear of overriding other‟s work– Assess overall risk to the business through real- time visibility of test coverage and associated defects against business needs– Assess the impact of change before changing requirements– Easy adoption by Business Analysts– Define and track multiple requirement types– Establish templates to ensure higher quality requirements– Manage requirements dependencies– Bi-directional traceability from requirements to requirements, tests and defects– Manage change requests through formal approval process– Familiar rich test editor interface– Version Control and Baselining– Leverage existing assets in MS Word 11
  12. 12. Baselining Track changes made to entities in your project over time– Compare releases and cycles– A snapshot of the project state at a particular point in time– Enable rollback of assets to key points in the application lifecycle– Compare baselines to assess changes– Ability to use baselines in test execution– Create baselines – including traceability between assets– Compare baselines both at high level and drill down to specific changes including traceability– Report and print baselines for sign-off– Control who can create and modify baselines 12
  13. 13. Release & Cycle Management Track and measure the progress and quality of releases and cycles– Manage projects and assets against release and cycles– Track release progress– Gain visibility into release readiness and release quality– Compare releases and cycles for continuous process improvement– Manage requirements & tests according to application releases and dev/QA cycles– Compare planned vs. actual progress for each application release and cycle– Assess release quality across multiple cycles 13
  14. 14. Project Planning & Tracking Track and measure project milestones and KPIs– Visibility into application projects progress and status– Visibility into key project milestones– Identify delays or poor quality before it impacts the schedule– Proactively adjust plans when a project is not on track– Define and track project milestones– Define and measure key performance indicators across the application lifecycle– Automatically updates project status against milestones– Drill down on KPIs for more details 14
  15. 15. Test Plan Define, manage and track all test scripts– Create test cases to adequately test the requirements– Manage all test types (Functional, Performance, Security) in one place– Ensure requirements coverage through traceability to requirements– Manage change associated with requirements changes– Design, build and manage manual and automated test cases– Link test cases to requirements– Leverage test cases from MS Word and MS Excel– Version Control and Baselining 15
  16. 16. HP Sprinter The next generation of manual testing– Accelerate manual testing– Improved testing accuracy– Better communication of defects to development (accelerated defect remediation)– Unobtrusive, easy to use user interface– Supports exploratory testing– Reduce time and resources by testing multiple environments at once– Toolkit – Screen capture and annotations, measure objects position etc.– Record user actions to support exploratory testing– Automatic data injection– Mirror testing– Log defects with associated information directly into HP Quality Center/ALM 16
  17. 17. Defect Management Track defects across the entire application lifecycle– Manage, track and enforce defects across the application lifecycle– Create defects from manually or directly from the execution of manual and automated tests– Clearly communicate bugs to developers with context from requirements and tests– Integrated into developers IDE– Enforce defect lifecycle process through custom fields and workflow– Log defects directly from test runs with test details (test step, result, screenshots, attachments) to ensure the developers can reproduce the issue– Traceability to requirements, tests and other defects– Alert key stakeholders through automated emails 17
  18. 18. Reports, Graphs & Dashboard Clear communication and visibility into your application projects– Make informed business decisions– Clearly communicate to all stakeholders– Understand the real-time status of an application– Full trending analysis and insight into application projects– Understand release and cycle progress– Fully customizable project reports (export to MS Word)– Integrated Dashboard– Out of the box or customizable analysis reports and graphs– Report to MS Excel– Embed graphs and charts into web portals– Requirements traceability matrix 18
  19. 19. Development tools integration©2010 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice
  20. 20. Working with defects in Developervarious tools• Traceability between Quality Center Other tools requirements, code changes and defects Release & JIRA DEMAND• Adjustable processes Requirements STRATEGY/ Strategic• Side-tools integrations Test Plan BugZilla demand • New applications • New services StarTeam • Application Test Lab integrations Operational demand CCE Tracker Defects VersionOne
  21. 21. Development tools integrations• Tasktop & • CollabNet • CollabNet CollabNet • AccuRev • Hudson • Eclipse & Eclipse- • Perforce • Electric Cloud based clients • • CVS AnthillPro • Visual Studio • ClearCase
  22. 22. HP Application Lifecycle Intelligence Waterfall Scrum TFS Hudson TFS SVN RUP Hybrid HP ALM Req Build Test Electric Jenkins SVN CVS Cloud Single Platform
  23. 23. The Decision Support System for ALM… Waterfall Scrum TFS Hudson TFS SVN RUP Hybrid HP ALM Req Build Test Electric Jenkins SVN CVS Cloud Single Platform
  24. 24. Key ALI Features Code changes from Subversion Code changes from TFS NewModule Drill down
  25. 25. Key ALI FeaturesEnd-to-end appdelivery Report
  26. 26. HP ALM 1 and TFS integration 1 Work ItemsDefects Reqs Bugs & TFS adapter for QC Reqs Synchronizer 1.4QC TFS
  27. 27. Part 2: Performance validation28 HP Application Lifecycle Management & HP Quality Center 11.00
  28. 28. HP Performance CenterSolutions for management and automation of performance optimization HP Performance Center Family Performance Testing: LoadRunner / Performance Center VuGen Controller Load Generator Monitors Analysis Management Framework: Performance Center User/privilege Infrastructure Central Global access & Requirements & Resource pooling management management repository collaboration Defects & scheduling Diagnostics J2EE .NET SOA SAP Oracle
  29. 29. HP LoadRunnerThe market leading solution for performance validation Reduce risk of Enables a consistent, deploying systems that do repeatable process to: not meet performance requirements • Emulate production workload to test a broad range of applications Reduce cost by predicting scalability and • Identify performance capacity and eliminating over-deployment bottlenecks, diagnose the root cause of issues • Optimize application Reduce time by shortening test cycle time. performance before Accelerate delivery of deployment high quality applications
  30. 30. The challenge: Scripting processCurrent scripting process Record a Parameterize Add Correlations Add logic script data transactions– Sequential process takes time– Scripting expertise needed (especially for complex apps)– Protocol level scripts break with minor app changes
  31. 31. The challenge: Scripting for web 2.0 appsNew Web 2.0 technologies Web Data Client Server App Server Web Server Cloud Server Services Networks Web EDW App Data Server Server Web Server Web Services Server– Many technologies, frameworks and toolkits– Very customizable – almost no “vanilla” usage– Dynamic, asynchronous behavior– Client-side processing and logic– Most testing solutions do not support all the customizations
  32. 32. LoadRunner VuGenRecord an application business process into an automated scriptAdd end-user transactions, parameterize input data, add verificationcheckpointsApply user behaviors such as network connectivity or pacing Business process Browser Application Scripts VuGen Recorder
  33. 33. LoadRunner Controller Design and execute load scenarios that mimic production conditions Emulate hundreds or thousands of concurrent users Gather end-to-end transaction response times Collect application, network and system data across all architectural tiers and components Diagnostics Controller Load Network Load Web App Database generators balancer server servers Europe VUsers US Monitors
  34. 34. INTRODUCING: LoadRunner TruClientThe revolutionary way of scripting modern apps Scripting engine embedded within browser No correlations required! Data can be parameterized while scripting Interactive, dynamic Transactions, and scripting! logic can be added while scripting Supports all Ajax Automatic applications suggestion of automated steps
  35. 35. Monitoring and DiagnosticsDiagnose performance bottlenecks Key benefits • Monitor system performance during tests • Diagnose performance bottlenecks as they occur • Identify application code level problems • Resolve issues faster Key capabilities • Sitescope monitoring for system performance metrics • Diagnostics to drill down to the method level • Data available during run time as well as for post-test analysis36 6 November 2011  LoadRunner  Performance Center
  36. 36. Extensible PlatformCustomize and extend the industry-standard platform Key benefits • Build niche/custom protocols • Export analysis data for HP  Partners --> •HP supports 46% of protocols customers need. custom reports •New Demand = 30% of current usage • Include custom monitoring •25% of customers requested AJAX, RDP and VoIP •Smaller demand for multiple protocols sums up to and diagnostics big opportunity Key capabilities • Custom protocol development via a SDK • Analysis API to export data for custom reports • Extensible monitoring and diagnostics framework37 6 November 2011  LoadRunner  Performance Center
  37. 37. Performance Center 11.0Performance Requirements & Defects, Coverage reports Requirement Performance coverage Requirement Requirements statusNew PC Lifecycle User to access performance requirements and defects
  38. 38. Performance Center 11.0Next-gen platform: based on the ALM platform Performance Performance Performance Performance Lab Mgmt Requirements & Test Testing (scheduling) Defects Management  Collaboration  Admin Console  Authentication  Traceability  Users & license  Authorization  Reporting  Projects  Workflow  CoE  Version Control  APIs, Integrations ALM Platform Performance Center
  39. 39. Performance Center 11.0Topology Drag & drop from Define machine list to build your names and topology diagram properties Right click to set up monitors
  40. 40. Performance Center 11.0Miscellaneous– Timeslots • Redone in Outlook fashion • More information available regarding booked timeslots • Allows for maintenance windows Shows when you can start a test based on your requirements
  41. 41. Performance Center 11.0Timeslots Gives information on used timeslots
  42. 42. Performance Center 11.0Miscellaneous– Timeslots • Redone in Outlook fashion • More information available regarding booked timeslots • Allows for maintenance windows Shows when you can start a test based on your requirements
  43. 43. Performance Center 11.0Timeslots Gives information on used timeslots
  44. 44. Performance Center – fully integratedinto the application lifecycleA repeatable quality management process mitigates risk Align with management and stakeholders STRATEGY/ REQUIREMENTS RISK-BASED TEST MANAGEMENT Go/ No Go OPERATIONS DEMAND MANAGEMENT TEST PLANNING AND EXECUTION Business Create manualStrategic demand Connect to production Connect to production requirements test cases Integrate with demand Execute functional• New tests applications Assess and Automate• New services Functional regression test• Application requirements Analyze risk cases integrations Establish Operational Operational Performance testing Identify and security demand requirements Execute security priorities customize security management scans policies• Defects• Enhancements Security Create Production• Change requests requirements test plans Create monitoring Execute tests, performance Enterprise diagnose and scripts and Architecture and resolve problems Other non- scenarios Policies functional Service desk requirements DEFECT MANAGEMENT• SOA• Security4545 6 November 2011 6 November 2011 Collaborate with design and development teams
  45. 45. Part 3: HP UFT
  46. 46. Part 3.1 Functional Testing and Service Test
  47. 47. A Modern testing team has new needsPeople• Expand skills to test beyond the GUI• Extend testing to more members of the team• Increased collaboration with developers and business analystsProcesses• Automate as early as possible• Accommodate faster changeTechnology• Complex integrated systems• Test GUIs, APIs, web services and vertical integration• Test RIA, Web 2.0
  48. 48. Late stage testing is too expensive…Iteration 1 Iteration 2 Iteration 3 TestWhen it was injected way back here? Why wait to find a bug here? Cost to Fix Remember, the age old QA principle… Time between injection and identification
  49. 49. Test & automate as early as possible C1 C1 C1 C1 C1 C1 C1 C2 C2 C2 C2 C2 C2 G1 G1 G1 G1 G1 C3 C3 C3 C3 … … Automate tests when G2 G2 G2 available components become stable G3 G3 BP1 BP2 Regression test at the end of each iteration or as part of continuous build C GUI-less G GUI BP business stable component element process component
  50. 50. Modern applications have many layers End to end Process Layer business A new transaction process Graphical Web 2.0 Billing SAP Exchangeuser interface Screen 1 (Ajax) Screen 2 (Java) Screen 3 (.NET) Screen 4 (Ajax) Screen 5 Screen 6 (Java) (.NET) layer Headless Component 1 Component 2 Component 3 Component 4 Component 5 Component 6 (.Net API) (Java API) (web service) (JMS service) (web service) (REST service) LayerInteract with modern GUI interfaces, Web 2.0Continue the scenario to the headless layer; test the integration between the layersReport on the quality of the entire end to end process
  51. 51. HP Software solutionsEnd to end Process Layer business A new transaction process Graphical Web 2.0 Billing SAP Exchangeuser interface HP Unified Functional Testing Screen 1 (Ajax) Screen 2 (Java) Screen 3 (.NET) Screen 4 (Ajax) Screen 5 (Java) Screen 6 (.NET) layer Headless Component 1 Component 2 Component 3 Component 4 Component 5 Component 6 (.Net API) (Java API) (web service) (JMS service) (web service) (REST service) Layer HP Unified Functional Testing includes the ability to test the GUI layer + the Headless layer + the end to end process.
  52. 52. What‟s new withHP Unified Functional Testing 11.0? Unified Functional Testing HP Functional Testing HP Service Test Integration (HP QuickTest Pro + all add-ins) Call activities from one product to the other Transfer complex data structures Unified report
  53. 53. Report for a Unified Functional Testing scenario Shows activities from both HP Functional Testing & HP Service Test Multiple views of the flow Full traceability for easier debugging
  54. 54. Testing Web 2.0 GUIs OOB Web technology support • Ajax toolkits including Dojo, GWT, YUI • RIA such as Flex, Silverlight v2.0, v3.0, (v4.0 soon), WPF extensibility • New support for Firefox Community • Wizard-like tool helps users extend support to more web 2.0 controls • Partners and customers can share code created to customize support for add‟l toolkits
  55. 55. Opening web extensibility to the online community: The Extensibility AcceleratorEasy, wizard-like interfaceExtends HP Functional Testingfast; uses IntellisenseAutomated deploymentFree for anyone? Yes, FREE!
  56. 56. HP Service Test 11.0 - new Visual Test Designer Modern, intuitive test design interface Reusable modules improve maintainability Visual test designer extends testing to non-developers Flexible and extensible service11/6/2011 processing
  57. 57. BPT Positioning What is BPT?Business Process Testing is a framework for QA organizations to build andreuse structured testing components in both manual and automated testing. Target usersNon-technical QA personnelSubject Matter Experts BPT is ideal toBuild frameworks to maximize reuse of testing componentsMitigate risks by aligning business requirements with end-to-end testscenarios
  58. 58. Part 3.2 BPT
  59. 59. BPT 1 1.0Simpler. Faster. Better.
  60. 60. What‟s new in BPT 11 Significantly improved performance Built an all new simple and intuitive user interface Provided the ability to leverage QC 1 Capabilities 1.0 Enhanced Traceability via Component Criteria Rapid Manual Testing with Sprinter Integration
  61. 61. BPT 11 Exponential Performance Boost Performanc e Number of components More Components = Better Performance
  62. 62. Test Script view – New look & FeelCleaner UI showing all BPT Resources and Dependencies Clean and Simple Iteration View •Tree structure instead of a flat list • Improved Iterations dialog (multi- iterations view & Single Iterations view) • new ‘Comments’ column for each Component instance in the test • Search option inside the Iterations dialog
  63. 63. One-step Component Creation ProcessSimplified the component creation process from three steps to just onestep
  64. 64. Enhanced Drag and Drop FeatureProving the ability to drag and drop multiple test components into the testscript area
  65. 65. Summary of Key Enhancements and Features• Major performance improvements with a redesigned execution engine• All new BPT UI built using QC web toolkit for seamless look and feel• Simple and easy to understand all new test scrip view• Ability to create a manual testing component in just one step• Ability to drag and drop multiple components into test script pane• Enhanced test iteration view to show different combinations• Cleaner look and feel of QC resource dependencies in BPT• Application area is a standard test resource in BPT• BPT integration with QC Sprinter• Enhanced traceability to map requirements to component level
  66. 66. HP Lifecycle Solutions & Operating Principles PLAN DELIVER RUN RETIRE COMPLETE LIFECYCLE Project & HP ALM Business Records Modular COVERAGE Portfolio Availability Management Management Quality Integrated Center Deployment Archiving Automation Heterogeneous Perform Center SOA Systinet Best-in-class App. Service Security ManagementCORE LIFECYCLE Center EXCELLENCE HP APPLICATION LIFECYCLE MANAGEMENT Project Planning & Tracking Enterprise Release Management Traceability & Metrics Single Platform Requirements Development Quality Management Management Management Simplified Web Access Versioning & Link to Defect Secure Definition Change Control IDE, SCM Mgmt. Code Function Perform Secure Technology Agnostic Powerful Automation ALM Foundation Process standardization Reporting Extensibility Quality at the Center
  67. 67. Why HP? Integrated management & automation: instead of loosely coupled point tools, HP provides a cohesive platform for lifecycle management and automation. Technology & methodology agnostic: broadest support for heterogeneous environments (.NET, Java, SAP, Oracle, etc.) – covers more than 70 environments. Covers traditional and Agile methods in single solution. Project ready, enterprise scale: whether for a team of ten, or an enterprise of tens of thousands, HP solutions offer proven configurability and scalability. Complete application lifecycle: no other company offers the integrated product coverage of HP to support applications from beginning to end.