From Requirements to high quality deliverables - Visure Solutions & Wind River
Software Test Optimization:Reducing the cost of testing, using arequirements-driven, change-aware processAn Integrated Solution from Wind River and Visure SolutionsFernando Valera, Alliances Manager– VisureIdo Sarig, VP Marketing – Wind River
Our View of the Market: Market Pains 1 Trend 1: Growing Complexity of Software and Product Variants 50 45 •Key issues: •Relative growth of software, increase 40 of 33% compared to mechanics and 35 electronics Millions lines of code •Complexity of product lines and 30 2006 large-scale systems 2007 •Examples: 25 2008 2009 •Aircraft carrier, 1,000,000 20 2010 requirements 2011 •Hospital information system: 90,000 15 2012 requirements 10 •Train control system: 14,000 contract requirements 5 •Modern airliner: 6,500,000 LOC 0 2006 2007 2008 2009 2010 2011 2012•Ford Motor Company’s Prediction of Growth of Software Content in Automobile embedded software. Calendar years•CPDA, The Requirements Engineering Process - Panel Discussion of Critical Issues in Requirements Management•Nationale Roadmap Embedded Systems, ZVEI, www.zvei.org, 2010•Requirements Engineering: A Roadmap, Bashar Nuseibeh & Steve Easterbrook, 2000
Our View of the Market: Market Pains 2 Trend 2: Missing Connection to Market Needs and Business Requirements Key issues: •< ~52% of requirements go into product release •1% to 3% per month of requirements become obsolete •RDM tools do not seamlessly connect upstream and downstream activities •Lack of communication and visualization of RDM features for scoping & descoping incl. decision support•Nationale Roadmap Embedded Systems, ZVEI, www.zvei.org, 2010•Systems Engineering Vision 2020•Information Technology for European Advancement (ITEA) 2009 Roadmap, http://www.itea2.org/itea2_roadmap_3/step/2, p.18, 3rd Edition•M. Bone, R. Cloutier: The Current State of Model-Based Systems Engineering:Results from the OMG SysML Request for Information 2009
Our View of the Market: Market Pains 3 Trend 3: Requirements Quality as an Essential Driver for Software Product Quality Key issues: ~ 60% of all software defects traceable back to poor RDM practices Changing requirements: Lack of effective change process and tool enforcement Absent formal RDM approach, > 90% of companies use still use natural language approach No defined RDM process 80% of organizations do not use RDM tool at all Connecting adjoint development activities: Testing Design Product risk management Project management•Institute for Experimental Software Engineering (IESE), Fraunhofer Gesellschaft, Survey on Root Causes of Software Defects in MedicalDevices, 2007; http://www.iese.fraunhofer.de/Images/ studie_denger_070507ple_v3 _e_tcm27-7391.pdf•Information Technology for European Advancement (ITEA) 2009 Roadmap, http://www.itea2.org/itea2_roadmap_3/step/2, p.18, 3rd Edition•Chaos Chronicles v 3.0, Standish Group International, 2003, http://www.standishgroup.com/
Our View of the Market: Market Pains 4Trend 4: Compliance with standards and agile approach Software is becoming more safety–critical, and therefore, more regulated: New standards: IEC 62304 ISO 26262 IEC 61508 DO-178B/C DO-254 CMMI SPICE Shifting to different development model V-model and Waterfall to a more agile approach
Visure’s Differentiated Approach Process Organization with mature Requirements Engineering PROCESSES are proven to consistently produce better quality products on time, and within budget
Visure’s Differentiated Approach Collaboration A COLLABORATIVE requirements platform has proven to have a direct impact in the project success bridging the gap between business and IT, communicating distributed teams, integrating clear information between the teams, and establishing workflows
Visure’s Differentiated Approach Product quality QUALITY needs to be introduced straight from the beginning and at every stage of the development, from writing good requirements, keeping changes under control, to reusing high quality assets
Visure RequirementsVISURE REQUIREMENTS INTEGRATESPROCESS, QUALITY & COLLABORATION IN A SINGLE PLATFORM
Visure RequirementsA COMPREHENSIVE PLATFORM FOR VERIFICATION & VALIDATION OFPRODUCT AND EMBEDDED SOFTWARE DEVELOPMENT Unit test Cases
Visure Requirements: CollaborationVISURE REQUIREMENTS ALLOWS USERS TO KEEP USING THEIR OWN TOOLS, WHILE KEEPING EVERYTHING CENTRALIZED
The Testing Process Development Costs Requirements Maintenance 15% 10% Design 10% Test 40% Development 25%
Full Lifecycle TraceabilityQuality Management Needs Strong Linkages to Requirements, Tests and Code User Design Code Test Cases Requirements Specification Bi-directional traceability • Traceability maintained throughout the whole ALM, including WindRiver’s solution • Identify un-tested requirements and code • Measure requirements, code and test coverage • Identify feature creep Understand the impact of change • Identify test suites impacted by requirements changes • Identify what tests to run based on code changes Traceability is the key to compliance 14
Requirements change? Automate where it matters most! Executing all the tests on each change is unrealistic… expensive and inefficient! Alternatively… 1. Highest Return: Execute tests that contribute to high coverage of the requirements, then any others… 2. Quickly Lower Risk: Execute tests for the most critical requirements, then any others… 3. Practically Safe: Execute tests for all the critical requirements, then any others… Especially since often ~20% of the test cases are covering ~80% of the business value No. of Requirements High Requirements coverage Most critical Requirements All critical Requirements Low contribution Test Suites 15
The Value of Wind River Test Management Build 6 Build 7 TC1 TC3 TC2 Compare builds’ binaries to identify Only a subset of test cases are testing the changed, new and deleted functions actual changes in the code Identifies the real Works on unmodified, impact of change fully optimized binaries Maps tests to the code Does not require access they exercise to source code Enables easy fault- Facilitates collaboration HIGH RISK: 85% of functions were not injection tested amongst distributed team members16
Build Automation: Requirement Management: IBM Buildforge, Jenkins, Electric Cloud Excel, Visure Requirements Central repository to store manual and automated test cases Dashboard Virtual lab managerFlexible and views Reports generator Easy and clear interpretation of individual test case resultsPowerfulFrameworkEasily integrated with existingbuild and test infrastructure Defect Tracking: JIRA, Rational, ClearQuest Test results parsed for each test case and each step within STDOUT for each test case step test case captured individually
Wind RiverTest ManagementIdentifies high risk code areas Coverage After running all test cases what areas of the code were NOT tested HIGH RISK: 85% of functions were not tested Map test cases to code Increase preparedness for certification process Performance Measure actual functions response time See impact of change on performance …all on production code. HIGH RISK: Performance degradation between builds
Wind River Build 6 Build 7Test ManagementManages the impact of change Build Differences Compare builds to know the impact of change Compare builds’ binaries to identify changed, new and deleted functions Gain visibility to new code introduced to the build Identify what tests to run to TC1 verify only the changes TC3 TC2 Only a subset of test cases are testing the actual changes in the code …all on binary code.
Wind RiverTest ManagementOptimizes test execution The test suite generator utilizes existing and generated test suites and enables test suite optimization for criteria such as coverage, fixed defect validation, and requirements traceability. What tests will generate maximum coverage if I have only 5 hours to run my tests?
Wind RiverTest ManagementTests edge conditions and errorhandlers Fault Injection Validate fault handlers. Automate fault injection. Do away with complex test scenario setups. Fully validate exception conditions fast. Simulate data input/output. Increase coverage, save time, and improve quality. …all on production code.
Summary:Full lifecycle quality management for embedded SW Requirements Management Quality Management Visure Requirements Wind River Test Management Visure Requirements Wind River Test Management • Dynamic runtime instrumentation, data capture • Integrated requirements traceability and test and analytics coverage • Change-driven, optimized automated test suite • Semantic Analysis generation • Centralized information repository • Open execution engine • Collaboration platform • Host-based and on-board execution • Requirements dashboard and reporting • Virtual lab management