Quality Engineering Helps You Clean Up Your ActBy Roger Nessier, VP, Client Management, Estuate, Inc. - January 14, 2013Trends such as cloud based software delivery and emphasis on uptime create even greater demandson software quality. Independent software vendors (ISVs) and software-enabled businesses mustforego their usual pre-release quality control approaches and instill quality principles into every aspectof product development, from whiteboard to long term maintenance. Users no longer accept thetradeoff of whiz-bang technology for buggy, hard to use software. A recent example of this was therelease disaster of Apple Maps, from a company which normally produces high quality innovativesoftware. Software providers, software enabled businesses, must focus on high quality, secure, andreliable deliverables. The question is how will they address this?Software development organizations must respond to this changing market dynamic by placing agreater emphasis on quality engineering (QE). QE imbues quality principles into every aspect of thesoftware product development lifecycle (PDLC), as well as technology and architecture decisions.The impact of taking a QE approach to software development can be huge. Catching process or designflaws early on reduces the impact and cost of remediating those issues as you get closer to therelease. The most mature QE centric development organizations clearly understand the impact ofquality on their PDLC process and software design. They use this information to ensure that finaldeliverables are of the highest quality.For some organizations this can be a big cultural shift where architects and developers were the powerplayers. In some organizations, QA is a place where those who aspire to development roles attempt toprove themselves, or those who can’t cut it as developers are relegated to lesser roles. In a QE centricorganization the Quality Engineer is an equal to the architect or developer. In communicating thischange, quality should be regarded as the mutual goal for both the developer and quality engineer.Teamwork is the hallmark of a QE organization. The quality engineer should work to help developmentengineers preemptively identify issues and address them early in the development cycle. The qualityengineer needs to be a highly capable engineer that combines the problem solving skills of adeveloper with the eye for quality of a QA engineer.The first step in adopting a QE centric approach to development is to understand your QualityEngineering Maturity Level (QEML). QEML is a framework we have developed to help you betterunderstand where your development organization is in QE adoption and what level of QE adoptionyour organization aspires to achieve. Once you understand your QEML, you set incremental QE goalsover times that are supported by business benefits.What is your Quality Engineering Maturity Level?Feedback mechanism from stakeholders to refine design/process on an ongoing basis. Strong relianceon metrics, surveys and interviews to identify quality trends. Use of trend data to take action tooptimize technology, design and process decisions to meet the changing needs of stakeholders.Leverage predictive mechanisms to better understand weak
links.Recognition that quality extends beyond catching defects early to include QE as part of design anddevelopment processes. Every aspect of development is analyzed and scrutinized to ensure itoptimally aligned towards achieving quality goals. Technology, architecture and design decisions aremade with an eye on quality. Stakeholders are tightly integrated into design and process decisions toensure their needs are met.QE oversight to processes ranging from requirements gathering to long term support. Empower QE to“stop the line” should defect trends show a need to reassess design, process or technology decisions.Extensive use of metrics to understand quality trends. Culture of quality is instituted.QA and development activities are quite coordinated through the PDLC. No large backlog of bugs. Testcases derived from use cases and user stories early in development cycle. Developers responsible forunit testing. Limited use of metrics to better understand “problem areas”.“Throw it over the wall” pre-release QA, little emphasis on unit testing, test cases developed late indevelopment cycle, releases regularly go out with several S1/S2 defects. Heavy reliance on point
releases to improve product quality.Development organizations should assess their QEML from two perspectives: Process and technology.It is possible to rank differently on the QEML scale from a technology and process standpoint. Forexample, you may have excellent development processes with strong stakeholder involvement, butare limited to how well you are able to serve your stakeholders because your applications are built onan inflexible architecture. Conversely, your application may be built on a modern web framework, but,your development processes are immature.Once an organization understands their technology and process QEML, then it sets goals over time forincreasing their QEML. QEML goals should be supported by business cases that have strong linkages tofinancial goals. For example, if no “severity 3” defects in releases doesn’t have a business justificationbecause the application isn’t used that often and users are tolerant of severity 3 defects, then thatshould not be a process QE goal.Organizations need to develop separate technology and process QE plans for each product if thoseproducts are developed using different technology and development processes. It is usuallyrecommended to make process QE improvements before making technology improvements, since yourdevelopment organization will ultimately benefit from those process improvements when makingtechnology changes. It not recommended to try to conduct both technology and process changesconcurrently, or to try to move too rapidly since the shock of these changes could impair yourorganization’s ability to “keep the lights on”.In the example below, a development organization has decided to revamp the development processfor Product A first (Step 1). Once the process changes are completed, the value of the changes will beassessed, tied to business benefit, before deciding to proceed with “Step 2”, improving the technologyplatform.
Development organizations stand to derive the greatest benefit from integrated, holistic process toQE. If a “big bang approach” to Process QE is too daunting, or stakeholders need convincing that thereis a business benefit to Process QE, incremental or point adoption can also be leveraged. The followingare process QE adoptions you may wish to consider.Process QE Test Regime: Use Cases/Test cases should be developed at inception and applied early in the development cycle. Make sure that use cases/test cases are comprehensively documented. With the aid of use cases, comprehensive unit tests should be created and leveraged to ensure developers understand the business issue being addressed and to catch bugs or misunderstandings early in the development cycle. When possible, peer reviews should be leveraged to identify issues. Automated testing reduces testing overhead and may help uncover more issues. A heavy reliance on testing early and frequently eliminates issues before they become serious and expensive to fix. Requirements Traceability: Trace completed requirements back to business needs to ensure the need is met. Iterative and complex developments sometimes lead to the original product mission getting changed or lost. If requirements evolve, then it is important to update business requirements to reflect this evolution. The completed software product and its subsequent releases must always be in sync with the business requirements document. Use Defect Analysis to Identify Root Causes: Finding clusters of defects may point to a particular root cause such as a design flaw. Developers should investigate issue clusters and not just focus on fixing observable defects. Defect analysis should be conducted to track the number and source of defects throughout the development cycle to identify trends and take early action if an endemic root cause is uncovered. Advanced techniques such as Taguchi
methodology identify applications areas that are prone to defects and helps focus testing activities to those areas. This results in fewer test cases to achieve full application coverage. Adopt Agile Processes: Adopting an Agile development approach can help you “test drive” a QE approach. Agile by it’s nature uses smaller teams and requires close collaboration across all disciplines. Each Agile team almost by definition results in a QE approach although it’s restricted to that team and what’s being accomplished in a given iteration. Applying any Agile approach (be it SCRUM, XP, FDD) to development helps build upon early successes and validates design assumptions. Instead of trying to release multi-faceted complex functionality, start simply and leverage market and stakeholder feedback to continually refine and enhance functionality. By building it simply the first time, you eliminate unneeded complex functionality that can detract from the user experience and introduce more issues. Only add in extra capabilities after you feel you have a strong justification to do so. Increase Focus on PSR Testing: Comprehensive performance, scalability, reliability (PSR) testing uncovers “weak links” in the product. PSR should cover all possible uses of the product, using different configurations, hardware/software environments, and data sets. PSR testing should help drive coding standards to ensure that as new functionality is added, PSR does not degrade. Document Processes to Enhance Repeatability and Consistency: Great software products are not created unless development processes are well documented and followed. Whether you follow a classic waterfall or an agile methodology (or something in between), it is important to have total process clarity from the requirements stage to end of life for a software product. Good process understanding is especially important if you have distributed development teams. Always look for opportunities to improve the process, and be sure to update process documents with those improvements. Leverage metrics to understand how well the team is tracking to expectations around productivity, quality and on time delivery. Report on those metrics using a real time dashboard that provides total transparency on the state of a release.Deciding on QEML process goals depends on a variety of factors: Are you taking an integrated onincremental approach? What are the cultural, educational, and business challenges you are facing thatcould impede adoption? What are the perceived benefits (business case) for QE process adoption?What is your expected ROI? How will you measure success?Technology QEDevelopment organizations are often faced with supporting and/or extending several generations ofapplications from Cobol to Java or .NET. Disparate technologies, design and skill requirements all
present challenges to a QE centric development organization. One of the biggest questions manyorganizations face today is: How can I afford to rewrite legacy applications to leverage the latestthinking in architecture/technology, while continuing to meet the needs of the marketplace? Anytechnology QE strategy must be balanced against market and business realities. Like process QE,many organizations take an incremental approach to technology QE, this can sometimes feel likechanging the engine in your car while driving down the freeway. Like process QE, technology QE mustbe substantiated by business benefits. For example, a development organization doesn’t adopt SOAbecause it seems like the technology savvy thing to do, or because developers like to have SOAexpertise on their resume. SOA adoption should be justified by a strong business case. In some cases,SOA may not make sense for your organization. The following are technology QE adoptions you maywish to consider. Reduce Design Complexity: Eliminating complexity and encouraging reusability can help simplify large system design, thus reducing quality challenges. Designs incorporating SOA , object orientation, and partitioning, are more extensible and able to adapt as the product evolves. Avoiding a hard design commitment is preferable if an abstraction serves the same need. Re-factoring code early in the development process will reduce complexity and help uncover flaws. Designing for ease of integration to other products/systems by encapsulating variation reduces 3rd party software integration/testing challenges. Keep external interfaces as stable as possible when changing internal application functionality. Improve Usability/Interactivity: Poor usability/interactivity should be considered a design defect. A rushed, poorly researched approach to UI design will result in users logging issues. These issues cannot be ignored just because the product was “built to spec” and is supposed to have a user interface that is unintuitive. Be cognizant of how the product is used, leveraging use cases and story boards to validate the final design. Take into consideration the different needs of casual users and power users when analyzing usability/ interactivity. Make sure UI designs are intuitive, self-explanatory, and self-documenting. Design for Maintainability: If fixing parts of the application causes an undue amount of collateral damage to other parts of the application, there may be issues with the design. Engineers need to look at memory usage, designs and data access to understand if there are fundamental flaws causing the issues. As well, if porting the application to different platforms/environments, causes an undue amount of functionality to break, there may be issues with the design portability that needs to be addressed. Poor maintainability is sometimes caused by developers who feel they can walk away from a software project after it is released. Developers should realize they their team (or a subset of the team) are responsible for bug fixes and improvements in future releases. Design for Extensibility: If adding functionality causes an undue amount of existing functionality to break, there may be issues with the design. Revisit the product architecture as a product evolves to ensue it meets the needs of product. The adding of new functionality can be constrained by an architecture design that does not anticipate the product evolution. Poor extensibility is sometimes caused by a “project view” towards software development, vs. a long term multi-release view. Take Security Seriously: Proactively identify security holes and look for vulnerabilities before users stumble on them. Conducting a thorough security audit as part of the release process not only uncovers security flaws, but also helps refine coding standards so designs are more secure. Make sure your security goals are aligned with the type of application you are developing/supporting, and how exposed the application is to hackers.
Deciding on QEML technology goals depends on a variety of factors: Are you rewriting all yourapplications or taking an incremental approach? What are the training and business challenges tomaking technology changes? What are the perceived benefits (business case)? Do your customersreally care if your application is written in Java or C++? What is your expected ROI? How will youmeasure success?The QEML planning process should be sequential and iterative. Checkpoints and metrics are used toensure value is being generated. The chart below describes the steps your organization should use toset QEML technology and process goals.Finally, instituting a “culture of quality” is critical across the entire company, not just the R&Dorganization. Develop training programs and make sure stakeholders in the rest of the organizationunderstand the QE mandate of the development organization. For example, sales or managementshould not over commit development, resulting in “death march” releases and the resultant qualitydegradation. Professional Services teams should create customizations or configuration changesleveraging the same QE principles that development uses. Creating a culture of quality that permeates
throughout the entire organization will return dividends in the form of higher customer satisfaction,greater ability to meet market needs, greater customer retention, and ultimately, a healthiercompany. Author Roger NessierRoger provides product development services to software companies and companies that use softwarein their business. He has delivered dozens of projects to a range of clients around the world, usingglobal teams, from gathering initial requirements, to design/build, to long term support. He iscurrently VP of Client Management at Estuate, and has held services management and productdevelopment executive positions at i2 Technologies, Symphony Services, and Steelwedge Software.Roger lives in the Bay area. In his spare time, Roger enjoys running or cycling the hills near his home,or taking a yoga class with his wife.