Download to read offline
Uploaded byTEST Huddle
Isabel Evans - Quality In Use - EuroSTAR 2011
The document outlines Dolphin Computer Access's commitment to improving the quality of its products for individuals with vision impairments, focusing on enhanced testing, metrics, and customer-centric development. Isabel Evans details various solutions implemented, including better test reporting, user stories, and engineering details, while emphasizing the importance of quality in use as integral to the customer experience. Challenges faced during this journey highlight the necessity of balancing testing improvements with company vision and customer needs.
More Related Content
Similar to Isabel Evans - Quality In Use - EuroSTAR 2011
![Vibe Coding vs. Spec-Driven Development [Free Meetup]](https://cdn.slidesharecdn.com/ss_thumbnails/vibecodingvsspecdrivendevelopment-251209105622-43f455e7-thumbnail.jpg?width=640&height=640&fit=bounds)
Isabel Evans - Quality In Use - EuroSTAR 2011
- 1. Quality in use: the beating heart of the customer experience Isabel Evans - Quality Manager Dolphin Computer Access Accessible document describing this presentation available from isabel.evans@dolphinuk.co.uk
- 2. Dolphin’s Purpose “Itis our purpose to deliver independence to people with vision and print impairments all over the World.” Noel Duffy Managing Director Dolphin Computer Access
- 3. “In an idealworld no-one would need our products – no-one chooses to be blind” • The Dolphin Challenge: • Even if the products are perfect no-one chooses to be print-impaired... Dave Williams Dolphin Blindness Product Owner and Q&D Tester
- 4. Our journey Quality in Use What? Why? How?
- 5. Our journey Dolphinrecognises an opportunity for improvement • Takes on a quality manager – To improve testing • What needs improving? – No specialist testers – No reporting on testing – No techniques – No automation
- 6. Solution 1: moretesting, better testing • Improving test skills and motivation • Night-siding to New Zealand • Regression test packs, personas, user stories • Technique of the Week & lunchtime briefings • Automated some testing – CTE and Python – Rhubarb – Beetroot – Sledgehammer • Lots more to do...
- 7. Solution 2 -Improve test reporting Example snapshot for one product – simple reporting: tests passed tests failed tests not yet run
- 8. Solution 3: improvemetrics Example snapshot for one product – simple reporting: Defects found in test Defects found after release Release X More defects found after release than in test More defects found in testing than after release Release Z Improvement target we have set for ourselves Release Y By July Aim for December
- 9. Side effects ofthe cure • Disadvantages of improving testing and acceptance criteria • Being wedded to your own solution • The “new cleaner” syndrome • Fear of change • Level of maturity • Loss of balance between pressures • Cannot automate yet... • TEST IMPROVEMENT IS NOT ENOUGH! Project stopped and restarted Morale drop Workload increase
- 10. As well astest improvement support the company vision and purpose: Deliver appropriate solutions Be loved by customers Deliver commercial success
- 11. Solution 4: useISO25000 - Quality in Use Usability Context conformity; Context extendibility; Accessibility Safety Effectiveness; Efficiency; Satisfaction Flexibility Commercial damage; Health & safety (operator & public); Environment
- 12. Quality in Use(QiU) and Dolphin U F People with Visual Impairments (VI) Marketing Sales S Buyers Dealers Quality in Use – “My tasks” and “My money”
- 13. People are different... Context of use Personas QiU scores User stories
- 14. QiU scores fortwo Dolphin product users Effectiveness 10 8 6 4 2 0 Efficiency Satisfaction Conformity Extendibility Environment Health & Safety Commercial VI - SuperNova VI - Guide
- 15. Solution 5: don’tforget engineering detail Security Functional suitability Operability Portability Compatibility Maintainability Performance Reliability Internal Quality (IQ) How was it built? Is it faulty? Were good techniques used?
- 16. Internal quality (IQ)– engineering detail Support F_O Sys-Ads Development S_P_R Po_Co_Ma Engineers and technologists care about these... Other people only notice them if they go wrong...
- 17. QiU and IQ– Dolphin Project Team Quality in Use Internal Quality U F S F_O S_P_R Po_Co_Ma Includes People with VI Sales Marketing Support Includes People with VI Development
- 18. Quality in Use Internal Quality U F S F_O S_P_R Po_Co_Ma Where are the testers ? Testers for QiU and IQ
- 19. U F S Testers for QiU and IQ F_O S_P_R Po_Co_Ma Market Readiness testing Beta testing Research, Design, Development, Code Review, Unit test, Product test
- 20. Solution 6 –customer-centric SDLC Market Requirements Iterations U F S IQ Post Implementation Review
- 21. Solution 7 -Project workshops with sales and marketing • Marketing, Sales, Support – Focus on emotional – Focus on practical – Focus on tasks • Vision • User stories • Acceptance criteria • Development, Technical testers – Focus on measureable acceptance criteria – Focus on architecture U F S F_O S_P_R Po_Co_Ma
- 22. w7 Solution 8– Sensible Iterations w1 w2 w3 w4 w5 w6 w7 w1 Choose Contents Release Market Ready test R&D&T for this iteration R&D&T for multi iteration items Release to focus the detail
- 23. Market readiness email • What is the verdict on this build? • Action: verdict by (date): – Yes – No – Don't know – No opinion • Proposed deadline for … – Date for all market readiness feedback – Date for release Market Ready test
- 24. Iteration content email • What is going to be in the next iteration? • What about the issue with (....)? • How did we come up with this content list? • What is the proposed schedule? • What is the contingency plan? • What is your verdict on this plan? • Response by (date) Choose Contents
- 25. What have welearnt? • Marketing, Sales and Support involvement • Improving project & quality management • Improving test skills and motivation • Iterations forced out process weaknesses • Night-siding to New Zealand • Technique of the Week & lunchtime briefings • It’s good when other people have “your” ideas • Automation is hard sometimes • Accessibility and white-boarding... • Lots more to do...
- 26. The journey continues “In an ideal world no-one would need our products no-one chooses to be blind”
- 27. Quality in Use: The beating heart of the customer experience Usability Flexibility Safety The journey continues... Thanks and goodbye Isabel Evans – Quality Manager – Dolphin Computer Access
Editor's Notes
- #2 Title slide: Quality in use: the beating heart of the customer experience Isabel Evans Quality Manager Dolphin Computer Access Graphics: Dolphin logo; illustration of heart and heartbeat; An accessible Word document describing this presentation is available from isabel.evans@dolphinuk.co.uk At Dolphin Computer Access, the company where I am quality manager, we build assistive technology for people who are print impaired. That is for people who are blind, have low vision or are dyslexic. In my EuroSTAR presentation "Quality in Use - the beating heart of the customer experience" I am going to discuss the importance of usability, accessibility, and flexibility of use for our customers and for yours.... How well are people supported to carry out tasks? I will illustrate the need to focus first on the customer to understand what technical product attributes are important. I will show you how we implemented user stories with acceptance criteria based on quality in use attributes. These help everyone focus on what is important for our customers. I will talk about what has gone well, and what we had to improve as well as where we have had to adapt standards and methods to fit our particular needs. You will hear the lessons we have learned.
- #3 Introducing Dolphin Computer Access – Dolphin’s Purpose Dolphin provides assistive technology for people who are print impaired. That is for people who are blind, have low vision or are dyslexic. Our managing director, Noel Duffy has set us a purpose as a company: “It is our purpose to deliver independence to people with vision and print impairments all over the World. With over 160 million visually impaired people and 600 million dyslexic or learning disabled people, that's no small challenge. But we're determined to make a difference and have been working hard on that goal since Dolphin started in 1986.” We have a particular challenge. Graphics - photographs of various Dolphin customers
- #4 As Dave Williams (Dolphin Blindness Product Owner and Q&D Tester) says “In an ideal world no-one would need our products – no-one chooses to be blind.” The Dolphin Challenge is that even if the products are perfect no-one chooses to be print-impaired... so we must work hard to make our products right for our customers. Graphic: photo of Dave Williams
- #5 Agenda diagram – radial circle, “Quality in Use” in the centre; items around the radial circle are: Our journey What? Why? How? Graphic – illustrative/decorative
- #6 Our Journey Dolphin recognised an opportunity for improvement with the help of Julian Harty (Google and then E-Bay) and as a result decided to take on a quality manager with the idea that the immediate need was to improve testing. Improvements were needed because there were gaps in the test capability: No (few) specialist testers Poor reporting on testing Poor application of test techniques No test automation. Graphics: illustrative/decorative
- #7 Solution 1: more testing, better testing We took these first steps: Improving test skills and motivation – identified testers for the different products, and started providing in-house workshops on test approaches and test design; We had a shortage of resources as well as a shortage of skills and so “Night-siding” to a test services company in New Zealand has given us extra capacity, fast turnaround, 24 hour work, and skilled testers providing role models for less experienced in-house testers; As a result of these the testers were able to build better regression test packs, with overview test conditions based on personas and user stories; We fed out information on test techniques to testers and developers across the company in weekly short bursts – a 2-4 side document “Technique of the Week” with lunchtime briefings in the following week to discuss and consolidate; We have started to automate some testing: CTE and Python has been very useful for unit tests on one product range For the access software is harder to do because of the way the software intercepts what is happening as in the operating system and any applications that are running so we have started to build automation in-house – we have three tools: Rhubarb, Beetroot and Sledgehammer in progress. Graphics: photos of Colin Shales (Q&D usability tester) and James Bowden (Developer and test automation)
- #8 Solution 2 - Improve test reporting Test coverage pie charts The graphic illustrates how we used pie charts to show test progress – the one chosen is from one product midway through testing in August – and shows about a third of the planned tests had been run at that stage, the majority having passed. The product has now passed all planned tests and is being released.
- #9 Solution 3: improve metrics The graphic show two graphs comparing the number of issues raised in testing with the number raised after release of a product plus a target improvement. Product A release X: three quarters of the issues were raised post release, before any improvements to process or technique for development or test. We then improved the testing. Product A release Y: some improvements to testing – now two thirds of the issues were found in testing and only a third post release. Our aim for December is shown in pie chart 3 – by release Z in December, we want the DDP to be at 85%, and this will be achieved by development improvements as well as test improvements. This include white box testing and code review, design and architecture reviews and more automation. Use of these pie charts had two benefits – the testers could see that they had improved and management could be persuaded what actions were necessary to take next.
- #10 Side effects of the cure Testing found so many defects and such important defects that we had to: Halt a project and restart; morale dropped especially among the developers; the developer workload increased. The side effects of the cure (improving testing) included meeting the change management problems of resistance and fear, trying to implement solutions beyond the maturity of the organisation, causing setbacks, and loss of balance between pressures. Automation was harder for the access products than I had anticipated. The lesson and key point: TEST IMPROVEMENT IS NOT ENOUGH! Other problems: Accessibility – in order to keep all team members up to date we had to ensure that all the artefacts around the project were accessible. We had to balance that with using non-accessible artefacts where that did not disadvantage colleagues with visual impairments and benefited sighted colleagues. E.g. flowcharts can be used and then expressed as text. This document is intended to take the place of the slide set. Level of maturity – it is important to make things as simple as possible, in order that everyone can buy in, and then increase the complexity gradually. Change gradually over time – it is easy to get impatient – change takes time. WAI compliance – our tools have to deal a wide range of software applications and websites – for example, every time a website is non-compliant to WAI guidelines and our software cannot read it, it is us that the customer rings...
- #11 As well as test improvement support the company vision and purpose: Improving testing had made some things worse and had exposed problems that could not be resolved by testing. Instead of just improving testing we needed to Support the vision and purpose: Deliver appropriate solutions; be loved by customers; and Deliver commercial success. Message to testers: just improving testing is not enough to help the business - Our skill set and comfort zone is often in the detail; we find excitingly obscure bugs and report these problems in detail. Our zeal sometimes causes us to miss the big picture of what the customer and the business really needs. We focus on software quality at the expense of human and commercial factors. We need to consider company vision and customers, we need to consider commercials. Graphic: decorative
- #12 Solution 4: use ISO25000 - Quality in Use Graphic shoes there are three Quality in Use attributes: Usability Flexibility Safety Usability has the sub-attributes Effectiveness; Efficiency; Satisfaction Flexibility has the sub-attributes Context conformity; Context extendibility; Accessibility Safety has the sub-attributes Commercial damage; Operator health and safety; Public health and safety; Environmental harm ISO125000 is a standard that describes software attributes, how to specify them and how to measure them: Software Quality and Requirements Evaluation
- #13 Quality in Use (QiU) and Dolphin The graphic shows the Quality in Use attributes of Usability, Flexibility and Safety, which reflect a user and value view of quality –“my tasks” and “my money”. Buyers and Dealers are looking for commercial safety and flexibility People with VI & TI are looking for usability and flexibility Dolphin sales and marketing are looking for all 3 attributes, but particularly flexibility – new things to sell.
- #14 People are different... The photographs show two Dolphin customers, both have visual impairments but their requirements from an access product are quite different. The gentleman on the left is using Guide. This give magnification voice output and – if wished – voice input. It also provides a simplified interface which replaces Windows and provides a simple internet browser, email, simple word processing. It is suitable for people who are less used to computers and/or have recently lost vision. The lady on the right is tech-savvy and uses applications such as Office, as well as all the work applications that her colleagues use. She is using SuperNova with multiple monitors to magnify the display. SuperNova is suitable for people who are tech-savvy and want to use all the applications you use at work or at home. These people have different personas and contexts of use. They have different quality in use scores. Even if they share user stories (I want to do on-line banking) these need to be implemented in different ways.
- #15 QiU scores for two Dolphin product users Quality in use – products compared: The graphic is a radar graph, showing each of the Quality in Use sub-attributes, and the scores for two Dolphin products – SuperNova and Guide. Each has users who have visual impairments but the two groups have quite different quality in use attribute scores. SuperNova users are typically working or at college. They are typically tech-savvy. They value being able to work efficiently and flexibly, using SuperNova to access a wide range of applications. They are conscious of whether SuperNova saves or costs them money. The typical Guide user has recent sight loss, and low IT experience. Typically a Guide users value the interface being stable and predictable (conformity) and knowing they are able to be effective (they are protected from making mistakes). The attributes are ordered: SuperNova typical user: Efficiency, Extendibility, Effectiveness, Commercial, Satisfaction, Conformity, Health & Safety, Environment. Guide typical user: Effectiveness, Conformity, Satisfaction, Efficiency, Extendibility, Health & Safety, Commercial, Environment.
- #16 Solution 5 – don’t forget the engineering details We also need the engineering details – that is Internal Quality – How was it built? Is it faulty? Were good techniques used? Have all the “ilities” been considered? The internal qualities are functional suitability, operability, security, performance, reliability, portability, compatibility, maintainability The customers who are systems administrators care about these. Dolphin Support and Development care about these. Technical testers care about these. Others only notice them when they go wrong...
- #17 Internal quality (IQ) – engineering detail Internal Quality – How was it built? Is it faulty? Were good techniques used? The graphic shows the internal qualities: Functional suitability, operability, security, performance, reliability, portability, compatibility, maintainability The customers who are systems administrators care about these. Support and development care about these. Others only notices them when they go wrong...
- #18 QiU and IQ – Dolphin Project Team The project teams need to have representatives for quality in use and internal quality. Dolphin Sales and Marketing are interested in quality in use Dolphin Support and Development are interested in internal quality Both groups include colleagues who have visual impairments. We make sure we have considered: Usability – Effectiveness; Efficiency; Satisfaction Flexibility - Context conformity; Context extendibility; Accessibility Safety - Commercial damage; Operator health and safety; Public health and safety; Environmental harm Internal quality is made up of: Functional suitability, operability, portability, compatibility, maintainability, security, performance, reliability And we make sure that we prioritise these based on the Quality in Use priorities.
- #19 Testers for QiU and IQ Where do the testers sit? We need test skills in quality in use and internal quality. We have a group called Q&D (Quality and Design) which includes the quality manager, some specialist testers and project management. Testing is carried out by: Static testing (reviews) by people across the project team Dynamic testing by developers, Q&D, New Zealand nightsiders, representative in-house product users, sales and marketing, support, dealers, customer beta testers. At the start of projects, Sales, Marketing, Support look at the vision, context of use, personas, user stories, acceptance criteria, market requirements, then pass these to development. Development does research, design, development, code review and unit test. Q&D (specialist) Testers carry out a Product test, helped by representative in-house product users, and by the New Zealand nightsiding team. When sufficiently stable it is passed for beta testing to representative dealers and customers. When the product test passes acceptance criteria, it is passed to Sales, Marketing, and Support to carry out market readiness testing.
- #20 Testers for QiU and IQ The graphic shows project stages above and below the quality in use line Sales, Marketing, Support look at the vision, context of use, personas, user stories, acceptance criteria, market requirements, then pass these to development. Development does research, design, development, unit test (specialist) Testers do Product test Sales, Marketing, Support carry out market readiness testing Externals carry out Beta testing
- #21 Solution 6 – customer-centric SDLC This graphic shows the steps through our project model. The Market Requirements phase is run by the product owner – typically from marketing - and project mentor – management team representative to identify the project vision for the customer, context of use, personas, user stories, the commercial aspects, benefits for the customer and for Dolphin, acceptance criteria. This is outlined in a Market Requirements Document. It could lead to a research project, prototyping or a delivery project, depending on our knowledge. Typically that would be in iterations. We run a brief post implementation review at the end of each iteration, or group of iterations as well as at project ends.
- #22 Solution 7 - project workshops include Sales and Marketing It is vital that Sales and Marketing are including in all the decisions. They take part in the workshops and other activities to build user stories. Marketing, Sales, Support: Focus on emotional Focus on practical Focus on tasks Focus on Vision for Customer and Dolphin User stories Acceptance criteria. Development and technical testers focus on measureable acceptance criteria and architecture. Q&D facilitate and encourage. The emotional level in the user stories provided by Sales, Marketing, and Support and in particular by product users in the company helps us focus on what is important. Example: As a person who is blind I wanted to manage my own bank account online Because it is so humiliating when someone has to read you your bank statement As an elderly person with declining vision I want to contact my grandchildren online Because I miss them and want to share news easily
- #23 Solution 8 – Sensible Iterations to focus the detail The graphic shows a series of overlapping iterations, each of 6 weeks plus week 7 for release. During the 6 weeks research, development, test for this and subsequent iterations are carried out. We found that we have work which needs to carry across several iterations. In the last 2 weeks market readiness testing is done by Sales, Marketing and Support, and they agree the contents for the next iteration.
- #24 Market readiness email In the last 2 weeks of the iteration we move to Sales, Marketing and Support carrying out market readiness testing and choosing items for the next iteration. A Market readiness email is sent from Q&D to everyone in Sales, Marketing, Support, and to Training, Documentation, and in-house product users. The email says: What is the verdict on this build? Action: verdict by (date – usually a week is allowed): Yes – there may be issues but they can be lived with for this iteration No – there are showstoppers Don't know – I have not finished my MR tests – estimated date to complete No opinion – I have not had time to test Proposed deadline for (iteration): Date for all market readiness feedback and planned date for release.
- #25 Iteration content email A rough iteration plan for several iterations ahead, plus a wish list, is kept running. Items for iterations are planned in and discussed with all the stakeholders, as required and confirmed in a weekly meeting as we monitor support calls. Just after the MR email, an iteration content email is sent out. This is a series of paragraphs headed: What is going to be in (next iteration)? List of items What about the issue with (....)? Items requested that cannot be covered in the iteration and why, plus a planned iteration How did we come up with this content list? What is the proposed schedule? What is the contingency plan? What is your verdict on this plan? Response by (date).
- #26 What have we learnt? Marketing, Sales and Support involvement is vital, but they have be shown the value and that progress has been made because of their contributions Improving project & quality management is very important and needs to done in small doses Improving test skills and motivation in very important and needs to be managed against other improvements in order that problems are uncovered and resolved at source Iterations forced out process weaknesses – the pain of moving to an iterative approach was large, but the problems were process, skill, control, tool or techniques problems that needed to be sorted out – the sort time spans exposed these more clearly Night-siding to New Zealand was really useful to provide a breathing space and also caused people in-house to want to improve what they were delivering it exposed time testing preventable issues as real costs Technique of the Week & lunchtime briefings helped people pick and choose topics to suit their immediate needs and meant iterative / incremental approach to improvement It’s good when other people have “your” ideas – it means they have adopted them Automation is hard sometimes – but worth persisting with Accessibility and white-boarding... don’t be dogmatic about the right way to do things – use the methods that suit the team Lots more to do...we have made huge improvements in the last year, but there is a lot more to do! What has gone well? Sales and marketing involvement has worked well in providing context of use, personas, user stories, setting acceptance criteria, and market readiness testing. Increasing awareness in the organisation of the importance of strong project managers to control budget, timescale, risk and cost benefit analysis. Increasing confidence and awareness in the testers (from all areas) of the importance of strong quality management and testing. They hold the line, with courage, information, appreciation of risk, they offer choices based on cost benefits. Realisation among the testers and elsewhere that test skills and motivation is not enough – we also need to understand the commercial, sales, marketing and customer impact. Quality in use as the focus. Night-siding works and helped internal testers improve. Iterative skills improvement with Technique of the Week, workshops, lunchtime briefings – push out the improvements a few at a time. We tried to do too much too quickly at first – pick off the targets, and persist!
- #27 The journey continues We need to remember everyday our purpose as an organisation, our customers and what they need from use. Quality in use (usability, flexibility and safety) keeps us focused on what is important. Remember “In an ideal world no-one would need our products – no-one chooses to be blind”. Quality in use is for us the beating heart of the customer experience. Graphics - photographs of various Dolphin customers and staff
- #28 Quality in use: the beating heart of the customer experience Usability, Flexibility, Safety The journey continues – thanks and goodbye Graphics: Dolphin logo; illustration of heart and heartbeat; text boxes with usability, flexibility and safety.