Choosing an alm tool set


Published on

These slides contain general advice for considering an ALM tooling solution. This includes management of requirements, tests and defects. It is a draft release.

Published in: Business, Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Choosing an alm tool set

  1. 1. 1 Ian McDonaldHND Dip BSc PGCE MBCS CITP CSci MInstP CPhys EurPhys MIET IEng ISEB Test Practitioner Choosing an Application Lifecycle Management (ALM) Tool Set © 2013 Ian McDonald
  2. 2. 2 Introduction I frequently get asked – which tool do you recommend? There is no short simple answer and a lot will depend upon specific needs. These notes are to help individuals consider the wider implications of tool choice. This is a first draft and this will be updated from time to time.
  3. 3. 3 Overview  There is no golden solution for the choice of Application Lifecycle Management (ALM) tool sets.  The wrong choice can adversely impact the ability for a company to:  Grow  Be competitive  Deliver successfully.  Too often key attributes for tool selection are forgotten, either due to the lack of experience of those choosing a tool set, or because they allow themselves to be led by tool vendors, tool partners or choices made in other companies for meeting different needs.  These notes are written as an independent consultant with no vested interests in any individual tool supplier.
  4. 4. 4 Application Life Cycle Management With ALM tool sets we are focusing on 5 specific areas that support the development lifecycle. These areas are: Management of Requirements. Management of Test Cases and Scripts. Management of Test Results. Management of Defect / Fault Reports. Creation of reports for managers, QA and process improvement.
  5. 5. 5 Lifecycle Types  In managing the application lifecycle, we need to remember that there are different types of lifecycle and some tooling may be less or more appropriate than others. These life cycles include among others:  Agile including lean, Scrum, RUP, ORUP and many others.  V-Model.  Waterfall.  Spiral (continuous development and deployment).  However many tools are not tied to one specific lifecycle and through correct use can be equally effective across many lifecycle types.
  6. 6. 6 First Things First  Before starting to look at tools, there are a number of key things that need to be addressed first:  What is the health status of the lifecycle?  Who are the lifecycle stakeholders?  What are the lifecycle strengths?  What are the lifecycle weaknesses?  How does the lifecycle benefit the business?  What are the strengths and weaknesses of the current support tooling?  How is the tooling used by stakeholders?  What are the problems in use?  What are the limitations?  Is the tooling fit for the business growth?  Is the tooling being used consistently across projects?
  7. 7. 7 Stakeholders  In looking at ALM tooling, there are a number of stakeholders – including, but not limited to:  Project Managers including Programme Managers.  Business Analysts  Developers  Testers including Quality Assurance Staff.  3rd parties who may need to review reports, submit defects, etc.  For successful tooling adoption, all stakeholders need to be fully engaged and involved.
  8. 8. 8 Strengths and Weaknesses  In choosing a replacement tool it is first important to understand why a replacement is required. There are many reasons for change, some possibilities are to:  Reduce licence costs?  Reduce tool maintenance costs?  Improve reliability of delivery?  Improve production costs?  Provide greater security in delivery?  Provide for process improvement?  In supporting your goal, where are the strengths and weaknesses for your current tool set?  Are there any other opportunities in changing tools?
  9. 9. 9 Cost Opportunity  Tooling must ultimately benefit the business. It is not there to make life simpler for a group of individuals. There must be a financial return for the business to justify the upfront investment.  The best approach to identifying the opportunity is to use metrics. However for companies who have as yet to collect metrics, the best they can do is to either refer to typical industry metrics or to employ a consultant who can independently assess the situation.
  10. 10. 10 Hidden Opportunities  While in some cases it can quickly be pointed out that by using tool X, the manufacturing cost will fall by Y% and so the tool will pay for itself in Z years, in most situations the reasoning can be more subtle. Some examples include among other points:  Opportunities to cut defect injection – so reducing production time and costs.  Ability to deliver more reliable testing, cutting defect delivery into products, so improving customer satisfaction and so increasing sales.  Allowing management greater insight into the health of a project, so improving delivery control.
  11. 11. 11 Stakeholder Benefit  While the business will need to change, people often do not like change. They have been working with a known and understood process and change will impact their daily work life. Sometimes it may mean for some individuals extra work in certain parts of the lifecycle. It may mean that an initial task takes longer. So it is important that stakeholders are engaged and that they see there is a clear overwhelming benefit to them personally in addition to the business argument.
  12. 12. 12 The case of the Business Analyst  The Business Analyst (BA) will typically create requirements. These will get passed over the fence to the developers, who often will reinterpret requirements, try to plug gaps and ignore requirements that cannot be tested and even over-engineer the solution, adding cost.  The Developer throws the product over the wall to the test team, who reinterpret some requirements, others they may get guidance from the developers and eventually they sign off a product and deliver it with defects still present and perhaps not doing exactly what the BA envisaged originally. Of course it will all go wrong and who is at fault – the authors of the requirements!  So if the developers are actually rewriting the tests – why have a BA?  The point is that this is happening usually because the lifecycle is waterfall and BA’s do not have the early support of testers and developers.  Review of requirements is poor, so the team do not appear to have joint responsibility for requirements.  The test team are supposed to champion the BA vision and the link between the BA and tester is probably non-existent in many companies.  Tests are to reflect the BA requirements. However these requirements have been recreated and changed with no audit trail, no notification of change and the tests at best may only partially test a given requirement.  So for the BA to be a winner – they need:  Greater control over interpretation and change of requirements.  Early buy in for requirements – so if there is a fault, it is in the requirement review. The team are responsible.  Ability to audit test cases – to ensure that what is being tested is what is required.  Maintaining the initial vision through to delivery.  Greater partnership with the test team.
  13. 13. 13 The case of the Developer  The developer typically faces a number of problems:  Incomplete requirements – needing gaps filled.  Changing requirements – with no process control. Leading to late detection of problems in delivery.  No prioritisation of requirements, making it difficult to make the right choices in an Agile delivery.  Multiple repeat defect reports.  No way of easily grouping defects to create work packages and so reduce defect fix time overall.  Defects that are incorrectly reported, due to interpretation of requirements.  If the developer can gain benefit in solving the above problems then they will want to buy-in. However usually developers want to maintain:  Tooling that supports unit testing and static analysis.  Use of Agile – however can be seen as being Fragile, if necessary processes and monitoring controls were incorrectly discarded. Agile uses no Unnecessary documentation – however some items and processes are necessary, for Agile to be successful.
  14. 14. 14 The case of the Project Manager  The project manager will try and estimate the project delivery time. depending upon experience, will often underestimate delivery and be very optimistic. Risk analysis may vary and usually testing is based upon a percentage of development effort, which can be very unrealistic and risky if existing code is being migrated as test time can be greater than development time.  The project manager will be in receipt of mixed messages, from the developers claiming delivery of untested code, from the testers, who will claim that test have failed or not run as functionality is missing. In practice no code should be counted as delivered unless it has been successfully tested. Often this type of problem arises due to company bonus incentives that are driving the wrong behaviour.  With delivery timescales sliding, the project manager wants to know quickly in real time:  What is happening with requirements creep – how is this being managed?  What code has passed testing and so can be claimed to be delivered?  Which prioritised requirements are delivered?  Which defects are holding up delivery?  When will testing stop – i.e. If the number of defects has been realistically estimated, when is it expected to reach that target figure, within +/- X days?
  15. 15. 15 The case of the Test Manager  The test manager usually needs far less convincing. They will want to:  Bring requirements under configuration control.  See requirements entered as single logical requirements and have the ability to link sub-requirements to the parent requirement.  Have well structured test cases (test vector values).  Have the ability to create work packages for the creation of test scripts.  Manage up issue of test scripts and cases and link results to the relevant version of scripts and cases.  Manage the running of test scripts (manual and automation).  Generate test reports for specific builds and configurations.  Manage defects and ensure that they are mapped to test results, scripts, cases and requirements.  Good reporting, so the test manager knows what is happening at any time.
  16. 16. 16 Other Stakeholders Other stakeholders outside the company may have specific needs – especially if you are working in a partnership. Do you need a web interface? Will 3rd parties submit defects? Will 3rd parties access reports? Are there wider licence implications? Are there interfaces that need to be considered?
  17. 17. 17 What Else is Happening?  It is very easy to get bogged down with functional tests, defects and requirements and forget about other needs within the lifecycle. These may include:  Use of unit test supporting tools for developers.  Risk analysis.  Automation needs – including GUI and Performance.  Reporting to higher management and QA.  Financial monitoring.  Support policies for tools and environments.  Database licences and standardisation across company.  User support issues.  Training and recruitment.  These will all need careful consideration.
  18. 18. 18 Capability Maturity Model Integration (CMMI)  CMMI focuses upon a number of core process areas and ranks these at specific levels. The levels are:  1 – Initial – Unpredictable, reactive, poorly controlled.  2 – Managed – Reactive.  3 – Defined – Proactive, meets organisation standards.  4 – Quantitatively managed – Measured and controlled.  5 – Optimising – Focuses upon process improvement.  Decide:  Where are you now on the CMMI scale?  Where on the CMMI scale do you want to be?  What is needed to get to the CMMI level you require?  What is your timetable to reach the CMMI level required?  Are there budget or cash flow limitations limiting your progress?
  19. 19. 19 CMMI Progress  For a company to progress, it really needs to be within the zone 3 to 5:  3 – Proactive, meets organisation standards.  4 – Measured and controlled.  5 – Process improvement.  Managers need to move away from having to react to problems and move towards predicting problems and proactively eliminating them.  This means that metrics need to be collected and used to help that process.  Standards will be put in place to facilitate the smooth transition of projects through the lifecycle. However the standards and processes have to be correctly focused to bring the right level of success.  The tooling deployed will (if correctly chosen and set up) assist to facilitate that success. However tooling is no substitute for good management.
  20. 20. 20 Tool Properties  Tool Properties need to be examined at various levels. Among others, the properties will include:  Requirements Management  Test Management  Defect Management  Reporting  Tool Structure  Tool Ownership  Environment
  21. 21. 21 Requirements Mapping For Requirements Management the following features are worth considering: The ability to link tests to single logical requirements, will mean that requirements will have to be broken down to Sub Requirements as single logical statements. So the tool needs to support the structure. Parent – A Level Requirement Daughter – B Level Requirement Daughter – B Level Requirement Daughter – C Level Requirement Daughter – C Level Requirement
  22. 22. 22 User Stories are No Substitute Some tools refer to user stories, this however is not a substitute for requirement nesting. Usually user stories can be handled by separate parent requirements. The important point is the ability to have multiple nesting, down to a single logical testable statement.
  23. 23. 23 Requirement Control  Controls to manage requirements are important. Key points to watch are:  Ability to update requirements and maintain copies of the old versions. This is important so developers and testers can view changes.  A rollback may mean that the previous version of the test and requirement needs to be restored.  For project managers they need to understand the impact of changes. Tests might need changing, additional tests may be required, tests regrouped and more importantly risk re-assessed.
  24. 24. 24 Requirements Change Changing a requirement under configuration control, will mean that both the development and test team will need notification. This can be displayed as: Warning - Requirement has been updated. Warning - linked test may require attention as a change in requirement was carried out. Automated email notification to relevant stakeholders.
  25. 25. 25 Test Case Management  Test cases are vector sets (inputs and outputs) used within test scripts. As data can be input in different ways, there may be many test scripts for a specific test case. There may also be many vector sets to test a single requirement (e.g. boundary values).  Test vector sets may potentially be generated with tooling such as the Classification Tree Editor (CTE – See separate presentation). Hence it could be valuable if the ALM tool had an interface to such tooling as CTE.  Be able to group test cases into work packages – for running / creating associated scripts.  It should be possible to report which test cases have been run and what the outcome is in terms of success and script coverage.  A requirement change should flag any associated test case / script that may be impacted.
  26. 26. 26 Test Script Running Can you run tests and make appropriate records to show the results for a test run against a specific: Browser Operating system Hardware platform Application version Work package (i.e. tests carried out by a specific tester on a specific day).
  27. 27. 27 Test Script Management  As with test cases, any change in requirement should flag the test script.  The test script matches the requirement and the test results. So if the requirement or test script changes it is vital that the associated data for the previous version is not lost. A test result should be traceable to the version of the test script, test case and requirement at that time. Many tool sets fail to do this and will incorrectly map old results to updated scripts.  It should be possible to roll back software builds, so the test data should also have the ability to roll back and not be subjected to rework. This can be achieved with good configuration control, however some ALM tools manage this far better than others.  You should be ale to filter results for specific test runs containing details of the configuration used for that set of tests. E.g. List tests run on FireFox 16.0.2, Winsows 7 SP1, AMD 4 core 300 MHz processor, Test Application version 0.71, on 05 Sep 2013.  Automated scripts often require maintenance and so it should be possible to quickly flip between an automated and manual version of a script.  Often automated testing is run as part of an extended tool set. The results from these tests are often required o be reported upon and so need to be interfaced to the ALM tool set.
  28. 28. 28 Test Result Management  Test results need to reflect the version of test scripts and test cases used at that time. This is important as it may be used for:  Further analysis to identify hidden defects.  Reporting of the exact testing carried out.  Analysis of gaps in testing.  Test results may have associated data and this needs to be managed. Data can include:  Screen captures.  Video playback of any data recorded.  Attached documents and reports.
  29. 29. 29 Video Capture Some companies may require video capture of test scripts this could be to: Provide evidence in case of legal proceedings. While this is not used widely, it may become the norm in future years and so the ability to respond to this can be important. Provide records to demonstrate a pattern in uncovering the fault behind a defect report.
  30. 30. 30 Defect Management  Key properties to ensure are included:  Ability to map defects to tests and requirements.  The defect lifecycle can be adjusted to suit your needs – you should have a defect lifecycle defined.  What process control does the tool offer? Are roles well defined and maintained within the tool?  If a defect references a specific test, does it reference the current updated version (incorrect) or the version that detected the fault (correct)? Or better still, does the defect report reference the correct test version and also provide information on other test versions?  Attachment of screen captures, video and other documentary evidence.  Ability to group defects into work packages.  Customisation of fields:  How easy is it to discover if a defect was previously reported?  Can you obtain process improvement information such as Root cause Analysis and test Escape Analysis?  Can you filter data to provide short lists of defects?
  31. 31. 31 Reporting  This is important – you will need to be able to report on processes and obtain metrics for process improvement.  Does the tool allow for reporting across multiple requirements. For example can you list a defect against a test script, parent and daughter requirement? Some major tools do not allow for this simple task.  Can you produce the appropriate metrics, tables, graphs and charts you require. How configurable is this?  How responsive is reporting? Some reports when run can slow the application considerably, so take care and check out the reports you want to use, under the load you expect.  If your company is using an Agile and/or V-Model lifecycle(s), how appropriate is the reporting for your needs with the lifecycle of choice?  Can you apply a consistent tool solution across your organisation?
  32. 32. 32 Licences  Tool licences can vary considerably. Key questions to ask are:  Is the licence for a network, site, company (country or global)?  A site licence may not be transferable between networks at the same site. For companies developing software on small team networks, some licence structures are not affordable.  Is the licence fixed to a specific user or a floating (concurrent) licence for any user who is logged on?  Some companies have licences that follow users.  Some licences are floating (concurrent), so when a user logs out, the licence is freed up for another individual.
  33. 33. 33 Licence Composition Some products will split licences by function. For example the defect management capability may require different licences. Decide what type of licences are required by which users and for how long – if floating (concurrent) licences are being used.
  34. 34. 34 ALM Licence Quantity A simple method of estimating concurrent license requirements:  Each tester needs a full time license. This includes the Test Lead/Manager if they are also doing full time hands on testing.  If the Test Lead/Manager is not carrying out full time testing, then they may only need 0.2 of a licence. This allows them to review, control processes and create reports. However if they are also carrying out additional tasks such as: configuration control, quality assurance and audits, they may require at least 0.5 of a licence and potentially a full time licence.  Each BA needs a minimum of 0.25 of a license. Assuming business analysts (BA’s), do not generate requirements continuously. This could be higher depending upon the number of BA’s and the number of parallel projects in the requirements gathering stage. Or it can be lower, if BA’s generate requirements on another tool then transfer data across to the ALM tooling. They may still however need licences for audits and requirement maintenance.  Developers who are working on defects need ¼ of a licence. They’re generally only reviewing defects and updating them and therefore would only require the Defect Manager element – although they may want to see the test, this can have the disadvantage of making the code pass the test as opposed to being rugged code. However licence estimates depend upon the number of projects in the testing/defect fixing phase at any one time. If there is a team configuration controller and a central point where defects are assigned from, then those users may require 0.5 of a licence.  Others users will monitor a project or view occasional detail. To support for this use, only 0.25 of a licence is required. Potentially this could reduce to 0.1 of a licence.  If you have a large team, say over 20, then additional licences may be required to avoid occasions where use demand overlaps.  To manage meetings where licence demand might outstrip supply, use a projector for presentation or create a Webex session and so avoid hogging licences from others.  You may want to have a policy for inactive use and automated tool logout.
  35. 35. 35 Database structure and compatibility  Your tooling will need to interface with a database.  Your company may have a database policy and an official database upgrade path – is the tooling compliant with that policy policy?  Do you already have an appropriate licence on the relevant server to support the tool database requirements?  The structure of the tool database needs to be maintainable, robust and fit for purpose. There can be surprisingly different qualities between even well known tool sets.
  36. 36. 36 Tool System Architecture  Make a list of the tooling that you will need to integrate. Consider:  Database requirements.  Business Analysis requirements.  Development tool requirements.  Manual testing requirements.  Test automation requirements.  Defect management requirements.  Reporting – including cross project reporting and metrics collection.  Decide which server(s) the tools will reside on and what the relationship mapping is.
  37. 37. 37 Operating in the Cloud For some companies a cloud solution – where the tool licence is hosted on a network outside of the control of the company – may be the right solution. However: Is there financial risk in publishing your systems vulnerabilities on an external network? Are there other commercial or security risks?
  38. 38. 38 Environment Needs Are there any specific mobile environments that need to be considered? Automation needs Reporting needs Are there other environment needs. E.g: Mainframe / Green Screen Windows, Unix (different flavours), Linux, etc.
  39. 39. 39 Support  What support does your tool vender offer?  A free tool may have little support and so will have specific risks:  The tool may not support new technology – denying you an upgrade path.  There may be a slow response to major errors – denying you access to the tooling.  What is included within the maintenance contract?  There may be phone support? What is the response time – some large companies can be instant to several hours of premium number time to access a first tier support person?  What training is offered?  Are there user groups, blogs and web help pages?  How good are the help pages?  What problems have been recorded on blogs?
  40. 40. 40 Access to Experience Some companies have tremendous difficulty in recruiting staff. They choose tools that are not popular in the market and so they cannot find individuals with the appropriate skills or who want to gain those skills. If you are choosing an uncommon tool, then your training budget needs to account for new staff.
  41. 41. 41 What are the Risks?  Risks should be assessed.  Will the tooling be there in 5 years time? Does the company have an upgrade path and long term development plan?  How does the company respond to changes in technology?  How certain are you that the tool will do what you want? Have you undertaken a trial?  List the properties you need and ask for at least a demonstration where those properties are witnessed.  If the tool became obsolete or stopped working – how quickly can you get a response and what would the commercial impact be on your company?  If the company stops trading – is the tool likely to become unsupported or taken over by a rival company? E.g. Popular tools like Mercury and rational were taken over by HP and IBM. Other smaller vendors have stopped trading.  What size team supports the tool? Will your concerns ever get answered?  If you need consultancy to create a specific plug-in or interface, does the company have the capability to offer that service?
  42. 42. 42 Open Source  If using Open Source code, then consider:  There will be commercial implications if you are selling a product. So make certain you pay for the right licence types.  Are there code security and performance issues. Do not assume the same rigour as with other commercial tooling. However it is also worth investigating with other commercial providers – especially if less experienced.
  43. 43. 43 Pick ‘n Mix Solution  A single tool may not always be the solution.  Alternatives might include:  A major ALM tool plugged into the database of another. E.g. you may want to use one tool for developers, but use alongside that a different ALM tool to manage requirements and tests. However you want to create reports from data collected by both tool sets.  Use an ALM tool, but enhance requirement management by interfacing a separate tool.  Use a range of separate tools interfaced together.
  44. 44. 44 Cost Benefit  What is the benefit of the tooling solution?  Who benefits and how?  What new capability does the tooling offer?  Is there any loss of capability?  How is efficiency changed?  What are the direct costs of ownership – licences, training, maintenance, etc.  What are the indirect costs of ownership – additional development costs, including short term and long term?  What is the impact upon recruitment?
  45. 45. 45 Try It Out  Do not take for granted that the tooling does everything it says on the label. Set up a project and try it out. Find out what works well and what does not.  Have a shopping list of key features and check them off to ensure they work as you would want them to.  This is a good time to now ask questions of people who have used a specific tool. Find out what their specific experience has been in their specific situation. Remember you are looking for problems. It might be good for them, it may not be for you. Here you are trying to identify risk so you can mitigate it.
  46. 46. 46 Finally  These slides only provide a glimpse at the wider questions.  Do not assume that large companies tools are best – it may be that their marketing is just better.  Do not assume free tools are cheap – there may be wider cost implications.  Do not assume that if a tool is good for a developer it will be good for everyone else. If you are trapped in this influencing decision you may be disappointed by what you thought was an ultimate solution. However you may want to consider interfacing tools to give the best of both worlds.  Do not assume that if a tool is right for one company, it will be right for you. They may have a different market place, different environments and different risks.
  47. 47. 47 END Slides created by: Ian R. McDonald HND Dip BSc PGCE MBCS CITP CSci MInstP CPhys EurPhys MIET IEng ISEB Test Practitioner   Available for consultancy.