Most Common Validation ErrorsIdenti Fy  Frequent Deficiencies To Accelerate Your Validation Projects (Based on Frank Houst...
About Orcanos<br />Develop and implement QPack Medical ALM system<br />Expert in Application lifecycle management<br />Lea...
Customers<br />
Working Without ALM System<br />Market<br />Definition<br />Develop<br />Test<br />9 Most Common validation Errors<br /><u...
Inconsistency
Lack of Needed Detail
Traceability
 Vague Wording
 Unverifiable Test Results
 GDP
 Incomplete Testing
 Ambiguity</li></ul>Test Results<br />
QPack Application Lifecycle Management<br />Market<br />Definition<br />Develop<br />Test<br />Design<br />Test<br />Plan<...
QPack ALM Solution<br />Teamcenter PLM<br />ALM ANALYTICS<br />Product <br />Requirements<br />Tasks<br />Marketing<br /> ...
Lessons Learned from 1,720 Observations<br />The goal was to bring the company’s software validation evidence up to the le...
Missing Information<br />Documents or records omitted fundamental information or content that should have been included.<b...
Inconsistency<br />Documents contained statements inconsistent with other statements about the same topic in the same docu...
Lack of Needed Detail<br />This deficiency applied mostly to requirements documents. The requirements in the validation pa...
Traceability<br />3 frequent traceability problems:<br />The traceability matrix did not account for a traceable specifica...
Vague Wording<br />Documents used generalities<br />“in accordance to an approved procedure”, or “applicable regulatory re...
Unverifiable Test Results<br />Expected results were not described sufficiently so that an independent reviewer could comp...
GDP<br />3 frequent Good Documentation Practice problems:<br />Hand-recorded data and testing evidence, such as test resul...
Upcoming SlideShare
Loading in …5
×

Orcanos medical-common-validation-errors

1,869 views
1,895 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,869
On SlideShare
0
From Embeds
0
Number of Embeds
119
Actions
Shares
0
Downloads
5
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Orcanos medical-common-validation-errors

  1. 1. Most Common Validation ErrorsIdenti Fy Frequent Deficiencies To Accelerate Your Validation Projects (Based on Frank Houston research)<br />QPack™<br />Application Lifecycle Management<br />May 2010<br />
  2. 2. About Orcanos<br />Develop and implement QPack Medical ALM system<br />Expert in Application lifecycle management<br />Leading ALM 2.0 and CFR 21 Part 11 Forums <br />Over 50 installations<br />Industry based solutions: Software/Medical Device/Semiconductors<br />Siemens Business and Technology partners<br />Oracle partner (OEM)<br />Offices: Israel, UK, Germany<br />
  3. 3. Customers<br />
  4. 4. Working Without ALM System<br />Market<br />Definition<br />Develop<br />Test<br />9 Most Common validation Errors<br /><ul><li>Missing Information
  5. 5. Inconsistency
  6. 6. Lack of Needed Detail
  7. 7. Traceability
  8. 8. Vague Wording
  9. 9. Unverifiable Test Results
  10. 10. GDP
  11. 11. Incomplete Testing
  12. 12. Ambiguity</li></ul>Test Results<br />
  13. 13. QPack Application Lifecycle Management<br />Market<br />Definition<br />Develop<br />Test<br />Design<br />Test<br />Plan<br />Definition<br />Market<br />Gathering<br />Requirement<br />Definition<br />2.0<br />Risk Mng.<br />1.0<br />Customer<br />Facing<br />Test<br />Execution<br />Delivery<br />Defect<br />Reporting<br />
  14. 14. QPack ALM Solution<br />Teamcenter PLM<br />ALM ANALYTICS<br />Product <br />Requirements<br />Tasks<br />Marketing<br /> Requirements<br />Test<br />Plan<br />Project <br />Management<br />QPack<br />Test <br />Execution<br />Risk<br />Management<br />Defects<br />Tracking<br />SOA<br />
  15. 15. Lessons Learned from 1,720 Observations<br />The goal was to bring the company’s software validation evidence up to the level of the U.S. FDA expectation.<br />Identified which documents most frequently contains errors<br />The results are typical problems most companies face<br />Applying Pareto Analysis<br />Finds 9 Types of deficiencies representing about 41% of categories<br />
  16. 16. Missing Information<br />Documents or records omitted fundamental information or content that should have been included.<br />Compliancy to specific standards<br />Missing other engineering information HW & Mechanics<br />
  17. 17. Inconsistency<br />Documents contained statements inconsistent with other statements about the same topic in the same document or in the same validation package.<br />Jargon<br />Varying Terminology<br />Contradiction in logic<br />
  18. 18. Lack of Needed Detail<br />This deficiency applied mostly to requirements documents. The requirements in the validation package did not adequately describe the software.<br />Poor requirements documents<br />Data<br />User Interaction<br />Key process<br />
  19. 19. Traceability<br />3 frequent traceability problems:<br />The traceability matrix did not account for a traceable specification or an observation step in a test script<br />Broken trace (Barren or Orphans)<br />Requirement details were not explicitly numbered and traced to associated test steps.<br />
  20. 20. Vague Wording<br />Documents used generalities<br />“in accordance to an approved procedure”, or “applicable regulatory requirements”, or “all associated GxP and business processes”<br />vague words such as “may”, “possibly”, “more or less”, and “approximately”<br />
  21. 21. Unverifiable Test Results<br />Expected results were not described sufficiently so that an independent reviewer could compare and verify actual results. The IEEE Standard for Software Test documentation, Std. 829.1988, Clause 6.2.4 says you should, “...provide the exact value (with tolerances where appropriate) for each required output or feature”<br />For executed scripts, actual results were not recorded or captured in a way that allowed an independent reviewer to compare them to expected results.<br />Actual-result column with no reference to a screen shot<br />
  22. 22. GDP<br />3 frequent Good Documentation Practice problems:<br />Hand-recorded data and testing evidence, such as test results, were presented in a way that could cause doubts about their authenticity (Ex. Cross-Out) <br />Data that confirmed a specific requirement was hard to find in the evidence provided (for example, a busy screen shot crammed with data).<br />Handwritten corrections were made that changed the sense of a requirement or an expected test result<br />
  23. 23. Incomplete Testing<br />Test scripts did not fully or adequately test the associated requirement.<br />
  24. 24. Ambiguity<br />Text could be interpreted more than one way<br />The words “either” and “or” in a requirement are strong clues the text is ambiguous.<br />
  25. 25. Identifying the Most Vulnerable Documents<br />categorized the documents and records where most frequent deficiencies were found. They discovered that about 85% of findings were concentrated in six key documentation areas<br />
  26. 26. User dashboard<br />Personalized views<br />Easily add charts, and filters<br />
  27. 27. Full Coverage matrix<br />
  28. 28. Risk Management (FMEA)<br />Reduce project risk<br />Relate risk objects to ALM artifacts<br />Generate Risk Management Document<br />Easily track high-risk features<br />
  29. 29. Advanced ALM Analytics - BI<br />Include external data sources<br />Personalize your own dashboard<br />drill down reports<br />
  30. 30. Thank You<br />www.orcanos.com<br />
  31. 31. QPack Main Screen<br />Manage Full Lifecycle<br />Manage Your project<br />Html/Word editor<br />
  32. 32. Solution –Multiple Projects/Versions<br />Solution<br />
  33. 33. Requirements Management<br />Central repository<br />Automate process to improve communication<br />Full Traceability of customer needs<br />Freeze content to assure quality<br />Requirements backlog for future design<br />MS-Word Integration allow working with MS-Word<br />
  34. 34. Requirements Management<br />requirements tree (SRS) <br />workflow<br />Set priorities and effort estimation<br />Customized templates<br />
  35. 35. Market -> Software Traceability<br />
  36. 36. Requirements Relations <br />Manage logical relations<br />Change in specific item might affect other places<br />
  37. 37. MS-Word Integration (2003/2007)<br />Manage multiple document types with full traceability (MRD, PRD, SRS, STP, STR…)<br />Generate MS-Word documents and synch changes back<br />Embed tractability reports in documents<br />
  38. 38. MS-Word Integration<br />
  39. 39. Requirements Backlog/Pool<br />Manage future releases in backlog<br />Evaluate change before implementation<br />Customers<br />Market<br />Product<br />Product Road Map - Pool<br />Product Implementation and versions<br />
  40. 40. QA – Test Plan<br />Plan tests according to requirements to assure requirements are tested<br />Manage test parameters for complex hardware configurations<br />
  41. 41. Test Plan<br />Test is part of release content<br />Automatic cover on creation<br />Test covers multiple requirements reduce test maintenance <br />Requirement<br />Requirement<br />Test case<br />
  42. 42. Test Parameters<br />Test steps<br />Test steps ensure testing consistency<br />Test parameters<br /><ul><li> Operating systems
  43. 43. WinXP Pro
  44. 44. Windows Vista
  45. 45. Linux
  46. 46. Browser type
  47. 47. Explorer 6
  48. 48. Explorer 7</li></ul>Test parameters reduce test maintenance overhead<br />Check scenarios:<br />Run test on Linux and IE6<br />Run test on Linux and IE7<br />Run test on WinXP and IE6<br />Run test on WinXP and IE7<br />…..<br />…..<br />
  49. 49. QA – Test Execution<br />Easily build test run suites for QA teams<br />Report defects from test run improves QA and development team communication<br />Automatic traceability of requirements by defects and test results<br />Track test results for better decision making and quality assurance<br />
  50. 50. Run Tests – QPack Test Execution<br />QA work plan for each release<br />Distribute tests to run between testers<br />
  51. 51. Test Run Screen<br />Tester receives test to dashboard<br />Run tests by steps<br />Pass/Fail test on step level<br />Parse test data by parameters<br />
  52. 52. Automatic Defect Report On Test Fail<br />Report defect from test on step failure<br />Defect automatically connected to the tests<br />Defect automatically routed to specific developer<br />Defect contains found version, severity, priority, etc.<br />Defect contains test data<br />Steps from test inserted to defect<br />
  53. 53. Test execution results<br />Create and store filters by release, status, execution sets, etc.<br />
  54. 54. Defect Tracking<br />Create custom views, reports and charts to make sure no defect is overlooked<br />Manage defects versions, branch defects to coop with multiple versions management<br />Dynamic defects workflow for team collaboration<br />Defects are traced back to requirements provides tracking in lower resolutions<br />
  55. 55. Defect Tracking<br />Report defects on specific requirement<br />Use dynamic filters to generate reports<br />Manage defects delta between versions<br />Allow customer to report defects<br />Defects list<br />Dynamic filters<br />
  56. 56. Tracking and Reports<br /><ul><li>Bring information together in a single business information tool with QPack reporting and QPack BI
  57. 57. Common access to all ALM artifacts
  58. 58. Get Up-To-Date coverage of requirements by all ALM artifacts
  59. 59. Create unlimited filters with QPack Open Framework</li></li></ul><li>Requirements Coverage By Defects <br />Create and store custom filter such as:<br />All defects breakdown by feature and status<br />All defects found in specific release<br />All open defects from previous release<br />Breakdown of high priority defects by assigned developers <br />Internal/External defects<br />
  60. 60. User dashboard<br />Personalized views<br />Easily add charts, and filters<br />
  61. 61. Internal/Customer Defects<br />Create and store filters by release, status, execution sets, etc.<br />
  62. 62. Defects Per Build<br />Create and store filters by release, status, execution sets, etc.<br />
  63. 63. Release manager<br />Track changes on each release<br />Monitor release quality<br />Monitor test execution progress<br />
  64. 64. Implementing QPack<br />Organization Analysis<br />Organization structure<br />Products and solutions<br />R&D teams<br />Existing tools<br />Existing methodologies<br />Implementation<br />Standards and documents management<br />Market requirements<br />Product requirements<br />Coverage and traceability<br />Test plan and execution<br />Defects tracking<br />Reporting<br />Administration<br />
  65. 65. Roadmap<br />Integrate with Siemens Teamcenter PLM solution<br />Provide regulations support kits (ISO, CMMI, FDA…)<br />Hardware testing support (Complex configurations, integration with HW testing platforms)<br />Agile support<br />3rd party plug-ins such as test automation tools, IDE, source control, CRM and helpdesk<br />Document Management System<br />BI next generation – ALM prediction<br />Provide internet based solution<br />
  66. 66. Delivering Real Value<br /> “QPack really gives us the possibility to control a project. We set up a workflow according to our company´s structure and size. This workflow ensures a standardized development process..." Quategra<br /> We have chosen the QPack system because we believe that it provides a comprehensive solution for managing our products development process in one integrated tool, with its intuitive user interface and reporting capabilities. Incredimail<br />"Orcanos and QPack™ system are the ultimate solution for us; Orcanos managers helped us with professional experience on the methodology and with system customization to our needs...“<br />Alvarion<br />
  67. 67. Summary<br />
  68. 68. QPack Benefits Summary<br /><ul><li>Track projects using a unified ALM platform
  69. 69. One Stop-shot for better collaboration
  70. 70. Full traceability from market to R&D
  71. 71. Regulatory compliance (ISO, CMMI, FDA…)
  72. 72. ALM analytics and BI executive dashboards
  73. 73. Easy implementation, small learning curve
  74. 74. Better visibility of change impact on resources and budget
  75. 75. Track customer needs to improve satisfaction
  76. 76. Make the right investments according to customer, risk, quality, and timeline</li>

×