Download as PDF, PPTX
Uploaded byMontrium
Implementing Metrics & Completeness Reporting in TMF Management
The document is a webinar agenda and content overview focused on improving metrics and completeness reporting in TMF (Trial Master File) management. Key topics include current flaws in completeness and timeliness measurements, the importance of integrating process modeling and data management techniques to enhance quality metrics, and the potential for risk scoring to improve inspection readiness. The speaker, Paul Fenton from Montrium, emphasizes the need for a more accurate and data-driven approach to TMF management to ensure compliance and effective clinical trial operation.
More Related Content
Similar to Implementing Metrics & Completeness Reporting in TMF Management
Implementing Metrics & Completeness Reporting in TMF Management
- 2. Implementing Metrics & Completeness Reporting inTMF Management eTMF Bootcamp Webinar Series Tuesday, 27th August 2019 2
- 3. 3 Agenda LIVE WEBINAR • Whatare the key metrics we should be tracking • How we generate metrics today • Why is today’s approach flawed • New approaches for tomorrow
- 4. 4 House Keeping L I VE W E B I N A R • This webinar is being recorded and will be made available after this session • Feel free to use the chatbox to submit your questions at anytime • Q&A will take place at the end of the webinar • We will send these slides to your email at the end of the webinar
- 5. 5 Meet Your Speaker Paul Fenton— Presidentand CEO, Montrium
- 6. • Founded in2005 • Working Exclusively in the Life Sciences • Headquartered in Montreal, Canada • EU HQ in Brussels • Clients in North America, Europe & Asia • Leading Content Management Platform • Over 8000 Users in 20+ Countries • Experienced Professional Services Group 6 About Montrium Connecting People, Processes & Technology A B O U T T H E C O M P A N Y
- 7. 7 Electronic Content Management &Business Intelligence for Clinical Trials C O N N E C T P L A T F O R M Regulatory Document Management & Submission Planning for Drugs and Devices Integrated and Data Driven Quality Management Solutions for Life Sciences
- 8. Accurate measurements areimportant….
- 9. Dimensions: Program, Study, Country,Site, Process Zone, time etc. • Complete • Expected • Missing • Lost • Revised • Artifacts reviewed • ICF reconciliation • Artifacts audited • Rejection rates • Anomaly rates • Risk scores What are the key metrics we should be tracking? Quality Completeness Timeliness • Days to review • Days to filing • % late filing • Issue resolution time Is the TMF of sufficient quality to tell an accurate story? Does the TMF contain all information to tell an accurate story? Was the information provided in a timely manner?
- 10. Completeness: How dowe do it today? % Complete = Total Number of Final Artifacts Expected Artifacts • Based on a predefined list of expected artifacts at the study, country, site and investigator level • Not typically dynamic based on study events • Is often maintained manually • Typically only a very high level indicator and prone to inaccuracy
- 11. Timeliness: How dowe do it today? % Filed on Time = (Final Artifact File date – Artifact Creation Date) < Expected Days to Filing Total Final Artifacts • Timeliness is key when compiling the TMF as the longer it takes to get information into the TMF the higher the risk that information may be inaccurate or lost • Defining standards for how long it should take to file is key • Identifying when you consider something is final and what dates you will use to calculate is important • Many systems today are not sophisticated enough to allow for variability in methods of timeliness calculation • Timeliness may go beyond just the filing of TMF artifacts…
- 12. Quality: How dowe do it today? % Reviewed = Final Artifacts Reviewed Total Final Artifacts • There are multiple metrics which allow us to evaluate the quality of the information stored in our TMF • We typically evaluate quality on an artifact by artifact basis through manual QC and Review • To truly evaluate quality we should look more holistically at our ability to tell an accurate story across artifacts • We should also focus on logical quality checks across artifacts • Risk based metrics would also greatly improve our ability to assess quality Rejection Rate = Total Artifacts Rejected Total Final Artifacts
- 13. How accurate doyou feel that your eTMF metrics are today? A) Very accurate B) Somewhat accurate C) Not accurate at all D) Don’t know 13 POLL A B O U T T H E C O M P A N Y
- 14. Why are todays methods flawed? • Completenessis inaccurate because it does not take into account study event and study process information • Artifacts can be distributed in multiple systems and organizations making overall calculation of completeness challenging • Quality checks are not targeted based on risks and logical checks and is often based on 100% manual reviews which are subject to human error • Timeliness is difficult to calculate and tends to be universal rather than specific to different types of artifacts
- 15. How can weimprove the accuracy of our metrics? 15
- 16. Predictive completeness § Completeness todayis far from accurate § To calculate accurate completeness we need information § By taking a process modeling approach to TMF coupled with system integration we can improve things § Process modelling allows for the generation of placeholders based on expected processes, event and artifacts § Processes are triggered from underlying data sources § Machine learning could be used to continuously improve on this
- 17.
- 18. • Today’s eTMFsystems organize content based on the reference model • It is difficult to reconstruct events using the RM • Clinical process models will allow better visualizations of specific processes • Artifacts are clustered around these models • Processes are triggered from events • Significantly improve the telling of the story during inspections 18A B O U T T H E C O M P A N Y Process-based visualizations
- 19.
- 20. Data driven approach to improve quality metrics •Today we tend to focus more on filing documents • While the quality of the individual documents are important, the eTMF should also be logical in its content • By applying data management type techniques, similar to what we use in EDC, we can run logical checks on our eTMF metadata to ensure that it tells an accurate story • This approach will allow us to gain a degree of confidence in the quality of our TMF in addition to the quality of the artifacts themselves
- 21. • Anomalies couldbe detected based on predefined rules which are very similar to edit checks that we use in EDC • Rules would typically be logical in nature and involve the verification of event sequence, dates, versions etc. • Anomalies would automatically raise queries • Anomalies would contribute to risk scoring • An eTMF health score could be generated to give an idea of the logical health of an eTMF TMF Health Metrics: Anomaly detection
- 22. Have you experiencedissues with missing or incorrect TMF information before or during an inspection? a) Yes b) No c) Don’t know 22 POLL A B O U T T H E C O M P A N Y
- 23. • Inspectors requiredirect access to systems that contain TMF content • Building systems that can track inspector activity is key to understanding inspector activity • Remote access is becoming a reality and so we need better tools to manage remote inspections • We should correlate findings with inspector activity to gain better understanding of what was reviewed and how findings were qualified • We could generate inspection related metrics which could feed in to risk scoring 23A B O U T T H E C O M P A N Y Inspection tracking and trending metrics
- 24. Risk Scoring: TheUltimate Quality Metric eTMF Risk Score Timeliness/ Delays Completeness/ Missing Artifacts Artifacts Linked to Safety, Efficacy or Endpoints Subject Recruitment Levels Previous Performance (Site/Country) Number and Type of Events Previous Inspection Trends Anomaly rate Rejection Rate 24
- 25. Risk modelling and scoring • Riskscoring would allow us to focus on high risk areas which may be more prone to inspection • Risk scoring over time would improve through machine learning and by taking into account eTMF inspection trends • Risk scores could be calculated on individual artifacts, processes, sites, countries, contributors etc. • Scores could be rolled up to create aggregate scores to the clinical program level • Risk metrics could be presented in an easy to consume visual manner similar to completeness metrics • Risk scoring could not only help manage inspection risk but also clinical risk in the future
- 26. Benefits of new approaches • Bytaking more of a processed based approach we can improve our ability to generate accurate completeness metrics • We should aim to leverage data management techniques to run health checks on our eTMFs and generate health metrics • Risk scoring will allow us to create more comprehensive quality metrics which not solely based on manual reviews and rejection rates We need to get a better handle on our ability to tell an accurate story and accurate comprehensive metrics will pave the way…
- 27. 27 • Facilitates exchangeof clinical trial information to all stakeholders • Accurately tracks the progress of TMF completeness • Allows you to quickly comply with regulatory requirements, audits and inspections A Complete eTMF Platform Engineered for Growth Organizations
- 28. Are you interestedin receiving more information about eTMF Connect? 1) Yes, It could be useful 2) No, not interested 28 POLL A B O U T T H E C O M P A N Y
- 29. QUESTIONS? 29A B OU T T H E C O M P A N Y
- 30.