Download to read offline
![[Merit trac webinar] - it is assessments that cause improvement](https://cdn.slidesharecdn.com/ss_thumbnails/merittracwebinar-itisassessmentsthatcauseimprovement-170404072823-thumbnail.jpg?width=640&height=640&fit=bounds)
Ka Overview June2010
- 1.
- 2. AgendaIntroductionsReview previous discussionsKnowledgeAdvisors’CapabilitiesOverview of data capture and analysis process/toolsDiscuss pilot program2© 2010, KnowledgeAdvisors
- 3. Who we areKnowledgeAdvisorsis a human capital analytics firm. We are the world’s largest provider of learning and talent measurement solutions. 3© 2010, KnowledgeAdvisors
- 4.
- 5.
- 6.
- 7.
- 8. Dr. Don Kirkpatrick(retired)
- 9.
- 10.
- 11.
- 12. Outsourced Solutions4© 2010,KnowledgeAdvisors
- 13. Metrics that MatterAutomatedData CollectionPush ReportingRobust Filtering and Analytics 5© 2010, KnowledgeAdvisors
- 14. Benchmarks & Analytics750Million Data Points with many benchmark “slices”Reporting that puts data into context and communicates resultsDashboards to visually interpret trends and relationships Information distribution that reduces administrative burden6© 2010, KnowledgeAdvisors
- 15. What we doAutomatethe collection of learning evaluation data using validated and standardised instrumentsAggregate and present the data in a format that is meaningful for learning & HR professionalsProvide benchmarking to bring relevance to the dataProvide expertise to drive continuous improvement7© 2010, KnowledgeAdvisors
- 16. Why we doitAll the work by designers, facilitators, participants, coaches and managers is for naught unless it manifests as results of importance to the business. Credible, reliable measures are needed to determine whether the training investment is paying dividends and what refinements are needed to further improve the program’s effectiveness.The only way to prove that resources have been well used is to continuously document and communicate relevant results. Outcomes must be communicated to drive continual improvement in learning and development. Effectiveness and efficiency are increased by rigorously assessing the quality of the current output, implementing ideas for improvement, measuring the impact and repeating the procedure.8© 2010, KnowledgeAdvisors
- 17. Benefits of Metricsthat MatterProvide ROUGHLY REASONABLE INDICATORS of learning performance
- 18. Shift focus toleading, versus lagging indicators
- 19. Trainers, program manager,managers, coaches, and mentors are ranked and benchmarked, resulting in improved preparation & performance
- 20. Enables productive dialoguebetween HR and business.
- 21. Data informs allaspects of HR including Leadership, Recruiting, L&D, Performance, Engagement, and Talent.9© 2010, KnowledgeAdvisors
- 22. Organisations Served10© 2010,KnowledgeAdvisors
- 23. Streamline Evaluation11Post EventEvaluationLMS EnrollmentInstant feedback
- 24.
- 25.
- 26.
- 27. Learner demographicsFollow UpEvaluationImpact, results indicators
- 28. Post-class support feedbackManagerEvaluationImpact and results indicators and validation
- 29. Support reinforcementData istransformed into Business Intelligence© 2010, KnowledgeAdvisors
- 30. Smile Sheets intoSmart Sheets© 2010, KnowledgeAdvisors
- 31. Value Creation vs.Value DemonstrationEfficiencyconstructs (how much we train)Performanceconstructs (how well we train) Impactconstructs (value we create) Availability of DataImportance to Senior Management
- 32. Reporting Strategy -Identify Stakeholder KPI’s14© 2010, KnowledgeAdvisors
Editor's Notes
- #4 Thank you for joining us today at KnowledgeAdvisors for a brief introduction & demonstration of Metrics that Matter. My name is Megan and I will be spending the next 10-15 minutes walking you through an overview of our measurement and evaluation software – Metrics that Matter – and after that KnowledgeAdvisors staff will be available to answer any questions that might come up during the presentation. For those of you not familiar with KnowledgeAdvisors I will give you a quick introduction to who we are.We are a human capital analytics firm and we focus solely on the measurement of learning and talent processes. There are 3 basic building blocks to the value of the KA solution. ….
- #5 Our fundamentals are guided by our advisory Bd includes Dr. Don Kirkpatrick, who developed the most widely leveraged method of lrng measurement w/ 4 levels of lrng; Dr Jac Philips – responsible for the ROI methodology or 5th level as it is referred to, Dr. Robert Brinkerhoff who developed Success Case Method, Dr. Jac Fitz-Enz who is leader in human capital measurement, and our latest addition is Josh Bersin who is a contemporary thought leader in learning space.Our on-staff consulting team is comprised of experts in strategy and statistical analysis to help our clients create sustainable and actionable measurement practices.
- #6 Our on-demand evaluation software is called MTM….this is the tech that automates data collection and reporting. It is a true database with a user interface to make slicing and dicing of data simple for the talent program practitioner.
- #7 The system houses about a billion benchmark datapoints which are leveraged in reporting so your data becomes more actionable.
- #12 With MTM technology integrations, there is no more manual administration involved in the evaluation, testing, assessment or surveying process. Integrating MTM with your LMS allows the two technologies to ‘talk’ to each other as the LMS passes registration details to MTM for easy and error-free data collection, processing and reporting.The client can control the timing of data collection as well.
- #13 The KA SmartSheet has been shaped over the past 10 years by industry experts…Grouped to enable aggregation of predictive data according to levels of learning.99.9% of our clients leverage our evaluation templates and take advantage of the benchmarking as a result. Many have done some customization to fit their culture and unique circumstances. Slightly modified versions that leverage standard templates allow our customers to gather data to help them better manage their business as well as provide their clients with evidence of success and impact.The questions are mapped to Kirkpatrick, Philips and Bersin methodologies. Learners are first asked to predict results and then later asked for historical results to shore up their predictions. Are there questions?.....People in org’s heavily influencedKMatt Barney – Motorola University – Leaders in Measurement/Analysis – approach based on constructs and normative data that would allow you to convert any scale into a normative scale for benchmarkingLess about individual questions, the power is in the construct – Matt was also responsible for the 6 sigma at MotorolaTherefore, this is a methodology based on 6 sigmaShaped by executives from leading technology, accounting, government agencies, and MicrosoftProven reliable/predictive model, and KPMG applies in internally to accomplish thisBontis research is reliable/valid
- #15 One area that is very important is to identify your various stakeholders and what is important to each. Your KA solution can support a wide range of needs…so for example….
- #16 We have designed reports around commonly asked questions and objectives…..If you want to look through the entire reporting catalog you can do that as well and find reports by data source and analysis objective. Now what I am going to do is pull up four reports to demonstrate the variety of outputs available. While these are pre-run each report does come with a full query interface to establish filters, time frames, and other preferences. This first example is a great way to highlight sucesess and quickly find areas for improvement
- #17 All reports can be shared via email, scheduled, or downloaded into Excel. This is a report from the detailed results catalog – these reports will show results at student or question level detail.
- #19 Once a report is run you can drill down in many ways for analysis.
- #20 MTM can track not only against a benchmark but goals can be set at the question category or learning level as well.
- #21 This is a sample from the Human Capital ROI Scorecard. This report helps illustrate where there may be scrap learning and ROI indicators.
- #22 This is the Analyst Worksheet. One of the business results tools. This worksheet can be used before or after a training program to discuss what other factors are driving results and get a more realistic view of the impact of training.
- #25 This is an example of an Actual Results tracker. This is where data from other systems can be input manually or uploaded via .csvThis data can also be displayed in dashboard widgets.
- #26 Metrics that Matter is divided into 3 main sections:Dashboards: Roll up data collected through out Metrics that Matter Reporting: Ad hoc access to report catalogAdmin: Manage account & Authoring This page shows the MTM dashboard that can be customized by user and can display data from learning evaluations or business metrics entered into an Actual Results tracker.
- #31 This is one of two “document” reports for learning. The Talent Development Summary report rolls up many classic MTM reports into one document across all training. The other report is the “Instructor Summary Report” that is the same format but is created for an individual instructor for their training only.