Recommended
Recommended
More Related Content
What's hot
What's hot (15)
Viewers also liked
Similar to Overview of KnowledgeAdvisors' Learning Analytics Capabilities
Similar to Overview of KnowledgeAdvisors' Learning Analytics Capabilities (20)
Overview of KnowledgeAdvisors' Learning Analytics Capabilities
- 2. Agenda Introductions Review previous discussions KnowledgeAdvisors’ Capabilities Overview of data capture and analysis process/tools Discuss pilot program 2 © 2010, KnowledgeAdvisors
- 3. Who we are KnowledgeAdvisors is a human capital analytics firm. We are the world’s largest provider of learning and talent measurement solutions. 3 © 2010, KnowledgeAdvisors
- 8. Dr. Don Kirkpatrick (retired)
- 12. Outsourced Solutions4 © 2010, KnowledgeAdvisors
- 13. Metrics that Matter Automated Data Collection Push Reporting Robust Filtering and Analytics 5 © 2010, KnowledgeAdvisors
- 14. Benchmarks & Analytics 750 Million Data Points with many benchmark “slices” Reporting that puts data into context and communicates results Dashboards to visually interpret trends and relationships Information distribution that reduces administrative burden 6 © 2010, KnowledgeAdvisors
- 15. What we do Automate the collection of learning evaluation data using validated and standardised instruments Aggregate and present the data in a format that is meaningful for learning & HR professionals Provide benchmarking to bring relevance to the data Provide expertise to drive continuous improvement 7 © 2010, KnowledgeAdvisors
- 16. Why we do it All the work by designers, facilitators, participants, coaches and managers is for naught unless it manifests as results of importance to the business. Credible, reliable measures are needed to determine whether the training investment is paying dividends and what refinements are needed to further improve the program’s effectiveness. The only way to prove that resources have been well used is to continuously document and communicate relevant results. Outcomes must be communicated to drive continual improvement in learning and development. Effectiveness and efficiency are increased by rigorously assessing the quality of the current output, implementing ideas for improvement, measuring the impact and repeating the procedure. 8 © 2010, KnowledgeAdvisors
- 18. Shift focus to leading, versus lagging indicators
- 19. Trainers, program manager, managers, coaches, and mentors are ranked and benchmarked, resulting in improved preparation & performance
- 20. Enables productive dialogue between HR and business.
- 21. Data informs all aspects of HR including Leadership, Recruiting, L&D, Performance, Engagement, and Talent.9 © 2010, KnowledgeAdvisors
- 22. Organisations Served 10 © 2010, KnowledgeAdvisors
- 29. Support reinforcementData is transformed into Business Intelligence © 2010, KnowledgeAdvisors
- 30. Smile Sheets into Smart Sheets © 2010, KnowledgeAdvisors
- 31. Value Creation vs. Value Demonstration Efficiency constructs (how much we train) Performance constructs (how well we train) Impact constructs (value we create) Availability of Data Importance to Senior Management
- 32. Reporting Strategy - Identify Stakeholder KPI’s 14 © 2010, KnowledgeAdvisors
- 33. 15 © 2010, KnowledgeAdvisors
- 34. Automate and Share Reports Detailed Result Catalog View Results by Category, Student and Question 16 © 2010, KnowledgeAdvisors
- 35. Crosstab Compare question categories, courses, instructors, locations, etc Color codes scores relative to your selected benchmark: red (under benchmark) yellow (at benchmark) green (exceeding benchmark) 17 © 2010, KnowledgeAdvisors
- 36. Find Strengths & Weaknesses Compare Results Internally and Against a Benchmark Drill Into Details 18 © 2010, KnowledgeAdvisors
- 37. Compare to Internal Goals and External Benchmarks 19 © 2010, KnowledgeAdvisors
- 38. Understand when training is applied or why it isn’t being applied Calculate Reasonable and Scalable ROI Metrics 20 © 2010, KnowledgeAdvisors
- 39. Consultatively Discuss Training Impact 21 © 2010, KnowledgeAdvisors
- 40. 22 © 2010, KnowledgeAdvisors
- 41. “Business Results” Imported into MTM 23 © 2010, KnowledgeAdvisors
- 42. Import or Manually Track Business Results Determine Goals and Color Coding Display Results in Dashboard Widgets 24 © 2010, KnowledgeAdvisors
- 43. Customizable Dashboards Add / Delete Tabs Add Delete Widgets Email and Download Import Results
- 44. 26 © 2010, KnowledgeAdvisors
- 45. Instructor Performance Summary Communicates Value Provides an overview of performance of an individual Instructor. Highlights Areas for Improvement Performance by InstructorQuestion Category. Top/bottom analysis of Coursesand Classes Trend Analysis on Ratings and Post Event submissions Qualitative Feedback 27 © 2010, KnowledgeAdvisors
- 46. Instructor Performance Summary A single report containing the full story of an instructor’s performance 28 © 2010, KnowledgeAdvisors
- 47. Talent Development Summary Communicates Value Provides a comprehensive view of talent and development Quarterly report format (Word 2007 or PDF) Highlights Areas for Improvement Top/bottom analysis of Courses, Instructors, Learning Providers and Locations Trend Analysis on Learning Methodologies andQuestion Categories Impact and Value analysis by Demographics and Courses On the Job impact and Barriers to Use analysis 29 © 2010, KnowledgeAdvisors
- 48. A single report containing the full story of an organization’s performance 30 © 2010, KnowledgeAdvisors
- 49. Next steps Pilot Program 4-6 month trial of Metrics that Matter Outsourced Administration Identify one or multiple programs to evaluate Participate in all steps Outcomes include comprehensive review of reporting and analytics using your own program results 31 © 2010, KnowledgeAdvisors
- 50. Contact Us Stuart Halliday Sales Director 07717 507 945 shalliday@knowledgeadvisors.com www.knowledgeadvisors.com
Editor's Notes
- Thank you for joining us today at KnowledgeAdvisors for a brief introduction & demonstration of Metrics that Matter. My name is Megan and I will be spending the next 10-15 minutes walking you through an overview of our measurement and evaluation software – Metrics that Matter – and after that KnowledgeAdvisors staff will be available to answer any questions that might come up during the presentation. For those of you not familiar with KnowledgeAdvisors I will give you a quick introduction to who we are.We are a human capital analytics firm and we focus solely on the measurement of learning and talent processes. There are 3 basic building blocks to the value of the KA solution. ….
- Our fundamentals are guided by our advisory Bd includes Dr. Don Kirkpatrick, who developed the most widely leveraged method of lrng measurement w/ 4 levels of lrng; Dr Jac Philips – responsible for the ROI methodology or 5th level as it is referred to, Dr. Robert Brinkerhoff who developed Success Case Method, Dr. Jac Fitz-Enz who is leader in human capital measurement, and our latest addition is Josh Bersin who is a contemporary thought leader in learning space.Our on-staff consulting team is comprised of experts in strategy and statistical analysis to help our clients create sustainable and actionable measurement practices.
- Our on-demand evaluation software is called MTM….this is the tech that automates data collection and reporting. It is a true database with a user interface to make slicing and dicing of data simple for the talent program practitioner.
- The system houses about a billion benchmark datapoints which are leveraged in reporting so your data becomes more actionable.
- With MTM technology integrations, there is no more manual administration involved in the evaluation, testing, assessment or surveying process. Integrating MTM with your LMS allows the two technologies to ‘talk’ to each other as the LMS passes registration details to MTM for easy and error-free data collection, processing and reporting.The client can control the timing of data collection as well.
- The KA SmartSheet has been shaped over the past 10 years by industry experts…Grouped to enable aggregation of predictive data according to levels of learning.99.9% of our clients leverage our evaluation templates and take advantage of the benchmarking as a result. Many have done some customization to fit their culture and unique circumstances. Slightly modified versions that leverage standard templates allow our customers to gather data to help them better manage their business as well as provide their clients with evidence of success and impact.The questions are mapped to Kirkpatrick, Philips and Bersin methodologies. Learners are first asked to predict results and then later asked for historical results to shore up their predictions. Are there questions?.....People in org’s heavily influencedKMatt Barney – Motorola University – Leaders in Measurement/Analysis – approach based on constructs and normative data that would allow you to convert any scale into a normative scale for benchmarkingLess about individual questions, the power is in the construct – Matt was also responsible for the 6 sigma at MotorolaTherefore, this is a methodology based on 6 sigmaShaped by executives from leading technology, accounting, government agencies, and MicrosoftProven reliable/predictive model, and KPMG applies in internally to accomplish thisBontis research is reliable/valid
- One area that is very important is to identify your various stakeholders and what is important to each. Your KA solution can support a wide range of needs…so for example….
- We have designed reports around commonly asked questions and objectives…..If you want to look through the entire reporting catalog you can do that as well and find reports by data source and analysis objective. Now what I am going to do is pull up four reports to demonstrate the variety of outputs available. While these are pre-run each report does come with a full query interface to establish filters, time frames, and other preferences. This first example is a great way to highlight sucesess and quickly find areas for improvement
- All reports can be shared via email, scheduled, or downloaded into Excel. This is a report from the detailed results catalog – these reports will show results at student or question level detail.
- Once a report is run you can drill down in many ways for analysis.
- MTM can track not only against a benchmark but goals can be set at the question category or learning level as well.
- This is a sample from the Human Capital ROI Scorecard. This report helps illustrate where there may be scrap learning and ROI indicators.
- This is the Analyst Worksheet. One of the business results tools. This worksheet can be used before or after a training program to discuss what other factors are driving results and get a more realistic view of the impact of training.
- This is an example of an Actual Results tracker. This is where data from other systems can be input manually or uploaded via .csvThis data can also be displayed in dashboard widgets.
- Metrics that Matter is divided into 3 main sections:Dashboards: Roll up data collected through out Metrics that Matter Reporting: Ad hoc access to report catalogAdmin: Manage account & Authoring This page shows the MTM dashboard that can be customized by user and can display data from learning evaluations or business metrics entered into an Actual Results tracker.
- This is one of two “document” reports for learning. The Talent Development Summary report rolls up many classic MTM reports into one document across all training. The other report is the “Instructor Summary Report” that is the same format but is created for an individual instructor for their training only.