Research Bulletin | 2011       BERSIN & ASSOCIATES                                  June 10, 2011                         ...
Research Bulletin | 2011                                    Research Bulletin | 2011       BERSIN & ASSOCIATES         BER...
Research Bulletin | 2011                                                                                                  ...
Research Bulletin | 2011                                            Research Bulletin | 2011       BERSIN & ASSOCIATES    ...
Research Bulletin | 2011                                   Research Bulletin | 2011       BERSIN & ASSOCIATES            B...
Research Bulletin | 2011                                    Research Bulletin | 2011       BERSIN & ASSOCIATES         BER...
Research Bulletin | 2011                                  Research Bulletin | 2011       BERSIN & ASSOCIATES          BERS...
Research Bulletin | 2011                                     Research Bulletin | 2011       BERSIN & ASSOCIATES          B...
Research Bulletin | 2011                                                  Research Bulletin | 2011         BERSIN & ASSOCI...
Research Bulletin | 2011                                      Research Bulletin | 2011       BERSIN & ASSOCIATES         B...
Research Bulletin | 2011                                   Research Bulletin | 2011      BERSIN & ASSOCIATES       BERSIN ...
Research Bulletin | 2011                                                     Research Bulletin | 2011       BERSIN & ASSOC...
Research Bulletin | 2011                                     Research Bulletin | 2011        BERSIN & ASSOCIATES          ...
Research Bulletin | 2011                                  Research Bulletin | 2011       BERSIN & ASSOCIATES         BERSI...
Research Bulletin | 2011                                  Research Bulletin | 2011       BERSIN & ASSOCIATES         BERSI...
Research Bulletin | 2011                                  Research Bulletin | 2011       BERSIN & ASSOCIATES         BERSI...
Research Bulletin | 2011                                       Research Bulletin | 2011       BERSIN & ASSOCIATES         ...
Research Bulletin | 2011                                     Research Bulletin | 2011       BERSIN & ASSOCIATES         BE...
Research Bulletin | 2011                                  Research Bulletin | 2011       BERSIN & ASSOCIATES         BERSI...
Research Bulletin | 2011                                   Research Bulletin | 2011       BERSIN & ASSOCIATES         BERS...
Research Bulletin | 2011                                  Research Bulletin | 2011       BERSIN & ASSOCIATES         BERSI...
Research Bulletin | 2011                                   Research Bulletin | 2011       BERSIN & ASSOCIATES         BERS...
Research Bulletin | 2011                                  Research Bulletin | 2011       BERSIN & ASSOCIATES         BERSI...
Research Bulletin | 2011                                  Research Bulletin | 2011       BERSIN & ASSOCIATES         BERSI...
Measuring On Demand Learning Bersin and Safari Books OnlineReport
Upcoming SlideShare
Loading in …5
×

Measuring On Demand Learning Bersin and Safari Books OnlineReport

1,539 views

Published on

Bersin & Associates - June 2011 Case Study

Published in: Education, Business
  • Be the first to comment

  • Be the first to like this

Measuring On Demand Learning Bersin and Safari Books OnlineReport

  1. 1. Research Bulletin | 2011 BERSIN & ASSOCIATES June 10, 2011 Volume 6, Issue 24 Measuring On-Demand Learning: Safari Books Online and the Value of Digital Libraries About the Author Introduction: Making an Impact Where Performance Happens Up until a few years ago, supporting on-demand learning was not something learning and development (L&D) functions had to worry about too much. L&D traditionally was more accustomed to designing, developing and delivering content. However, the modern Janet Clarey, learning organization is less concerned with formal learning – and Senior Analyst is more concerned with “where” performance happens and making sure that employees have the resources available to do their work1. As a result, new roles for L&D as “enablers and facilitators” of learning have emerged. This is especially important because first-line managers and supervisors know they need to keep their employees informed on industry advances and trends, but they simply cannot do that with constant change and the expectations of employees to have everything at their fingertips. The growing recognition of the importance of on-demand and the increasing speed at which the human capital needs of the organization are evolving, along with the changing needs andBERSIN & ASSOCIATES, LLC 180 GRAND AVENUE SUITE 320 OAKLAND, CA 94612 1 For more information, The High-Impact Learning Organization: WhatWorks® in (510) 251-4400 the Management, Governance and Operations of Modern Corporate Training, Bersin & INFO@BERSIN.COM Associates / Josh Bersin, May 2008. Available to research members at www.bersin.com/ WWW.BERSIN.COM library or for purchase at www.bersin.com/highimpact. THIS MATERIAL IS LICENSED TO SAFARI BOOKS ONLINE FOR DISTRIBUTION ONLY. BERSIN & ASSOCIATES © 2011
  2. 2. Research Bulletin | 2011 Research Bulletin | 2011 BERSIN & ASSOCIATES BERSIN & ASSOCIATES expectations for learning on the part of employees, bring a new KEY POINT challenge for L&D. How do you measure the value of on-demand Informal learning learning and more specifically an on-demand digital library? This research bulletin will help to answer that. is learning that is accidental, ad- This report will cover: hoc, unplanned and which happens without the discipline of instructional design. The Increasing Role of On-Demand Learning We categorize approaches to learning as “formal” and “informal.” Informal learning is learning that is accidental, ad-hoc, unplanned and which happens without the discipline of instructional design. It has typically referred to learning that training professionals did not necessarily pay attention to, but knew was occurring. We categorize informal learning into three broad types – on-demand learning, social learning and embedded learning. To help you to make sense of how on-demand learning fits as a corporate learning approach, we have developed the Bersin & Associates Enterprise Learning Framework2. (See Figure 1.)BERSIN & ASSOCIATES, LLC 6114 LA SALLE AVENUE SUITE 417 OAKLAND, CA 94611 2 (510) 654-8500 For more information, The Enterprise Learning Framework: A Modern Approach to INFO@BERSIN.COM Corporate Training, Bersin & Associates / Josh Bersin and David Mallon, April 2009. Available to research members at www.bersin.com/library. WWW.BERSIN.COM Measuring On-Demand Learning Janet Clarey | Page 2 THIS MATERIAL IS LICENSED TO SAFARI BOOKS ONLINE FOR DISTRIBUTION ONLY. BERSIN & ASSOCIATES © 2011 THIS MATERIAL IS LICENSED TO SAFARI BOOKS ONLINE FOR DISTRIBUTION ONLY. BERSIN & ASSOCIATES © 2011
  3. 3. Research Bulletin | 2011 Research Bulletin | 2011 BERSIN & ASSOCIATES BERSIN & ASSOCIATESFigure 1: Bersin & Associates Enterprise Learning Framework®Enterprise  Learning  Framework® Leadership Career Customer Project  and   Customer Learning   Development Development Onboarding Service Process Education Systems   Programs Management Technical Compliance Sales Product Channel IT  Skills Organization,  Governance  and  Management Professional Knowledge Training Audiences Jobs Roles Competencies Proficiencies Preferences Demographics Geographies Business  Problems & Problems Informal Learning  Architecture Formal Approaches Instructor  Led  Training Virtual  Classroom   On-­Demand Social Embedded Games E-­Learning Wikis,  Blogs,  Forums Performance  Support Simulations Search Expert  Directories Feedback Testing  and  Evaluation Books,  Articles Social  Networks Rotational  Assignments E-­Learning Videos Communities  of  Practice After  Action  Reviews Podcasts Conferences  and  Colloquium Quality  Circles Learning    /Knowledge  Portals Coaching  and  Mentoring Development  Planning Performance   Information   Content   Change   Measurement  and Consulting Architecture Development Management Evaluation Disciplines Instructional   Knowledge   Program   Community   Business Designs Management Management Management Intelligence LMS,  LCMS Content   Rich   Performance Reporting  and Tools Learning  Portals Development Collaboration Media Assessment Support Analytics &  Technology Talent  Mgt. Systems Content Management Social Networking Mobile Search, Tagging Virtual   Classroom Measurement Systems Executive Development Performance  and Innovation Employee Support Planning Talent  Management Programs Feedback Culture Learning  integrated Knowledge Customer Mentoring  and Learning  from   With  Business  Planning Sharing Listening Knowledge  Sharing Mistakes Source: Bersin & Associates, 2009. On-demand learning, the subject of this research bulletin, is considered a learner-driven activity that employees can use when and as needed. There are many forms of on-demand learning, such as videos, podcasts, knowledge portals and on-demand digital libraries.BERSIN & ASSOCIATES, LLC 6114 LA SALLE AVENUE SUITE 417 OAKLAND, CA 94611 (510) 654-8500 INFO@BERSIN.COM WWW.BERSIN.COM Measuring On-Demand Learning Janet Clarey | Page 3 THIS MATERIAL IS LICENSED TO SAFARI BOOKS ONLINE FOR DISTRIBUTION ONLY. BERSIN & ASSOCIATES © 2011 THIS MATERIAL IS LICENSED TO SAFARI BOOKS ONLINE FOR DISTRIBUTION ONLY. BERSIN & ASSOCIATES © 2011
  4. 4. Research Bulletin | 2011 Research Bulletin | 2011 BERSIN & ASSOCIATES BERSIN & ASSOCIATESFigure 2: Bersin & Associates Informal Learning Approaches       E-­‐Learning   Wikis,  Blogs,  Forums   Performance  Support   Search   Expert  Directories   Feedback   Books,  Articles   Social  Networks   Rotational  Assignments   Videos   Communities  of  Practice   After  Action  Reviews   Podcasts   Conferences  and  Colloquium   Quality  Circles   Learning    /Knowledge  Portals   Coaching  and  Mentoring   Development  Planning   Source: Bersin & Associates, 2009. The On-Demand Digital Library for On-Demand Learning Digital asset libraries are one way to support the increasing speed at KEY POINT which the human capital needs of the organization change. It also Digital asset supports the changing needs and expectations for learning on the part of employees. This is especially important because we are moving libraries are one toward learning that is 70 percent on the job, 20 percent collaborative way to support the and 10 percent formal3. increasing speed at which the human What follows is an example of how CDW is using an on-demand digital library to meet the immediate needs of its organization and its capital needs of employees’ expectations. the organization change.BERSIN & ASSOCIATES, LLC 6114 LA SALLE AVENUE SUITE 417 OAKLAND, CA 94611 (510) 654-8500 3 INFO@BERSIN.COM This idea has multiple sources. It is commonly attributed to the L&D model according to the Princeton University Learning Process. WWW.BERSIN.COM Measuring On-Demand Learning Janet Clarey | Page 4 THIS MATERIAL IS LICENSED TO SAFARI BOOKS ONLINE FOR DISTRIBUTION ONLY. BERSIN & ASSOCIATES © 2011 THIS MATERIAL IS LICENSED TO SAFARI BOOKS ONLINE FOR DISTRIBUTION ONLY. BERSIN & ASSOCIATES © 2011
  5. 5. Research Bulletin | 2011 Research Bulletin | 2011 BERSIN & ASSOCIATES BERSIN & ASSOCIATES Case in Point: Using an On-Demand Digital Library to Facilitate Needs of a High-Technology Workforce at CDW CDW has a population of 600 engineers who primarily design, implement and support IT solutions for its customers. It is of the utmost importance that its engineers stay abreast of emerging technologies. However, getting engineers to a physical classroom for training on a regular basis is just not feasible. CDW’s workforce is highly distributed with field engineers and other workers in remote offices. This, along with the challenge of finding available time for training, means the company had to find new ways to keep the workforce out in front of technology advances. One way in which CDW is meeting the technical training needs KEY POINT of its engineers is by providing them with access to the newest One way in which technology-related books online. This not only keeps them current on emerging technologies, but also supports their need CDW is meeting for certification, reference and filling skills gaps. the technical training needs of At many organizations, licenses for various on-demand digital its engineers is by libraries sit unused. CDW, on the other hand, gets the most from its on-demand digital library because the company actively providing them manages the resource. In the two years that the company has with access to the been using the Safari Books Online library, CDW learners have newest technology- been requesting more licenses than the company has. This is a related books good “problem.” online. CDW credits excellent facilitation of the on-demand digital library by its organizational capability department as a factor in its success. For example, organizational capacity not only sources the right on-demand resources, but it also has been instrumental in theBERSIN & ASSOCIATES, LLC organization of book clubs. In effect a community of 6114 LA SALLE AVENUE SUITE 417 OAKLAND, CA 94611 (510) 654-8500 INFO@BERSIN.COM WWW.BERSIN.COM Measuring On-Demand Learning Janet Clarey | Page 5 THIS MATERIAL IS LICENSED TO SAFARI BOOKS ONLINE FOR DISTRIBUTION ONLY. BERSIN & ASSOCIATES © 2011 THIS MATERIAL IS LICENSED TO SAFARI BOOKS ONLINE FOR DISTRIBUTION ONLY. BERSIN & ASSOCIATES © 2011
  6. 6. Research Bulletin | 2011 Research Bulletin | 2011 BERSIN & ASSOCIATES BERSIN & ASSOCIATES Case in Point: Using an On-Demand Digital Library (cont’d) BEST PRACTICE practice4, the book clubs are popular because they support Book clubs different learner preferences, as well as the collaboration and are popular at communication among members. Employees read new, relevant books and meet weekly to discuss them. A facilitator (often CDW because a team lead) leads the face-to-face community and provides they support an available conference bridge for those who cannot join different learner the session. preferences, Of course not everyone likes reading books online. While some as well as the enjoy the flexibility and easy access afforded online, others find collaboration and reading online difficult. The community of practice also provides communication a traditional way in which to develop professionally, face to among members. face, while collaborating and networking with co-workers. A facilitator The community also provides structure and support – needed leads the face-to- elements for learning transfer. face community In addition to on-demand digital books, CDW uses on-demand and provides CBT “nuggets,” which are accessed via the company’s learning an available management system (LMS). The video “nuggets” of knowledge conference bridge supplement other training and certifications for internal IT for those who groups in various roles. cannot join the How CDW Measures the Value of Its On-Demand Digital Library session. The company measures the value of its on-demand digital library in several ways. First, the company uses traditional measures, such as average time spent and total usage by its employees. The company compares the cost associated with actually buying a book versus accessing it online. CDW also looks at how many hours have been saved, as compared with offering similar content in a formal setting. The company compares its training budget over the previous year, looking at the amountBERSIN & ASSOCIATES, LLC 6114 LA SALLE AVENUE SUITE 417 4 A “community of practice” (or “CoP”) is often defined as a group of people who share OAKLAND, CA 94611 an interest or concern about a common topic, and who deepen their knowledge in this area (510) 654-8500 through ongoing interaction and relationship-building within their group. While communities INFO@BERSIN.COM often come into being spontaneously, they nonetheless require nurturing if they are to become valuable to the members and remain viable over the course of their evolution. WWW.BERSIN.COM Measuring On-Demand Learning Janet Clarey | Page 6 THIS MATERIAL IS LICENSED TO SAFARI BOOKS ONLINE FOR DISTRIBUTION ONLY. BERSIN & ASSOCIATES © 2011 THIS MATERIAL IS LICENSED TO SAFARI BOOKS ONLINE FOR DISTRIBUTION ONLY. BERSIN & ASSOCIATES © 2011
  7. 7. Research Bulletin | 2011 Research Bulletin | 2011 BERSIN & ASSOCIATES BERSIN & ASSOCIATES Case in Point: Using an On-Demand Digital Library (cont’d) BEST PRACTICE set aside for books. CDW then takes an average of that number CDW has a “use and compares costs with the online resources used. Finally, it it or lose it” uses employee surveys to measure end-user satisfaction. policy in place for CDW has a “use it or lose it” policy in place for its on-demand its on-demand digital library, which encourages people to use their licenses digital library, and weeds out people who will not. While the company wants to encourage the use of the on-demand digital library, which encourages it realizes that resources are finite and strives to maximize people to use their the organization’s investment by only giving licenses to licenses and weeds employees who will use them. CDW monitors usage using an out people who administrative portal. will not. Initial Benefits One of the greatest benefits to CDW is the ability to provide a service that enables engineers to stay on top of current technology and trends. For instance, its engineers like “rough cuts” – books that are viewable even before publication. Not surprisingly, CDW is now able to target a much larger audience, rather than relying on a physical library. Its engineers like to access the resources directly from a website. The ability to download books, so that they can be read on an e-reader or iPad application, is well-received by employees. Another benefit is improved time management for engineers. Access to on-demand resources keeps engineers from wasting valuable time searching for credible information. CDW also likes these resources because they provide less reliance on, and yet another option over, costly and time-consuming formal training. Advice for Other CompaniesBERSIN & ASSOCIATES, LLC CDW suggests that other companies considering an on-demand 6114 LA SALLE AVENUE digital library should start small, rather than with an enterprise SUITE 417 OAKLAND, CA 94611 license model. Taking this approach, the company believes, (510) 654-8500 allowed it to evaluate the resource and determine ROI. INFO@BERSIN.COM WWW.BERSIN.COM Measuring On-Demand Learning Janet Clarey | Page 7 THIS MATERIAL IS LICENSED TO SAFARI BOOKS ONLINE FOR DISTRIBUTION ONLY. BERSIN & ASSOCIATES © 2011 THIS MATERIAL IS LICENSED TO SAFARI BOOKS ONLINE FOR DISTRIBUTION ONLY. BERSIN & ASSOCIATES © 2011
  8. 8. Research Bulletin | 2011 Research Bulletin | 2011 BERSIN & ASSOCIATES BERSIN & ASSOCIATES Case in Point: Using an On-Demand Digital Library (cont’d) The company cautions others from just turning the resource “on” and walking away. CDW has had that happen in the past with a different on-demand digital library. When the company introduced this resource to employees, it included a scavenger hunt to incent the use of the resource. As a result, the company experienced a bit of a groundswell with more people than expected asking for licenses. Where Does CDW Go from Here? One of its goals is to make the online digital library part of a standardized program by tying resources to roles and / or courses. The company also hopes to add some more structure around recommendations. Right now, someone may talk recommendations come from co-workers, book reviews or word of mouth. So, the company is seeking to formalize this resource as part of its overall strategy. Why Use On-Demand Digital Libraries? The use of on-demand digital libraries is growing. Part of the reason KEY POINT is the ability of online books and digital assets, in general, to make The use of on- knowledge sources available to employees’ on-demand – especially new releases of books. These resources can be downloaded and read (or demand digital listened to) instantly on a desktop computer or on a number of mobile libraries is devices – something physical libraries cannot fully support. growing – in part because these Digital libraries can also save time and improve productivity. Knowledge and information workers spend too much time searching for answers resources can be in a sea of unstructured information from non-vetted resources. Even downloaded and when they do find an answer, accuracy is often not at an expert level. read & ASSOCIATES, LLCBERSIN(or listened This “lost-time looking” is a hidden cost of both lost productivity and to) instantly on a AVENUE 6114 LA SALLE the inability to pinpoint time-critical issues. desktop computer SUITE 417 OAKLAND, CA 94611 or on a number of (510) 654-8500 mobile devices. INFO@BERSIN.COM WWW.BERSIN.COM Measuring On-Demand Learning Janet Clarey | Page 8 THIS MATERIAL IS LICENSED TO SAFARI BOOKS ONLINE FOR DISTRIBUTION ONLY. BERSIN & ASSOCIATES © 2011 THIS MATERIAL IS LICENSED TO SAFARI BOOKS ONLINE FOR DISTRIBUTION ONLY. BERSIN & ASSOCIATES © 2011
  9. 9. Research Bulletin | 2011 Research Bulletin | 2011 BERSIN & ASSOCIATES Enabling  the  Knowledge  Worker BERSIN & ASSOCIATESFigure 3: FindingrightRight Information atrightRight Timein the best Right Format Finding the the information at the the time and and in the format…. What  do  you  believe  holds  back  knowledge  workers  in  your  organization? Overwhelming  volume  of  information   makes  it  difficult  to  notice  and  keep   68% The  problem  is track  of  useful  information. “context” not “content” Lack  of  effective  tools  (such  as   search)  makes  it  difficult  to  find  the   34% most  useful  information. Frequent  change  of  information   Reuse  and makes  it  difficult  to  find  the  most   32% current  information. standards badly Inconsistency  of  information  formats   needed or  sources  makes  it  difficult  to  use   23% and  comprehend  new  information. Dynamic  nature  of  job  roles  makes  it   difficult  to  find  sufficiently  targeted  or   16% relevant  information. Job  roles  or  conditions  make  it  difficult   12% to  access  sources  of  information. 0% 10% 20% 30% 40% 50% 60% 70% 80% Source: Bersin & Associates, 2009. Copyright  ©  2009  Bersin  &  Associates.  All  rights  reserved. Page  14 When it comes to finding the right on-demand digital resources, lagging organizations may find their employees wondering, “How is it that a 10-minutes search session with Google can often result in more substantive learning than searching our intranet?” No one wants employees spending a large part of their workday simply looking for the information they need to do their job or making them wait for it.BERSIN & ASSOCIATES, LLC 6114 LA SALLE AVENUE SUITE 417 OAKLAND, CA 94611 (510) 654-8500 INFO@BERSIN.COM WWW.BERSIN.COM Measuring On-Demand Learning Janet Clarey | Page 9 THIS MATERIAL IS LICENSED TO SAFARI BOOKS ONLINE FOR DISTRIBUTION ONLY. BERSIN & ASSOCIATES © 2011 THIS MATERIAL IS LICENSED TO SAFARI BOOKS ONLINE FOR DISTRIBUTION ONLY. BERSIN & ASSOCIATES © 2011
  10. 10. Research Bulletin | 2011 Research Bulletin | 2011 BERSIN & ASSOCIATES BERSIN & ASSOCIATESFigure 4: Advantages for Using Digital Asset Libraries Knowledge  sources     Ability  to  pinpoint  time-­‐critical   accessible  on-­‐demand  through   issues  resulting  in  more   multiple  channels   substantive  learning     Digital  Asset   Libraries   Less  time  and  fewer     Authoritative,  relevant,     opportunities  lost  in     up-­‐to-­‐date  information     searching  for  answers   from  experts   Source: Bersin & Associates, 2011. This loss of productivity impacts executives, as well as front-line knowledge workers. When we asked senior leaders about their own habits in searching for information, they said staying informed is costly and time-consuming. While they felt that they were informed, these senior leaders admitted that they devoted a significant amount of time to the process – about four hours per week. Most were “too busy” to read business books to find “authoritative” information. When they did read, more than one-half did so while traveling or while at home.5 Digital books by experts in their fields allowed them toBERSIN & ASSOCIATES, LLC read authoritative information and choose the amount of information 6114 LA SALLE AVENUE SUITE 417 5 OAKLAND, CA 94611 For more information, How Executives Stay Informed: A Study of Resources Used and Time Spent in Locating Critical Business Information, Bersin & Associates / Josh Bersin (510) 654-8500 November 2005. Available to library members at www.bersin.com. INFO@BERSIN.COM WWW.BERSIN.COM Measuring On-Demand Learning Janet Clarey | Page 10 THIS MATERIAL IS LICENSED TO SAFARI BOOKS ONLINE FOR DISTRIBUTION ONLY. BERSIN & ASSOCIATES © 2011 THIS MATERIAL IS LICENSED TO SAFARI BOOKS ONLINE FOR DISTRIBUTION ONLY. BERSIN & ASSOCIATES © 2011
  11. 11. Research Bulletin | 2011 Research Bulletin | 2011 BERSIN & ASSOCIATES BERSIN & ASSOCIATES they needed, whether it was several pages, a chapter or an entire book. High-value, expert-written materials offered in a well-organized reference format were viewed as a valuable business tool. Measuring On-Demand Learning Measurement has always been a profound challenge for L&D. Organizations struggle to measure the effectiveness and efficiency of the training itself, the overall training function, the business value of training in general, and training’s alignment and contribution to overall business goals. Our research tells us that measurement is consistently rated as an area that senior and midlevel training managers want to improve.6 Measuring the approaches to learning that fall under the umbrella of the term “informal learning” (which encompasses everything from on- the-job performance support to communities of practice) creates yet another measurement challenge. For instance, how do you go about measuring the effectiveness and efficiency of reading a chapter in a book? Our Impact Measurement Framework7, adapted for informal learning environments, uses our Business Impact Model for Training8 to identify specific measures that can be used to monitor and evaluate the causal elements which drive business impact. (See Figure 5.) 6 For more information, High-Impact Learning Measurement: Best Practices, Models, and Business-Driven Solution for the Measurement and Evaluation of Corporate Training, Bersin & Associates / Josh Bersin November 2006. Available to research members at www.bersin. com/library or for purchase at www.bersin.com/measurement.BERSIN & ASSOCIATES, LLC 7 For more information, The Impact Measurement Framework: An Update for the 6114 LA SALLE AVENUE Measurement of Informal Learning, Bersin & Associates / Josh Bersin, May 3, 2011. Available SUITE 417 to research members at www.bersin.com/library. OAKLAND, CA 94611 8 (510) 654-8500 For more information, High-Impact Learning Measurement: Best Practices, Models, and Business-Driven Solution for the Measurement and Evaluation of Corporate Training, Bersin INFO@BERSIN.COM & Associates / Josh Bersin November 2006. WWW.BERSIN.COM Measuring On-Demand Learning Janet Clarey | Page 11 THIS MATERIAL IS LICENSED TO SAFARI BOOKS ONLINE FOR DISTRIBUTION ONLY. BERSIN & ASSOCIATES © 2011 THIS MATERIAL IS LICENSED TO SAFARI BOOKS ONLINE FOR DISTRIBUTION ONLY. BERSIN & ASSOCIATES © 2011
  12. 12. Research Bulletin | 2011 Research Bulletin | 2011 BERSIN & ASSOCIATES BERSIN & ASSOCIATESFigure 5: Impact Measurement Framework – Adapted for Informal Learning Satisfaction Learning Adoption Utility Efficiency Alignment Attainment Did  you  reach  the   How  well  do  the   How  efficient  and   How  well  were   How  well  did  you  meet   programs  solve desired  audience? cost-­effective  was   program  business   specifically  defined   the workforce’s Did  they  complete   particular  problems?   How  did  it  compare   priorities  defined? client  (business  user   or  comply  as  desired? How  well  did  it  align   to  other  similar   How  well  did  business   or  customer)  objectives? Who  did  not  comply   to  the  specific  job programs  or   units  buy  in  on   These  may  be  revenue,   and  why? related  problems   competitive  programs? the  value  of  this   time  to  market, and  issues? How  well  did  it  use   program  relative  to   compliance, Would  learners   the learner’s time ? other  investments? time  to  complete,  etc. recommend this   Program program  to   Design their  peers? Individual  Performance Organizational  Performance General  business  measures  or  HR  measures  which   Indicators  asked  of  learners  and  managers  to  gauge   are  already  captured  in  the  organization  (ie.  Engagement,   performance  improvements.  Specific  operational  measures retention).  Special  surveys  can  be  used  to  determine   identified  in  the  performance  consulting  process. indicators using the “wisdom of crowds.” Source: Bersin & Associates, 2011. For measuring on-demand learning, we like to use the word “utility” to KEY POINT indicate the “usefulness” of on-demand digital assets to the individual Copyright  ©  2011  Bersin  &  Associates.  All  rights  reserved. Page  1 The Bersin & “learning” and his / her workgroups. It is an indicator9 of performance which is easier to measure than the performance itself. Questions that Associates Impact are used to measure utility might include the following. Measurement Framework also includes efficiency and effectiveness of your current job? contribution, feedback and your current job? activity as measures of learning for digital assets. We also like to measure “attainment” – how well did people meetBERSIN & ASSOCIATES, LLC specifically defined client (business user or customer) objectives? For 6114 LA SALLE AVENUE SUITE 417 example, a senior software engineer at Nuance University relates his use OAKLAND, CA 94611 of the on-demand digital library to delivering a product to customers. (510) 654-8500 INFO@BERSIN.COM 9 An “indicator” is an individual’s assessment of a metric, rather than the metric itself. WWW.BERSIN.COM Measuring On-Demand Learning Janet Clarey | Page 12 THIS MATERIAL IS LICENSED TO SAFARI BOOKS ONLINE FOR DISTRIBUTION ONLY. BERSIN & ASSOCIATES © 2011 THIS MATERIAL IS LICENSED TO SAFARI BOOKS ONLINE FOR DISTRIBUTION ONLY. BERSIN & ASSOCIATES © 2011
  13. 13. Research Bulletin | 2011 Research Bulletin | 2011 BERSIN & ASSOCIATES BERSIN & ASSOCIATES “Without this [resource], we would probably not be able to ship Dragon Naturally Speaking 11.5 on time. I relied heavily on our digital content library to bring me up to speed. We keep referring to this project as having a tight timeframe, but our online library as made it possible for me to actually work fast enough in a new OS, in a new sublanguage, to meet the deadlines.” Specifically adapted for informal learning, our Impact Measurement Framework also includes contribution, feedback and activity as measures of learning for digital assets. One organization utilizing such measures is Unisys.Figure 6: Informal Learning Measures Contribution   Feedback   Activity   How  well  are  people   What  is  the   How  many  people   commenting  on  and   volume,  frequency   are  accessing  the   rating  comment   and  depth  of   system  and  at  what   in  a  fair  way?   contribution  made?   frequency?         Is  feedback  being   What  percent  of  experts   What  groups  have   read  and  used  by   are  contributing   greatest  access  and   authors?   in  what  areas?   which  are  not         using  it  yet?   Is  feedback  being   How  regular  and     ranked  and  analyzed   current  is  the   How  widely     by  the  organization?   response  to  comments   distributed  is     and  questions?   the  activity  in  the   Are  authors  and  others     program?   using  feedback   How  rapidly  are     well?   people  keeping   How  well  are  higher-­‐     up  with  changing   level  people  BERSIN & ASSOCIATES, LLC Can  we  “rate”   needs?   contributing?   6114 LA SALLE AVENUE authors  well?   SUITE 417 OAKLAND, CA 94611 (510) 654-8500 INFO@BERSIN.COM Source: Bersin & Associates, 2011. WWW.BERSIN.COM Measuring On-Demand Learning Janet Clarey | Page 13 THIS MATERIAL IS LICENSED TO SAFARI BOOKS ONLINE FOR DISTRIBUTION ONLY. BERSIN & ASSOCIATES © 2011 THIS MATERIAL IS LICENSED TO SAFARI BOOKS ONLINE FOR DISTRIBUTION ONLY. BERSIN & ASSOCIATES © 2011
  14. 14. Research Bulletin | 2011 Research Bulletin | 2011 BERSIN & ASSOCIATES BERSIN & ASSOCIATES Case in Point: Unisys University A worldwide information technology company, Unisys provides KEY POINT a portfolio of IT services, software and technology that solves While Unisys critical problems for clients. Due to the nature of its business, it is very important that employees keep up with leading-edge University finds technology. To support that business need, Unisys University that measuring uses, among other learning solutions, a variety of on-demand the value of on- learning resources from several providers, including the Safari demand learning Books Online library. resources is hard to Unisys University finds that measuring the value of on-demand pin down, despite learning resources is hard to pin down. How do you determine this challenge, the whether or not a person who read a book has improved his / her company has come performance? Despite this challenge, Unisys has come up with a up with a variety of variety of measures. measures. Unisys University uses surveys to flesh out any changes in employee productivity or time-savings after it utilizes an on- demand learning resource over some period of time. Although time periods vary, the survey is often scheduled to be completed prior to a program review meeting. A high and low metric is used (based on users’ salaries) and a range is determined (based on employees’ responses to the survey and usage figures). It is not a perfect measure, due to the difference in roles and responsibilities of the employees participating – and this is why the company uses a range and other methods of measurement. The company also poses open-ended questions to employees, such as, “Give us an example of a time that someone else noticed your improved skills.” Offering opened-ended questions helps Unisys to learn of problems about which it might not have otherwise been aware. For example, the company found that itsBERSIN & ASSOCIATES, LLC global audience had different technology issues based on how 6114 LA SALLE AVENUE and where audience members worked. SUITE 417 OAKLAND, CA 94611 The measurement process is time-consuming, so Unisys (510) 654-8500 University works with its solution providers to administer and INFO@BERSIN.COM WWW.BERSIN.COM Measuring On-Demand Learning Janet Clarey | Page 14 THIS MATERIAL IS LICENSED TO SAFARI BOOKS ONLINE FOR DISTRIBUTION ONLY. BERSIN & ASSOCIATES © 2011 THIS MATERIAL IS LICENSED TO SAFARI BOOKS ONLINE FOR DISTRIBUTION ONLY. BERSIN & ASSOCIATES © 2011
  15. 15. Research Bulletin | 2011 Research Bulletin | 2011 BERSIN & ASSOCIATES BERSIN & ASSOCIATES Case in Point: Unisys University (cont’d) help them analyze the surveys and responses. By having vendors do as much of the high-level analysis as possible, the University gains broader comparisons and benchmarks, since vendor analysis often includes data from other clients which have used similar surveys. Unisys University also solicits ideas from its employees about KEY POINT how the company can make the use of on-demand learning By having vendors resources more valuable. Other than determining the improvements in productivity associated with an online learning do as much of the resource, the most important feedback taken from the surveys high-level survey is the employee input about what would cause them to use the analysis as possible, resource more. (See Figure 7.) the University The company has learned some lessons about managing content gains broader and solution providers. For years, Unisys University maintained comparisons and an enterprise agreement with a vendor and realized it was benchmarks, since only using about 15 percent of the licenses for which it was vendor analysis paying. While the provider had a good digital library of content, often includes data it was not a good fit for the employees. A huge library with from other clients “something for everyone” is not really effective if it does not which have used reflect targeted skills. similar surveys. Unisys University is currently offering enterprise access to some on-demand resources, such as its online digital library of books. The ability to download chapters or portions of books was especially important to Unisys employees, and saves a great deal of time over traditional e-learning content. Where on-demand resources fit within the total learning environment varies over time, partially due to changes in corporate initiatives and partially due to contract requirements. Employees have a single access point to all learning contentBERSIN & ASSOCIATES, LLC through an internal portal. However, some content resides 6114 LA SALLE AVENUE within an individual’s learning folder and other content is SUITE 417 accessed by linking directly to a solution provider’s website. Easy OAKLAND, CA 94611 access to the resources is critical. (510) 654-8500 INFO@BERSIN.COM WWW.BERSIN.COM Measuring On-Demand Learning Janet Clarey | Page 15 THIS MATERIAL IS LICENSED TO SAFARI BOOKS ONLINE FOR DISTRIBUTION ONLY. BERSIN & ASSOCIATES © 2011 THIS MATERIAL IS LICENSED TO SAFARI BOOKS ONLINE FOR DISTRIBUTION ONLY. BERSIN & ASSOCIATES © 2011
  16. 16. Research Bulletin | 2011 Research Bulletin | 2011 BERSIN & ASSOCIATES BERSIN & ASSOCIATES Case in Point: Unisys University (cont’d) Unisys University does tie most of its on-demand resources to some formal learning programs. While it does very little instructor-led training (ILT), the University does use its on- demand learning resources as pre-work for both face-to-face training and self-paced e-learning. It has built L&D guides that provide a listing of all of the learning assets that support an employee’s role (e.g., project management), a corporate initiative or program, or a skills set / certification. These guides include e-learning, white papers, books, abstracts and other learning resources. The director of operations and finance at Unisys University BEST PRACTICE offered some advice to organizations that want to provide employees with on-demand learning resources, such as Unisys University books online. has built L&D guides that If you’re just getting started, don’t buy an enterprise license. provide a listing of It’s going to take some time to get some traction and you will be spending more than you need during that time. all of the learning assets that support an employee’s company. It is easy to find a provider with a lot of content, role, a corporate but not so easy to find the right one. initiative or program, or a skills more or less licenses? Do you need multiple partners? set / certification. that doesn’t change about the specific content needs of any organization is that those needs are always changing. Using multiple providers has worked out very well for Unisys University. The content meets its needs and keeps solutionBERSIN & ASSOCIATES, LLC providers on their toes. 6114 LA SALLE AVENUE SUITE 417 OAKLAND, CA 94611 (510) 654-8500 INFO@BERSIN.COM WWW.BERSIN.COM Measuring On-Demand Learning Janet Clarey | Page 16 THIS MATERIAL IS LICENSED TO SAFARI BOOKS ONLINE FOR DISTRIBUTION ONLY. BERSIN & ASSOCIATES © 2011 THIS MATERIAL IS LICENSED TO SAFARI BOOKS ONLINE FOR DISTRIBUTION ONLY. BERSIN & ASSOCIATES © 2011
  17. 17. Research Bulletin | 2011 Research Bulletin | 2011 BERSIN & ASSOCIATES BERSIN & ASSOCIATESFigure 7: An Example of Unisys University’s Survey Questions 1. Would you recommend the content to a colleague? 2. How would you rate the training resources? 3. Please explain why you rated the training resource as you did. 4. What was the main reason why you accessed the (xxx) training resources identified above? 5. Is the learning content relevant to skills you use on your job? 6. Primarily, what skill were you seeking to develop? 7. How often does your job require you to use this skill? 8. Have you had the opportunity to use what you learned on your job? 9. As a result of the training, how much of a skill increase would you estimate you experienced? Do you feel that your productivity has increased as a result? If so, how many hours would you estimate you 10. save on a weekly basis? 11. Is the skill you were seeking to improve required for you to advance within the company? 12. If you did not have access to the (xxx) resources listed above, what alternatives would you have considered? 13. How much do you estimate the alternate option would have cost? 14. How often so you use the resources listed above? Source: Unisys University, 2011. Other Thoughts on Measurement We can take some guidance from Johnson, Trabelis & Tin, who wrote, The Successful Virtual Library: Partnership and Collaboration, in establishing a guide for workplace learning. They said, “… an emphasis on technology and information without a focus on human components (such as service and interaction) are the most important determinants of the success of the virtual libraryBERSIN & ASSOCIATES, LLC encouraging people to work together, develop their 6114 LA SALLE AVENUE skills, and form strong and trusting relationships.10” SUITE 417 OAKLAND, CA 94611 (510) 654-8500 10 Source: “Library support for online learners: E-resources, e-services and human factors,” INFO@BERSIN.COM In Theory and Practice of Online Learning, Johnson, K., Trabelsi, H. & Tin, T., Editors Terry Anderson & Faith Elloumi, Athabasca University, 2004. WWW.BERSIN.COM Measuring On-Demand Learning Janet Clarey | Page 17 THIS MATERIAL IS LICENSED TO SAFARI BOOKS ONLINE FOR DISTRIBUTION ONLY. BERSIN & ASSOCIATES © 2011 THIS MATERIAL IS LICENSED TO SAFARI BOOKS ONLINE FOR DISTRIBUTION ONLY. BERSIN & ASSOCIATES © 2011
  18. 18. Research Bulletin | 2011 Research Bulletin | 2011 BERSIN & ASSOCIATES BERSIN & ASSOCIATES Whether or not there is focus on human components is an elusive measurement, but it is important to keep in mind. As these researchers continued, “This approach considers the library as an active partner of the learning community, helping learners to become ‘information literates’ by integrating information literacy skills into the curriculum. For example, the library can help learners to evaluate critically the authority and authenticity of the resources they find, and to enhance their critical thinking skills. The library can also offer support to learners, and can mentor their work by offering one-to-one communication and interaction, and by achieving a deeper level of understanding of what learners need.11” KEY POINT Replace the word “library” with “L&D organization” and the message is the same, which starts to introduce the concept of “metalearning,” or An early decision being aware of and taking control of one’s own learning, an important was made to aspect of learning today at Nuance University. establish Nuance University as an online university, Case in Point: Nuance University – Metalearning and with all content Trending Analysis and resources Nuance is a speech and language expert technology firm. delivered through The company has 6,000 employees on all continents, is a self-directed headquartered in Burlington, Massachusetts and has achieved online platform 40 percent growth over six years. Nuance University focuses on alignment. that allows for a cost-effective In 2007, there was no established technology learning program solution and at Nuance. The company was growing very quickly at this point consistent deliveryBERSIN & ASSOCIATES, LLC (via many acquisitions), leading to an environment with a across the globe. AVENUE 6114 LA SALLE tremendous amount of change, intense timeframes for delivery SUITE 417 OAKLAND, CA 94611 (510) 654-8500 11 Source: “Library support for online learners: E-resources, e-services and human factors,” INFO@BERSIN.COM In Theory and Practice of Online Learning, Johnson, K., Trabelsi, H. & Tin, T., Editors Terry WWW.BERSIN.COM Anderson & Faith Elloumi, Athabasca University, 2004. Measuring On-Demand Learning Janet Clarey | Page 18 THIS MATERIAL IS LICENSED TO SAFARI BOOKS ONLINE FOR DISTRIBUTION ONLY. BERSIN & ASSOCIATES © 2011 THIS MATERIAL IS LICENSED TO SAFARI BOOKS ONLINE FOR DISTRIBUTION ONLY. BERSIN & ASSOCIATES © 2011
  19. 19. Research Bulletin | 2011 Research Bulletin | 2011 BERSIN & ASSOCIATES BERSIN & ASSOCIATES Case in Point: Nuance University (cont’d) and complex, varied technical environments. Based on these factors (and others), the employees had started to demand a development program. The pace of its engineers’ work is fast, requiring engineers to get up to speed on a variety of technologies quickly. The company’s technology is very complex may be required to work on multiple projects with vast technology differences and they are expected to adapt. An early decision was made to establish Nuance University as BEST PRACTICE an online university, with all content and resources delivered through a self-directed online platform that allows for a cost- To determine the effective solution and consistent delivery across the globe. best way to handle Outside and / or instructor-led training is only available as an the technical exception if business needs require something unavailable learning needs of through Nuance University. A small development team also the organization, creates some proprietary resources, as business needs require. as a first step To determine the best way to handle the technical learning Nuance University needs of the organization, as a first step Nuance University put together a put together a steering committee to address the business steering committee needs and develop a strategy. While working with the to address the steering committee, as well as evaluating available content business needs and and resources, it was clear early on that an informal learning develop a strategy. resource would be of greater value than a formal learning program. The feeling was that formal learning tools required too much effort to transfer the knowledge from the learning experience to the job. An informal tool would allow the “heads-down” engineers to easily transfer from their work to a learning resource and back – allowing for better integration and workflow. To test the chosen tool, the company initially started with aBERSIN & ASSOCIATES, LLC small pilot group made up of 200 R&D people. The pilot group 6114 LA SALLE AVENUE SUITE 417 was given licenses to an on-demand digital content library. Not OAKLAND, CA 94611 long after that, word of mouth about the value of the resources (510) 654-8500 “incited a riot.” Clearly, the employees had found a valuable INFO@BERSIN.COM WWW.BERSIN.COM Measuring On-Demand Learning Janet Clarey | Page 19 THIS MATERIAL IS LICENSED TO SAFARI BOOKS ONLINE FOR DISTRIBUTION ONLY. BERSIN & ASSOCIATES © 2011 THIS MATERIAL IS LICENSED TO SAFARI BOOKS ONLINE FOR DISTRIBUTION ONLY. BERSIN & ASSOCIATES © 2011
  20. 20. Research Bulletin | 2011 Research Bulletin | 2011 BERSIN & ASSOCIATES BERSIN & ASSOCIATES Case in Point: Nuance University (cont’d) resource. The company’s technology resources included Safari Books Online and the IEEE Digital Library, a niche provider of speech science and technology. Neither are the cheapest, but each are a great fit for the company and the employees loved it. Nuance University uses its digital content library as both a business tool and a learning tool. In fact, as a department, it considers itself more of an internal consultant, rather than a traditional learning and development team. The measuring mechanism is unique in that it includes the KEY POINT measurement of business trends, as well as a very direct effort to align learning content and resource usage with business As the metrics programs and priorities. For example, the company looks at have matured, trending around things like Android and other “hits” during Nuance University a particular development process. This helps the University to has found that manage the resources and understand how they are used within quantitative the business. measures of As the metrics have matured, Nuance University has found that the on-demand quantitative measures (like usage) of the on-demand digital digital library do library do not hold much value for their executives on their not hold much own, so it has focused heavily on including qualitative analysis value for their in reporting to provide the much-needed context around executives, so it learning numbers. Much of this effort is achieved through the has focused heavily use of a learner survey that is sent to employees some period of on including time after they have accessed a particular learning or reference resource. The focus is almost entirely on the open-ended qualitative analysis comments in the survey, as well as on the employee comments in reporting to that can be worked into actual success (and other) stories. The provide the much- University reaches out to those employees and builds out a needed context series of internal case studies that can be used for a variety of around learning purposes (e.g., marketing and business-case development).BERSIN & ASSOCIATES, LLC numbers. SALLE AVENUE 6114 LA Looking ahead, Nuance University is thinking about how to SUITE 417 further entrench its resources across the organization, to: OAKLAND, CA 94611 (510) 654-8500 INFO@BERSIN.COM WWW.BERSIN.COM Measuring On-Demand Learning Janet Clarey | Page 20 THIS MATERIAL IS LICENSED TO SAFARI BOOKS ONLINE FOR DISTRIBUTION ONLY. BERSIN & ASSOCIATES © 2011 THIS MATERIAL IS LICENSED TO SAFARI BOOKS ONLINE FOR DISTRIBUTION ONLY. BERSIN & ASSOCIATES © 2011
  21. 21. Research Bulletin | 2011 Research Bulletin | 2011 BERSIN & ASSOCIATES BERSIN & ASSOCIATES Case in Point: Nuance University (cont’d) Be available in all “natural places” in which employees look Help employees to collaborate on projects around these Help learners personalize their learning experience, so that Allow learners to add their own knowledge and insights to the body of knowledge accessed by other employees. Safari Books Online: A Unique Approach to Driving Value from a Digital Library One provider that is worth a look is Safari Books Online, an on-demand KEY POINT digital library provider which offers an integrated source for more The breadth and than 15,850 technology, digital media and business titles from various publishers, including more than 1,200 video training tutorials. depth of Safari’s content goes well Safari Books Online, an organization headquartered in Sebastopol, beyond books. Its California, is a joint venture between O’Reilly Media, Inc. and Pearson content in video Education. The company’s offerings include access to the collections form is extensive of O’Reilly Media, Addison-Wesley, Prentice Hall, Peachpit Press, New Riders, Sams, Que, Exam Cram, Cisco Press, Adobe Press and Adobe and is one of the Developer Library, as well as hundreds of additional titles from reasons why this John Wiley & Sons, Apress, Manning, Microsoft Press, McGraw Hill, company stands Oracle Press, IBM Press, Syngress, Morgan Kaufmann, Focal Press, out in the on- Total Training, VTC, Class on Demand and Skill-Pill. Safari supplies demand digital on-demand digital libraries to each of the companies profiled in this library space. research bulletin. The company’s depth and breadth in offerings is illustrated byBERSIN & ASSOCIATES, LLC the number of content categories, including digital curricula for 6114 LA SALLE AVENUE SUITE 417 OAKLAND, CA 94611 (510) 654-8500 resources in video and text formats. (See Figure 8.) INFO@BERSIN.COM WWW.BERSIN.COM Measuring On-Demand Learning Janet Clarey | Page 21 THIS MATERIAL IS LICENSED TO SAFARI BOOKS ONLINE FOR DISTRIBUTION ONLY. BERSIN & ASSOCIATES © 2011 THIS MATERIAL IS LICENSED TO SAFARI BOOKS ONLINE FOR DISTRIBUTION ONLY. BERSIN & ASSOCIATES © 2011
  22. 22. Research Bulletin | 2011 Research Bulletin | 2011 BERSIN & ASSOCIATES BERSIN & ASSOCIATES Safari Books Online offer products for three different needs: For the enterprise, its products are Safari Business Library (all business books), Safari Bookshelf (up to 10 books per month), Safari Tech Library (technology books only) and Safari Premium Library (all content, including books and videos). The breadth and depth of Safari’s content goes well beyond books. Its content in video form is extensive and is one of the reasons why this company stands out in the on-demand digital library space. Another reason Safari stands out is the availability of interactive tools on its platform. With these interactive tools, corporate users are able to manage, organize and internalize the content. This is especially important today because organizations increasingly need fingertip knowledge. No business can avoid the need to operate better, smarter and faster.Figure 8: Safari Books Online Products and SubscriptionsBERSIN & ASSOCIATES, LLC 6114 LA SALLE AVENUE SUITE 417 OAKLAND, CA 94611 (510) 654-8500 INFO@BERSIN.COM WWW.BERSIN.COM Source: Safari Books Online, 2011. Measuring On-Demand Learning Janet Clarey | Page 22 THIS MATERIAL IS LICENSED TO SAFARI BOOKS ONLINE FOR DISTRIBUTION ONLY. BERSIN & ASSOCIATES © 2011 THIS MATERIAL IS LICENSED TO SAFARI BOOKS ONLINE FOR DISTRIBUTION ONLY. BERSIN & ASSOCIATES © 2011
  23. 23. Research Bulletin | 2011 Research Bulletin | 2011 BERSIN & ASSOCIATES BERSIN & ASSOCIATES Safari Books Online (Release 6.0) has many capabilities including: KEY POINT Through the availability of interactive tools on Safari’s platform, corporate users are able to manage, organize and internalize the content. Some of the more unique capabilities that differentiate Safari Books Online include: 12 -type capabilities, such as mashups13 and “share this” social 12 “Web 2.0” refers to a second generation of web-based communities and hosted services (such as social-networking sites, wikis, folksonomies, weblogs / blogs, social bookmarking, podcasts, RSS feeds, social software, web application programming interfaces / APIs, and online web services), that aim to facilitate creativity, collaboration and sharing betweenBERSIN & ASSOCIATES, LLC users. Although the term suggests a new version of the World Wide Web, it does not refer to an update to any technical specifications, but to changes in the ways in which software 6114 LA SALLE AVENUE developers and end-users use the web. SUITE 417 OAKLAND, CA 94611 13 In technology, a “mashup” is a web application that combines data from more than one (510) 654-8500 source into a single integrated tool, thereby creating a new and distinct web service that was not originally provided by either source. Content used in mashups is typically sourced from INFO@BERSIN.COM WWW.BERSIN.COM include web feeds (e.g., RSS / really simple syndication), web services and screen-scraping. Measuring On-Demand Learning Janet Clarey | Page 23 THIS MATERIAL IS LICENSED TO SAFARI BOOKS ONLINE FOR DISTRIBUTION ONLY. BERSIN & ASSOCIATES © 2011 THIS MATERIAL IS LICENSED TO SAFARI BOOKS ONLINE FOR DISTRIBUTION ONLY. BERSIN & ASSOCIATES © 2011
  24. 24. Research Bulletin | 2011 Research Bulletin | 2011 BERSIN & ASSOCIATES BERSIN & ASSOCIATES Conclusion Measuring the approaches to learning that fall under the umbrella of KEY POINT the term “informal learning” creates a new measurement challenge Supporting on- for organizations. Many companies (such as those featured in this bulletin) are taking an innovative approach to measuring the value of demand learning informal learning. is a function of modern learning Supporting on-demand learning is a function of modern learning organizations that organizations that are concerned with going to the point at which performance happens and making resources available to learners. This are concerned has created a new role for L&D – that of the “enabler and facilitator” with going to the of all types of performance support, including the on-demand learning point at which content from digital libraries. Specifically adapted for informal learning, performance our Impact Measurement Model14 takes into account contribution, happens and feedback and activity as measures of learning for digital assets. making resources The increasing speed at which the human capital needs of the available to organization are evolving, as well as the changing needs and learners. expectations for learning on the part of employees, customers and partners suggests on-demand digital libraries can fill an important need. As a result, many companies are taking an innovative approach to measuring the value of informal learning. If your organization is ready to begin the transition to a modern learning organization and is looking to leverage informal learning (and mobile learning) in the enterprise, a simple starting point is providing on-demand learning support through a digital library. The use of on-BERSIN & ASSOCIATES, LLC demand digital content libraries is growing and we expect that trend 6114 LA SALLE AVENUE to continue. SUITE 417 OAKLAND, CA 94611 (510) 654-8500 INFO@BERSIN.COM 14 For more information, The Impact Measurement Framework: An Update for the WWW.BERSIN.COM Measurement of Informal Learning, Bersin & Associates / Josh Bersin, May 3, 2011. Measuring On-Demand Learning Janet Clarey | Page 24 THIS MATERIAL IS LICENSED TO SAFARI BOOKS ONLINE FOR DISTRIBUTION ONLY. BERSIN & ASSOCIATES © 2011 THIS MATERIAL IS LICENSED TO SAFARI BOOKS ONLINE FOR DISTRIBUTION ONLY. BERSIN & ASSOCIATES © 2011

×