Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Lean & Agile Learning: The CGS Approach
to Delivering Effective Learning Solutions
Elizabeth Woodward
Learning Strategist ...
2
Contents
L&A Learning: The CGS Agile Approach to Delivering Effective Learning Solutions................1
Abstract.........
3
Abstract
Developing and delivering effective learning solutions is more challenging than ever. Enterprises
are continuou...
4
market, 74% to increase productivity, and 81% to improve their adaptability and to better
manage change.1
In 2004, CGS f...
5
Figure 1. Scrum
Using Scrum we have an ordered dynamic list of all of the work that might be needed for the
learning sol...
6
development in order to improve efficiency of the learning development and delivery process
wherever possible.6
The prin...
7
unnecessary time spent by learners. The CGS team collaborates with clients to determine the
most effective approach for ...
8
Efficient and Effective Instructional Design
Instructional design is the practice of creating "instructional experiences...
9
Figure 2. ADDIE and Waterfall Software Development
Today, there are many variations of the ADDIE model that take an iter...
10
 Digital experience delivery
 Digital convergence
 Data and analytics
 Mobile technology
 Cloud computing
 And ev...
11
and health monitoring sensors is expected to reach $400 million in 2017, up from less than $2
million in 2012.15
Sensor...
12
quickly implement foundational capabilities that increased the team’s capacity to deliver more
sophisticated e-learning...
13
 2010: Development of a virtual data center (a highly private area that is not accessible to
outside people) to host a...
14
Figure 5. Increased Value Delivery over Time
Agile Value Delivery
Figure 6 demonstrates taking an agile approach to the...
15
increments weekly for longer projects. This means that every week, clients are able to interact
with a tangible deliver...
16
Figure 8. Traditional Approach Delivered Iteratively
Embracing Change with Adaptability and Flexibility
One of the key ...
17
Flexibly Meeting Client Schedule Changes
As most learning and development professionals will attest, no matter how care...
18
Daily Planning
Daily, the CGS development team meets briefly to discuss the work completed the previous
day, the work t...
19
Commitment to Quality
L&A Learning builds quality into the learning solution development and delivery process.
Definiti...
20
Expense/Revenue as a Percentage
The expense/revenue percentage is helpful in quantifying the success of L&A Learning. A...
21
Helping Clients Measure Success
CGS’s position is that CGS is a partner in helping clients to develop and implement lea...
22
Example 1. Measuring Success
As CGS was working with a client to create a proposal to deliver at a global virtual confe...
23
Summary
CGS delivers measurable impact and value by taking a lean and agile approach to creating
learning solutions tha...
24
Author Biographies
Elizabeth Woodward
Elizabeth Woodward is a Lean and Agile Consultant and Learning
Strategist with Co...
Upcoming SlideShare
Loading in …5
×

Lean and Agile Learning: The CGS Approach

568 views

Published on

How CGS collaborates with clients and stakeholders to deliver great learning solutions.

Published in: Education
  • Be the first to comment

  • Be the first to like this

Lean and Agile Learning: The CGS Approach

  1. 1. Lean & Agile Learning: The CGS Approach to Delivering Effective Learning Solutions Elizabeth Woodward Learning Strategist and Agile Transformation Consultant ewoodward@cgsinc.ca Micah White Director of Research and Development pwhite@cgsinc.ca August 17, 2015
  2. 2. 2 Contents L&A Learning: The CGS Agile Approach to Delivering Effective Learning Solutions................1 Abstract..............................................................................................................................................3 Agile and Lean Learning Innovation................................................................................................3 A Foundation of Scrum.................................................................................................................4 Lean Principles..............................................................................................................................5 Efficient and Effective Instructional Design.................................................................................8 Harnessing the Rapid Evolution of Technology .............................................................................9 CGS History of Continuous Innovation ...............................................................................................11 Speed to Value................................................................................................................................13 Agile Value Delivery.......................................................................................................................14 Iterative Delivery .........................................................................................................................14 ADDIE Delivery...............................................................................................................................15 Embracing Change with Adaptability and Flexibility ....................................................................16 Agile Adaptation of Requirements and Design.........................................................................16 Flexibly Meeting Client Schedule Changes ..............................................................................17 Demonstrating Progress of Development and Delivery...........................................................17 Commitment to Quality...................................................................................................................19 Measuring Success ........................................................................................................................19 Measure #1: Cost Effectiveness of Solutions...........................................................................19 Measure #2: Delighting Clients..................................................................................................20 Helping Clients Measure Success.............................................................................................21 Top-Down Approach to Measuring Learning Solutions...........................................................21 Summary .........................................................................................................................................23 Author Biographies .........................................................................................................................24 Elizabeth Woodward...................................................................................................................24 Micah White.................................................................................................................................24
  3. 3. 3 Abstract Developing and delivering effective learning solutions is more challenging than ever. Enterprises are continuously asking their learning organizations to do more with less. Total corporate spending on training and the amount spent per employee has been declining since 2010. Chief Learning Officers (CLOs) and their learning organizations are arguably experiencing the most dramatic pace of technology change and convergence of technologies ever experienced. Additionally, emerging technologies, business competition, and learner expectations are increasing the pressure on learning organizations to reduce time-to-value—to deliver value more quickly than ever. Learning organizations that can harness the rapid evolution of technology to improve learning outcomes and demonstrate business impact are more likely to receive investment in the future. Computer Generated Solutions (CGS) has been a leading provider of learning solutions for over 30 years. They have managed to thrive, innovate, and help clients to deliver high-value learning solutions in a climate where many learning organizations have closed their doors. CGS’s bold success is the result of passion for continuously innovating within the field of enablement. Their approach to creating learning solutions is deeply rooted in a collaborative culture of delighting clients using a strong foundation of lean and agile principles and values. Agile and Lean Learning Innovation Learning professionals who have developed a proposal or designed, developed, or delivered a learning solution using traditional methods are likely to have experienced challenging projects throughout their careers where:  Learning solutions may have taken much longer to deliver than originally anticipated.  Cost estimates were exceeded just to meet minimum expectations.  Delivery was delayed—solutions took much longer than they had originally expected.  The true progress (percent completion) was difficult to assess.  Rich media versions of learning solutions didn’t meet the learning organization’s or the learner’s expectations.  The enterprise may have questioned the business impact of the investment in learning.  Budget cuts may have left the learning organization to deliver less effective learning solutions or meet fewer of the learner’s needs. Similar problems have plagued software development organizations. In 2011, a group of software development thought leaders met to discuss the essential principles and values that lead to successfully delivering software solutions. Through those discussions, they created the Agile Manifesto—a set of principles and values that describe agile software development methods. Today, more than 52% of organizations are using agile approaches to manage the majority of their software development projects. Scrum and Scrum variants are the most popular agile methodologies in use at 73%. 75% of companies using agile are doing so to accelerate time to
  4. 4. 4 market, 74% to increase productivity, and 81% to improve their adaptability and to better manage change.1 In 2004, CGS first began embracing elements of Scrum, the leading agile software development framework today as a result of analyzing their own successes and challenges with delivering learning solutions. The Practical Guide to Distributed Scrum shares some of CGS’s early experiences with implementing Scrum in a complex environment with distributed team members and clients. CGS combined core agile principles, Scrum, best practices for instructional design, and lean principles championed by Tom and Mary Poppendieck to create a lightweight approach to continuous improvement.2 3 CGS continues to evolve the approach and has most recently extended it with DevOps concepts of further optimizing the whole. CGS refers to their lean and agile approach as L&A Learning. A Foundation of Scrum Although CGS can and will accommodate most learning development methods and deliverable approaches requested by clients, the CGS team will internally leverage agile methods to the extent possible to help ensure the success of projects. To provide a project heartbeat, transparency, reduced time-to-value, and greater alignment with client expectations over the course of a project, L&A Learning uses Scrum. Scrum is a framework within which people can address complex adaptive problems, while productively and creatively delivering products of the highest possible value. Scrum employs an iterative, highly transparent, and adaptive approach to delivery. (See Figure 1.) 1 “8th Annual State of Agile Survey,” Version One, 2014, http://www.versionone.com/pdf/2013-state-of-agile- survey.pdf 2 “Manifesto for Agile Software Development,” http://agilemanifesto.org/ 3 See Implementing Lean Software Development by Tom and Mary Poppendieck
  5. 5. 5 Figure 1. Scrum Using Scrum we have an ordered dynamic list of all of the work that might be needed for the learning solution in a Product Backlog. Every Sprint--one week for CGS--a CGS cross-functional team (instructional designers, artists, writers, editors, and others) discusses the highest ranked work for the week based on client insight and agrees to the approach for delivering the week’s learning deliverables. The team commits to the work for the week and that becomes the Sprint Backlog. The team identifies the primary focus of the week and that becomes the Sprint Goal. The team consists of team members with all of the skills required to complete the learning project, including instructional designers, artists, writers, editors, and others. Each day, the team participates in a 15-minute meeting to discuss what they accomplished the previous day, what will be accomplished that day, and any issues. At the end of the week, the team engages in Sprint Reviews—demonstrations of what was accomplished that week and how it might impact the next week’s work. Note that the resulting learning deliverable is expected to be DONE—in some usable condition that clients can touch, interact with, and evaluate. Next, the team conducts a Sprint Retrospective where they discuss how they can improve in the following week. This is one of the critical events prescribed by Scrum. The Sprint Retrospective is a chance for the team to inspect how the last Sprint went with regards to people, relationships, process, and tools and to create a plan for implementing improvements in the next week.4 By inspecting and adapting, CGS weaves continuous improvements into the fabric of their delivery system. For more information on Scrum see the Scrum Guide on Scrum.org.5 Lean Principles L&A Learning uses Scrum as a foundational framework for managing projects in an adaptable, value-driven way. Additionally, L&A Learning embraces the seven principles of lean software 4 https://www.scrum.org/Portals/0/Documents/Scrum%20Guides/2013/Scrum-Guide.pdf 5 https://www.scrum.org/Portals/0/Documents/Scrum%20Guides/2013/Scrum-Guide.pdf#zoom=100
  6. 6. 6 development in order to improve efficiency of the learning development and delivery process wherever possible.6 The principles of lean software development are:  Eliminate waste. Anything that does not add value in some way to the learning solutions delivered to clients is considered a waste. By ruthlessly targeting and eliminating waste in processes, CGS is able to focus more intensely on learning capabilities that matter—advancing instructional design, leveraging the latest technologies to improve the learner’s experience, etc.  Build in quality. Processes are in place to prevent defects and issues from entering the process.  Amplify knowledge. Planning is useful, but both the client and the team create critical knowledge by experiencing the solution—even if the experience involves a fraction of the ultimate deliverable.  Decide as late as possible. Development and delivery of learning solutions is almost always associated with some uncertainty. By taking an options-based approach and delaying decisions until they can be made based on facts learned through the development process rather than uncertain assumptions and predictions, teams are able to deliver better results.  Deliver as quickly as possible. The sooner a solution (or part of a solution) is delivered with quality, the sooner clients are able to provide feedback that the team can incorporate into the work for the next week.  Empower the team. People are not resources.7 An empowered team attracts bright, creative people who collaborate to deliver wildly successful deliverables.  Optimize the whole. Optimizing the whole means observing the end-to-end value stream that CGS clients experience. From initial contact through to delivery of the solution and subsequent follow-up, CGS aggressively analyzes opportunities to improve the client experience. Fundamental to lean software development is the notion of actively seeking out many opportunities to eliminate waste—activities that deliver no value. Inventory: Unfinished goods. For learning organizations, unfinished goods are partially completed or undelivered courseware. Lean principles of eliminating waste promote delivery of smaller modules, learning modules, or other enablement artifacts as they become available during the process of completing larger courses that remain undelivered for a longer period. Overproduction: Producing more than required. Enabling learners to efficiently gain knowledge, skills, and behaviors is essential to delivering value to the enterprise. Overproduction, delivering more materials than required, can result in additional and 6 Mary Poppendieck and Tom Poppendieck, Lean Software Development: An Agile Toolkit (Boston, MA: Pearson Education, Inc., 2003). 7 Mary Poppendieck and Tom Poppendieck, The Lean Mindset (Upper Saddle River, NJ: Pearson Education, Inc., 2014).
  7. 7. 7 unnecessary time spent by learners. The CGS team collaborates with clients to determine the most effective approach for a given learning and business objective. Extra processing: Additional, unnecessary steps in processes. An example of extra processing is individually producing separate learning deliverables for mobile devices rather than using responsive design to create one design for multiple devices. This waste increases costs and opportunities for errors. Transportation: Shipping goods from place to place. Applying lean principles to learning solutions, transportation involves both movement between systems and movement between team members and handoffs. Waiting: Delays between process steps. Examples of waiting include delays in delivery of working software to customers, bureaucracy—decisions or approvals that create delays, blockers (inefficient tools, inputs, or reviews), and waiting for feedback. Daily planning meetings (also known as Daily Scrums) help to address blockers and inefficiencies. During the 15-minute meeting each day, the team answers three questions: 1. What work did you complete yesterday? 2. What will you complete today? 3. Do you have any impediments that prevent you from meeting commitments? Impediments/blockers are taken seriously by the team, and the team works together with management, clients, and others as needed to address the blocker within hours. Motion: Movement within the process. An example of movement within a process would be the case where an organization has no central repository and poor version control systems. In this case, team members and clients email files from person to person, which can result in confusion over which version is the most recent and where the most recent files are. Additionally, time is wasted as team members and clients seek and access files in different locations. Task switching is another example of movement within processes. Even brief mental blocks created by shifting between tasks can cost as much as 40% of someone's productive time.8 Defects: Flaws in deliverables. Costs of defects increase over time as the development team moves farther away from the point of injection. A team member may have to set up the development and testing environment from scratch and re-familiarize with the materials before troubleshooting and correcting the problem. Additionally, moving from one module or learning deliverable to another without completing all testing can result in defects accruing and “technical debt” increasing. By building quality into the process, L&A Learning not only improves quality but also reduces the costs of developing and delivering learning solutions. 8 David Meyer, as cited in Iain Gillespie, “Multitasking Makes You Stupid, Studies Find,” The Sydney Morning Herald, May 27, 2015, http://www.smh.com.au/digital-life/digital-life-news/multitasking-makes-you-stupid- studies-find-20150520-gh5ouq.html
  8. 8. 8 Efficient and Effective Instructional Design Instructional design is the practice of creating "instructional experiences which make the acquisition of knowledge and skill more efficient, effective, and appealing."9 Quality instructional design is essential to developing and delivering impactful learning experiences. Constructivist theory is used to create learning experiences that approximate real-world interactions and enable learners to construct their own knowledge.10 In the 1990s, CGS was an early adopter of remote lab environments to enable learners to gain hands-on experience with complex software as part of their learning experience. In the 2000s, as instructional designers recognized the need to integrate e-learning into the creation of learning objects and curricula, CGS implemented one of the earliest learning management systems, applying IEEE Learning Technology Standards Committee (LTSC) Learning Object Metadata (LOM) and ADL Sharable Content Object Reference Model (SCORM) standards. Throughout the 2000s, as cognitive load theory advanced, the CGS team embraced the new knowledge to create more effective learning experiences.11 CGS pursues new knowledge related to learning and digital experiences to continuously improve and evolve instructional design expertise. That said, CGS quickly recognized that ADDIE (a process for creating instructional materials), and most popular ISD models that are variants of ADDIE, suffered from the same challenges as waterfall software development.12 ADDIE provides a guideline for developing training and performance support tools in five linear phases: Analysis, Design, Development, Implementation, and Evaluation. The intent was for each phase to be completed before moving on to the next phase. This approach in the world of software development is referred to as a “waterfall” approach. 9 M. David Merrill, Leston Drake, Mark J. Lacy, Jean Pratt, and the ID2 Research Group. “Reclaiming instructional design.” Educational Technology 36, no. 5 (1996): 5–7. http://mdavidmerrill.com/Papers/Reclaiming.PDF 10 Robert A. Reiser. "A History of Instructional Design and Technology: Part II: A History of Instructional Design." Educational Technology Research and Development 49, no. 2 (2001): 57–67. https://files.nyu.edu/jpd247/public/2251/readings/Reiser_2001_History_of_ID.pdf 11 Fred Paas, Alexander Renkl, and John Sweller. "Cognitive Load Theory: Instructional Implications of the Interaction between Information Structures and Cognitive Architecture". Instructional Science 32, (2004): 1–8. 12 George M. Piskurich. Rapid Instructional Design: Learning ID Fast and Right (San Francisco, CA: John Wiley & Sons, 2006).
  9. 9. 9 Figure 2. ADDIE and Waterfall Software Development Today, there are many variations of the ADDIE model that take an iterative approach to development. However, most focus on iterative delivery and evaluation of phases as well as rapid prototyping after significant analysis. Figure 3. More Recent ADDIE Model The Scrum and agile approach to iterative development differs in that small slices of full- featured capabilities are delivered throughout the project in weekly time-boxed iterations. Early in the process, clients and learners are able to experience and provide feedback on the learning deliverables. In small bites, the CGS team works through full cycles to demonstrate functional courseware with reduced risk and more rapid time-to-value. Harnessing the Rapid Evolution of Technology CLOs and their learning organizations are arguably experiencing the most dramatic pace of technology change and convergence of technologies ever experienced. The ability to quickly analyze the trends and rapidly harness key evolving technologies to deliver more effective and efficient learning experiences is essential. Six trends today demonstrate the impact of changing technology on both the development and delivery of learning programs:
  10. 10. 10  Digital experience delivery  Digital convergence  Data and analytics  Mobile technology  Cloud computing  And even 3D printing The continuous improvement aspect of agile development means that harnessing the rapid evolution of technology is integral to the learning development process. Every Sprint, teams look for opportunities to innovate in both learning deliverables and learning development. Digital experience delivery makes or breaks firms13 today. Consumers, in general, have come to expect a high-quality digital experience from service and product providers across industries—retail, health care, entertainment, and others. Part of that expectation is set by the increasing budgets and influence of polished marketing organizations on all aspects of the digital presence of the companies they support. Consumer expectations spread to other online experiences, including online learning experiences, where learners expect high-quality graphic design, interactivity, fast-paced video, and integrated audio elements. Learning organizations that can take advantage of available tooling and rapidly evolve new techniques to create a more robust digital experience are able to deliver more engaging learning experiences. The digital convergence of physical and digital worlds is also changing how learners expect to experience learning. In many situations, learning is embedded in software solutions, rather than being part of a completely separate website or service. Learners can learn how to perform tasks without leaving the context of the application. The evolving field of wearables (miniature electronic devices that are worn) provides an ability to analyze the environment and provide learning based on contextual data.14 Additionally, digital convergence is slowly changing how learning is developed. As an example, a subject matter expert (SME) can record the execution of tasks using wearable technology, which can be used for development of new learning objects. Data and analytics are enabling the delivery of more personalized online experiences. Today’s software applications blend data from systems of record (HR, CRM, consumer databases, data warehouses, and others) with systems of engagement (email, collaboration, social, and learning systems) to provide more opportunities than ever before to create custom, tailored, and targeted learning solutions that efficiently meet an individual’s learning objectives. Additionally, the proliferation of sensors and sensor data is enabling a personalized learning experience that would have been impossible in the recent past. Global revenue for environment 13 “Forrester Top Technology Trends 2014 and Beyond,” Forbes, http://www.forbes.com/pictures/eimh45mkdi/2- digital-experience-delivery-makes-or-breaks-firms/ 14 “ELI 7 Things You Should Know About… Wearable Technology,” Educause, https://net.educause.edu/ir/library/pdf/ELI7102.pdf
  11. 11. 11 and health monitoring sensors is expected to reach $400 million in 2017, up from less than $2 million in 2012.15 Sensors delivered in mobile devices have continued to increase in diversity beyond accelerometers, electronic compasses, gyroscopes, ambient and proximity light sensors, humidity and temperature sensors, blood glucose monitors, and many others. Sensor data increasingly provides details of the learner’s environment, physical state, and needs that can be used to deliver increasingly personalized learning experiences. Mobile technology is growing as a learning platform. The proliferation of mobile creates many new opportunities for learning experiences that are no longer tied to a classroom or desktop machine. However, mobile also increases the complexity of learning development and delivery. Not only are there now several types of devices to accommodate, but there are variations across the devices. This creates significant challenges for learning developers—particularly those delivering to Android devices that vary according to device manufacturer, telecom provider, version of OS, configuration of the device, and applications included by default with the device. Furthermore, Bring Your Own Device (BYOD) policies are increasing the variations in devices that access the learning organization’s online resources. Gartner reports that 38% of companies expect to stop providing workplace devices to staff by 2016 and that 70% of mobile professionals will conduct their work on personal smart devices by 2018.16 Cloud computing is changing the speed at which learning organizations can adopt new technology and deliver solutions to their clients. Cloud computing provides access to a shared pool of configurable resources, such as networks, servers, stories, applications, and services, that can be rapidly provisioned and released with minimal IT management.17 Learning organizations can quickly provision sophisticated, tailored software environments and application environments to their learners for hands-on software experiences. Additionally, learning organizations can take advantage of rapidly evolving learning development technology by accessing learning development software solutions in the cloud. It is no longer necessary to spend days acquiring hardware, software, and software licenses to take advantage of the latest technology. CGS History of Continuous Innovation CGS is always moving forward with new technologies and innovations. One way they ensure this development is by reserving a certain amount of capacity each Sprint for targeted innovation development. For example, CGS steadily made progress over time on development of tooling to enable more efficient delivery of custom courses. The tooling exceeded the capability of other e-learning development tools on the market at the time and enabled CGS to 15 Jérémie Bouchaud, “Environment and Health Sensors in Smartphones Set for Mighty Growth,” IHS Technology, August 21, 2013, https://technology.ihs.com/446281/environment-and-health-sensors-in-smartphones-set-for- mighty-growth 16 David A. Willis, “Bring Your Own Device: The Facts and the Future,” Gartner., April 11, 2013, https://www.gartner.com/doc/2422315 17 “Final Version of NIST Cloud Computing Definition Published,” NIST, October 25, 2011, http://www.nist.gov/itl/csd/cloud-102511.cfm
  12. 12. 12 quickly implement foundational capabilities that increased the team’s capacity to deliver more sophisticated e-learning capabilities. As a result of this innovation, CGS received the 2009 Brandon Hall Excellence in Learning Technology bronze award for Best Advance in Technology for Rapid Authoring.18 Figure 4. CGS Innovations Over the years, CGS has often implemented futuristic learning solutions while others considered possibly implementing the technologies in the future.  2004: Flash-based training featuring an advanced case-based scenario role play to support the client’s sales methodology.  2005: Video-based “right and wrong” vignettes that incorporated tropes and visual cues of a television program, including feature interviews with executive stakeholders.  2006: Illustrated characters serving as financial advisors—each with a defined backstory (relevant to the scenario) and investment positions.  2007: Video-based “learning nuggets” that incorporated television-styled advertising with longer form infomercials to stimulate learners, generate interest, and issue a call to action to take the training module.  2008: Rapid authoring with a focus on inclusiveness: advanced closed captions, keyboard access, and support for enterprise compliance as a competitive differentiator.  2009: iPlus authoring tool received Brandon Hall Award for Best Advance in Rapid Authoring, beating out major vendors like Sun Microsystems and Adobe Systems Inc. 18 “CGS e-Learning Software Wins 2009 Brandon Hall Excellence in Learning Technology Award,” prweb, May 13, 2010, http://www.prweb.com/releases/2010/05/prweb3986674.htm
  13. 13. 13  2010: Development of a virtual data center (a highly private area that is not accessible to outside people) to host a fully realized simulation of security and assessment practices of the installation. The simulation was fully delivered in Unity3D.  2011: Blackberry sales playbook developed in Adobe AIR middleware for Blackberry Playbook, iPad, and Android tablets.  2012: Telecom service course with game mechanics (drive persona home, achievements, open-world).  2013: QBE Town, a fully realized 3D city with 15 locations (including a sports stadium, restaurant, and bowling alley), which uses dual outputs: Flash stage3D technology and/or WebGL; interior and exterior rendering engine; and points, scoring, and hidden objects, all part of simulation mechanics.  2014: Fully responsive device agnostic portal for sales transformation. Course completes increased from 100,000 to over 1 million. Sales enablement training used in the field on iPhone, tablets, and legacy desktop computers. Learning nuggets used to promote product sections. As Yuan and Woodman’s research has indicated, one of the biggest deterrents to innovation within organizations is the fact that employees are unaware that innovation is expected of them.19 Where innovation is encouraged and made part of the fabric of operations, teams innovate. CGS expects innovation from team members, and, as a result of that expectation, CGS delivers award-winning innovations. Speed to Value Learning organizations that can deliver learning more quickly have a significant competitive edge over others who follow more traditional delivery cycles. Delivering learning as smaller objects can:  Enable an enterprise to begin training their employees more quickly, so that those employees can begin reaping the benefits of knowledge or skill improvement more quickly  Help customers to have greater success with products more quickly, leading to improved customer satisfaction and future sales opportunities  Provide potential customers with the education that they need to make positive purchasing decisions Delivering more quickly allows the enterprise to begin accruing business value earlier in the process. Figure 5 demonstrates value to the enterprise when all modules are delivered at the end of delivery of an entire curriculum—in this case, four modules. Notice that development takes place from t1 to t4 of the entire curriculum, and it is not until t4 that the learners and enterprise begin extracting value from the investment. 19 Feirong Yuan and Richard W. Woodman. “Innovative Behavior in the Workplace: The Role of Performance and Image Outcome Expectations.” Academy of Management Journal 53, (2010): 323–342.
  14. 14. 14 Figure 5. Increased Value Delivery over Time Agile Value Delivery Figure 6 demonstrates taking an agile approach to the curriculum. In this example, the development team works with stakeholders to deliver Module 1 to learners. While Module 2 is in development, learners and the enterprise are gaining value from Module 1. While Module 3 is in development, learners and the enterprise are gaining value from both Module 1 and Module 2. Figure 6. Traditional Value Delivery As an example, one CGS client had a mission of “delivering a world class curriculum for software development methods.” By working with the client, CGS was able to develop smaller learning modules (learning objects) that enabled the organization to begin delivering education to improve their software testing capabilities much earlier than the client had originally anticipated. Given the business costs of finding defects later in the software development process, the client and their independent metrics consultant determined the approach resulted in millions of U.S. dollars of increased productivity. Additionally, the client eventually received a $3.4 million grant from the U.S. Department of Labor in collaboration with the State of Arizona to deliver the courses as part of the university curricula to raise the skills of Arizona's technology workers.20 Iterative Delivery One of the key aspects of Scrum is that the agile team delivers a useable increment of the solution to the client at the end of each fixed block of time. Typically, CGS teams deliver 20 “USDOL Announces $3.4 Million Grant to Increase Skills of Arizona IT Workers,” National Association of State Workforce Agencies, October 20, 2005, http://www.naswa.org/assets/utilities/serve.cfm?path=azitgrant05.htm
  15. 15. 15 increments weekly for longer projects. This means that every week, clients are able to interact with a tangible deliverable that demonstrates progress on their project to date. Tangible weekly deliverables provide CGS clients and the CGS team with transparency regarding progress of the project. For the modules described in Figure 7, CGS takes the approach of completing all aspects of development required for each delivery cycle or Sprint. For example, the CGS team engages in design, writing graphics, coding, QA, and any other required tasks for a smaller module before continuing to the next module. Courses are developed as a sequence of smaller chunks. A module consisting of three lessons that could be completed within a Sprint is decomposed into the three lessons, and the team makes an effort to complete each lesson before moving on to the next. At the end of the week, the team is able to demonstrate the three lessons to the client and obtain feedback that is used to inspect, adapt, and improve delivery in the following week. Figure 7. Approach to Delivering Rapid Value ADDIE Delivery For organizations that follow an ADDIE model or prefer/require that a highly detailed design be delivered first for an entire course, followed by a lightweight alpha process for the entire course, then beta, and then gold release, CGS takes an approach as shown in Figure 8.
  16. 16. 16 Figure 8. Traditional Approach Delivered Iteratively Embracing Change with Adaptability and Flexibility One of the key principles of agile methods is that “the best architectures, requirements, and designs emerge throughout a project.”21 Agile Adaptation of Requirements and Design Traditional Big Design Up Front (BDUF) approaches advocate spending significant time in requirements and design, expecting that enough time will enable the development team to get every detail exactly right in the earliest stages of a project. Some development organizations require a sign-off at each of these early stages to “lock in” requirements and designs during a paper or text stage before clients and the development team have an opportunity to experience those elements in a richer e-learning format. Unfortunately, clients in this situation have difficulty providing meaningful feedback until they (and their learners) have seen graphics options or treatments, tried an e-learning interactive that an instructional designer described in text, or have seen the elements laid out by a graphic artist or programmer in the actual deliverable format. As the client experiences education deliverables, they have more ideas and feedback regarding their preferences and what they believe might be most effective for their enterprise and their learners. As discussed in the previous section, delivering early and frequently enables learning organizations to inspect and adapt quickly. 21 “Principles behind the Agile Manifesto,” http://agilemanifesto.org/principles.html “In preparing for battle I have always found that plans are useless, but planning is indispensable.” ― Dwight D. Eisenhower
  17. 17. 17 Flexibly Meeting Client Schedule Changes As most learning and development professionals will attest, no matter how carefully a team plans, plans will change throughout a project. Plans may change for any number of reasons. Some examples include:  While working with their SMEs, clients may realize that certain content is more or less important than they previously understood. They may want to emphasize or devote more effort to one learning objective rather than another.  Clients may determine that they or their SMEs are not available as previously expected. SMEs may have to address issues with their own customers or unexpectedly meet with executives.  Urgent issues or changes in priority may adversely impact the client and their project schedule. Rarely does a project go exactly according to the initial schedule, and plans, schedules, priorities, and staffing plans change quickly. In most cases, strictly adhering to the schedule and initial requirements would result in unsatisfied customers. The challenge many organizations have is to be able to adapt to continuously changing requirements at scale—welcoming changes throughout the process in a sustainable way that enables the team to delight their customers. Demonstrating Progress of Development and Delivery Although plans change over time, planning—and collaboration—is essential. Planning takes place in four ways. Initially, as part of the Statement of Work, CGS works with clients to establish a high-level working plan. That plan evolves over the course of the project to accommodate clients and new learning that takes place throughout development. Release Planning During Release Planning, the CGS team (facilitated by the CGS client advocate) works with the client to determine an approach to delivery that reduces risk early in the development process, provides hands-on experience with all aspects of the learning solution early in the process, and enables the client to begin receiving value from completed deliverables as quickly as possible. This information helps the team to determine the best ways to decompose work over the course of the project and prioritize deliverables for optimal delivery of business value to the client. Sprint Planning In Scrum, every iteration begins with a Sprint planning meeting. At this meeting, the client advocate works with the learning development team to determine what work should be addressed during the next week. This brief weekly planning process is most often a checkpoint—an opportunity to discuss any changes in client plans, priorities, learning, and availability and what the team and client learned during the previous week regarding preferences and implementation details.
  18. 18. 18 Daily Planning Daily, the CGS development team meets briefly to discuss the work completed the previous day, the work that is scheduled to be completed that day, and any blockers or issues they have encountered. CGS has client advocates who help to ensure that projects progress forward and that any issues are addressed. (In a pure Scrum environment, the Scrum Master would work to address any potential blockers for the team and the client’s deliverable.) As an example of how the process works, if the development team determines that they have not yet received content from a client SME, the team makes the client advocate aware right away or—worst case— during the Daily Scrum. Within a short time, the client advocate follows up with the client to obtain the necessary content. In this way, daily planning helps assure clients that development is progressing at a high-performing pace. Demonstrated Results Every Sprint results in a demonstrable deliverable that meets the discussed criteria of acceptance and the definition of “done” for that deliverable at that point in the project. As an example, for a client who prefers the ADDIE model and expects to see an entire course delivered in phases, the team may deliver a version of the alpha within a Sprint. For a client that embraces agile methods, CGS may deliver a completed smaller module at the end of each Sprint. Clients can expect to view—and often interact with—the deliverable at the end of every Sprint. In the Scrum framework, demonstrations are identified at the Sprint Review meeting. For most CGS projects, clients can expect to see significant progress at least every week. Demonstrated results are the key to clients’ confidence that learning solutions are truly progressing according to expectations. Figure 9. Sprint Reviews
  19. 19. 19 Commitment to Quality L&A Learning builds quality into the learning solution development and delivery process. Definition of done. The definition of done is a way of gaining shared understanding regarding what “done” will look like for deliverables in a given Sprint. The definition applies to everything delivered during that Sprint. Editing, QA, SCORM packaging, and other items may be part of the definition of done checklist. Acceptance criteria. Conditions of acceptance relate to a specific deliverable. It’s possible that an early deliverable may not include a completed, sophisticated simulation, for example. What is important is that the team and clients have a common understanding of “done” for the iteration. Weekly reviews. Weekly demonstrations of progress with the client help to build quality into the deliverable, as well. Traceability. Traceability throughout a project is critical for building quality into the learning solution. Client feedback is traceable to a specific version of the deliverable and the CGS team responds with an action taken for each item of feedback. Measuring Success CGS measures success in two ways: cost effectiveness of solutions and delighting clients. Measure #1: Cost Effectiveness of Solutions By innovating for cost-effective development of high-quality, effective learning solutions, CGS enables clients to deliver more value to their learners for less investment. Additionally, finding ways to increase productivity of common, low-end tasks enables the team to invest more effort in leveraging new technologies and methods for delivery of more effective learning. Figure 10. Metric #1—Cost Effectiveness of Solutions Revenue per Headcount The revenue per headcount target is a fixed number that represents the team’s throughput. This metric is only used in conjunction with customer satisfaction. The team is only successful if an increase in revenue per headcount occurs in parallel with delighting customers.
  20. 20. 20 Expense/Revenue as a Percentage The expense/revenue percentage is helpful in quantifying the success of L&A Learning. Again, this metric is only viewed in parallel with client satisfaction. Measure #2: Delighting Clients Of course, cost-effectiveness is pointless if clients are not delighted by the results of their learning solutions. As a result, CGS’s primary metric for determining success is the degree to which deliverables, interactions with the team, and processes delight CGS clients. Figure 11. Metric #2—Delighting Clients CGS measures success in delighting clients in four ways. Net Promoter Score After each project completion, clients are asked to answer one simple question: “On a score of 0–10 with 10 being extremely likely and 0 being not likely at all, how likely is it that you would recommend CGS Enterprise Learning to a friend or colleague?” Growth in the Number of E-learning Projects Delighted clients are more likely to return for additional help with learning initiatives and projects in the future. CGS measures success on opportunities to subsequently support clients. Number of New Clients Delighted clients are more likely to recommend CGS to other departments within their enterprise or to their peers in other enterprises. Client References CGS views client references in the form of case studies, white papers, co-presentations, video testimonials, and others as being a significant testimony to CGS’s ability to delight clients. Additionally, client references co-created with CGS can often be used by clients to build awareness of their excellent learning programs.
  21. 21. 21 Helping Clients Measure Success CGS’s position is that CGS is a partner in helping clients to develop and implement learning solutions that drive a higher level of business value—increased revenue, reduced operating costs, improved quality, etc. Throughout the project, CGS works with clients to determine what their business goals are and to identify ways that they can support clients in measuring the impact of their solutions. Most learning organizations are familiar with Kirkpatrick’s Four Levels of Evaluation22 as a way of determining the effectiveness of learning solutions:  Level 1. Reaction  Level 2. Learning  Level 3. Behavior  Level 4. Results It can be more challenging to measure results at higher levels beyond the reaction “smile sheet,” but learning organizations that are able to demonstrate value to the business are more likely to receive future investment from the business. The corporate training market experienced a low point between 2009–2010 and total spending on training, as well as amount spent per employee, have been declining since then.23 Top-Down Approach to Measuring Learning Solutions During the initial proposal process, CGS takes a top-down approach to collaboratively deriving the measures for success. CGS asks important questions about the expected value of the learning program and the problem(s) that the program is intended to solve. Understanding the business impact enables CGS to propose a solution that helps clients to evaluate and demonstrate the value of their learning deliverables in terms of business value.  Level 4 Results. What is the expected business impact of this learning solution (e.g., employee retention, increased sales, greater customer satisfaction . . .)? This is often a longer-term metric that will require input, baseline, and tracking from outside of the learning organization.  Level 3 Behavior. What are you expecting for people to do differently on the job or as a customer in order to have the desired business impact?  Level 2 Learning. What do you need to teach your audience in order for them to behave differently? What is their current state and what should it be after they use your learning solution?  Level 1 Reaction. How will you know if they had a positive reaction to this deliverable and what might help you to improve for next time? 22 Donald L. Kirkpatrick and James D. Kirkpatrick. Implementing the Four Levels (Oakland, CA: Berrett-Koehler Publishers, 2008). 23 “E-Learning Market Trends & Forecast 2014 – 2016 Report,” March 2014, http://www.docebo.com/landing/contactform/elearning-market-trends-and-forecast-2014-2016-docebo- report.pdf
  22. 22. 22 Example 1. Measuring Success As CGS was working with a client to create a proposal to deliver at a global virtual conference, the team discussed the different types of functionality that would be required. The client focused on hands-on virtual software labs, conference kiosks/pedestals, virtual help desk, branding of the site, and much more. Eventually, the CGS facilitator turned the discussion to measures of success and asked, “What would it take to make you a hero?” The client paused thoughtfully and then responded, “We would have to have 1,500 people in specified parts of the organization and certain regions attend . . .” With that information, CGS proposed a solution that included active recruitment and follow-up with conference attendees to ensure that the client achieved the target demographics with their learning program. While participation is a low-level metric, in this client’s case the specifics of attendance were critical to achieving their goal of global awareness. Example #2: Tracking the Impact of Software Development Testing Modules CGS provided a learning solution for an organization that was interested in reducing the number of defects that were being identified by customers using their software. The expectation was that the defects should have been caught earlier during development and that by teaching software development team members new testing methods, the learning organization would help to reduce software maintenance costs. The following is a high-level summary of the measurement plan established for the learning solution. Note that only Level 1 and Level 2 metrics were solely in the purview of the client’s learning organization. The team collaborated with the QA organization and Support organization for Level 3 and Level 4 metrics. The successful measurement of this program was key to the client learning organization receiving increased funding the following year. The Level 4 metric was to reduce maintenance costs related to defect “escapes”—defects detected by the customer. A second supporting metric was the severity of the defects that customers were reporting to the Support organization. A “Severity 1” defect that shut down the client was expected to be more costly in terms of the fix, the urgency of the fix, and impact to other efforts in progress and long-term satisfaction of the client. The Level 3 metric involved QA metrics—defects being detected earlier in their processes. The client expected for the courseware to enable detection of potential defects earlier in the process. By detecting defects earlier, fewer defects were expected to reach customers. The Level 2 metric involved a survey sent out to teams a month after they had taken the learning modules. They were asked to indicate the degree to which they were using what they had learned in the course. Learners were also asked to identify the amount of time that they believed the methods had saved them in identifying problems before the software was passed on to production. Although the second metric is a “soft” metric, it was useful as an indicator of the value that learners believed the modules to be delivering. The Level 1 metric was gathered from a survey delivered immediately after modules were taken. Learners were asked about the value of the courseware, whether they had learned something new, whether they believed they would apply what they learned on the job, graphic design, quality of writing, quality of the hands-on exercises, etc.
  23. 23. 23 Summary CGS delivers measurable impact and value by taking a lean and agile approach to creating learning solutions that delight clients. The L&A Learning approach has resulted in a remarkable 95% rate of repeat business. Clients are better able to quickly seize opportunities related to market and technology changes and to deliver high-quality solutions that deliver the targeted results. The L&A Learning approach embraces continuous improvement. For additional information, contact CGS Enterprise Learning. CGS Enterprise Learning is an award-winning provider of custom professional development solutions with more than 30 years of experience helping successful businesses to continuously innovate and drive change through learning programs with measurable business impact.
  24. 24. 24 Author Biographies Elizabeth Woodward Elizabeth Woodward is a Lean and Agile Consultant and Learning Strategist with Computer Generated Solutions (CGS). She has more than 20 years of experience improving efficiency and effectiveness of learning organizations and learning deliverables. Elizabeth has held roles as diverse as senior IT operations administrator, programmer, architect, project manager, transformation consultant, instructional designer, and learning strategist for some of the world’s largest enterprises and small businesses. As a member of the IBM CHQ Agile Transformation Team from 2006–2010, Elizabeth was responsible for enabling more than 65,000 development team members, managers, and executives worldwide during the early years of IBM’s agile adoption. She has coached diverse agile organizations and teams, including large-scale product development, small-scale project, and systems/embedded. Elizabeth has been an active member of the agile community since 2007. She co-authored the book A Practical Guide to Distributed Scrum endorsed by Ken Schwaber (co-creator of the #1 agile approach worldwide) and has presented numerous conference sessions and courses in the United States, Canada, United Kingdom, China, Sweden, and Switzerland. Elizabeth is also an avid inventor and innovator with more than 20 patents filed and a special interest in agile methods for research, innovation, and learning programs. More information: www.linkedin.com/in/elizabethwoodward/ and twitter.com/liz_woodward Micah White Micah White is the Director of Research & Development for Computer Generated Solutions (CGS), a large privately held technology company with over 5,500 employees and with offices in four continents. Micah is a career graphics programmer who has been working in the e-learning industry since 1999. Holding a bachelor’s degree in Mathematics and Computing Science, Micah works with the world’s best IT and telecom companies to deliver training that can be used on all devices. Micah has a deep understanding of intelligent design, which he has used again and again in the e-learning field, which stems, in part, from his early association with Paul Allen’s Wired World. Micah’s work was nominated for a Canadian Gemini Award in 2003, and he won a Brandon Hall Award for Best Advance in Rapid Authoring in 2009. Since 2009, Micah has been working in the multi-device world, delivering responsive e- learning solutions to smartphones, tablets, and desktops.

×