Downloaded 18 times
Uploaded byHT2 Labs
Investigating Performance: Design & Outcomes with xAPI | LSCon 2017
The document discusses the importance of understanding data, its collection, and analysis in order to evaluate educational and training outcomes. It emphasizes the significance of data stewardship, quality, and actionable insights while proposing strategies for effective data management. The conversation highlights the need for clarity in content strategy and outcomes to drive meaningful changes in learning environments.
More Related Content
Similar to Investigating Performance: Design & Outcomes with xAPI | LSCon 2017
Investigating Performance: Design & Outcomes with xAPI | LSCon 2017
- 2.
- 3. • WE cansee the completions and failures, what is the next question we should ask?
- 4. • Why? Whydid people fail? Why?
- 6. Know Your Data Whatmakes the data come into being? How hard is it to get the data I want/need? What’s my Data Supply Chain?
- 7. Data Supply Chain ActivityProviders Systems architecture Data management, ownership, and accessibility Potential Data Integration concerns What’s “good enough”?
- 8.
- 9. Content Strategy Content What activitiescan provide measurable data? Analysis What data can we (and should we) act on? Technical What data can we get from our systems?
- 10. xAPI Format Janet presentedContent Strategy
- 11. xAPI Format Janet presentedContent Strategy
- 12. xAPI Format Janet presentedContent Strategy
- 13. xAPI Format Janet presentedContent Strategy With: Lscon 2017 Place: Orlando Result: 80% of room still awake
- 14.
- 15.
- 18. • xAPI –Collecting more than just completions and test scores
- 20. Who are thePlayers?
- 21.
- 22. Strategic Goals Informational Behavioral (micro) habits,individual professional development Behavioral (macro) large scale process or procedural change across BUs or across time
- 23.
- 24. Data Stewardship What LiesBeneath? You’ll have to teach someone. When everybody’s in charge, nobody’s in charge.
- 25. What about dataquality? Australian Bureau of Standards https://www.nss.gov.au/dataquality/aboutqualityframework.jsp
- 26. We’re ultimately interestedin what is: observable measurable actionable
- 28.
- 33.
- 34. So what isQualitative Data?
- 37. How do wecollect qualitative data?
- 43. • How canwe get this information earlier in the process?
- 48.
- 49. • Image ofpaper prototype
- 51.
- 52.
- 55.
- 58.
- 60.
- 61.
- 62.
- 64. From Data toAnalysis Available Relevant
- 65. What data amI likely to get?
- 66.
- 67. Explore Your Data Clean SenseCheck Spread Centers
- 68.
- 69. “He uses statistics asa drunken man uses lampposts – for support rather than for illumination” Andrew Lang
- 70. What Do YouWant to Know? Participation Performance
- 71. Participation Related • Howare people using resources available? • How much time are they spending? • What elements/resources are proving useful? • What aspects of participation affect results? • What aspects of participation affect completions?
- 72. Performance Related • Arewe teaching the right things? • How does course performance correspond to job performance? • Did X have the desired effect on Y?
- 73. The game is on… Timefor Analysis But we have a problem…
- 74. When it comes toanalysis, your brain may be your worst enemy
- 75. Illustration by GeraldFisher From Pscychology of Intelligence Analysis - Heuer
- 76. Analysis of CompetingHypotheses Richards J. Heuer
- 77.
- 78. • WE havetalked about a single course but think about if we had modules
- 79. • A seriesof modules we can see how people use the content and what the end result from that use looks like.
- 82. • Show finalresult
- 84.
- 86. • Wrap upstory
- 87. Data by itselfis useless. Data is only useful if you apply it. Todd Park Chief Technology Officer White House
- 89.
Editor's Notes
- #6 WE are going to talk about data in the intitial design process. And the use of data to assess the effectiveness of the learning intervention and provide for future design improvements.
- #11 Statement API looks a bit like this
- #17 We are looking for confirmation that the learner go what they needed to know. And we can ask the question
- #18 We need to figure out what change means to us. If we were just looking to see how many people accessed something numbers are enough. When it comes to really seeing what behavior change our interventaion has made we need to figure out how we measure the change. So we need to figure out
- #19 WE are looking at Collecting more than just completions and test scores, but if we are collecting completions and test scores, what does it actually mean. If I score a 90% on a quiz how does that affect my job performance? What path did I take to get to the quiz, does that affect my score. Why did I miss the last 10%? Was there a job role that scored better than others? Why?
- #20 Data strategy
- #21 L&D Owners L&D Customers Data Owners Data Customers
- #22 Org/Institution (Strategic goals, Impact) L&D (Courses, Strategies)
- #24 Examples: the data isn’t where it should be interoperability over time Change management
- #28 Did the intervention actually work?
- #29 Quantitative is what you nomrally would think of as data. It is numbers, something that is qunatifiable
- #30 Here we see a typical quantitative view of data. It is numbers, how many people looked at a page, how many people looked at a piece of content. Quantitative data is pure numbers.
- #31 How do we collect the data? Really two main ways to collect the datafrom our content. We can use Google analytic sor the experience API. How many peopl ehave heard of the experience api?
- #32 However, numbers mean almost nothing. In a lot of cases the numbers will simply show us a quantity.
- #33 We need to set some context to the numbers. Numbers are numbers, there is no meaning. By adding context to them we give them meaning.
- #34 Qualitative data is data the captures feelings and obsrervations that are not quantifiable.
- #35 So what exactly is qualitative data?
- #36 Qualitative looks at how you feel about something. In what we are doing it is watching someone use interface, watching their reaction to a piece of content or a design. It is basically trying to see how people are reacting to what is in front of them. This used a lot in interface design. Like I did in the opening slide.
- #38 So now that we know what qualitative data is, how do we collect it?
- #39 The second one we will look at is collecting feedback form the deployed material. We can seek out feedback through interviews where you sit down and interview the person who has performed the task. User interviews can be conducted as one on one interviews or group interviews.
- #40 Surveys can be sent out to the users who performed the test to gather how the felt about the design that was delivered to them. There are many ways to set this up with ratings or short answer questions. You can also use surveys as step one and then conduct an interview to gain more perspective from the user.
- #41 Be careful many studies have found that directly interviewing and providing someone with a survey, they are not always truthful. How many times have you been asked about an interface and said its OK. Or if the person who designed is the one talking to you, you might tell them it is fine when really there were portions that confused or aggravated you.
- #42 Actually observing users is that best way to get real information. Although it can be too late (and should be) to observe users after deployment for feedback. Really the best and most effective time to observe users is using a portotyping phase. Lets take a deep dive into prototyping and how it is effective for qualitative analysis.
- #43 Investigating performance not just about improving our learners, but also ourselves
- #45 IN most cases we start to look at qualitative data during interface design. Does it work, is there too much travel? Is there confusion on the part of the end user?
- #46 The last thing we want is to require our users to have a gps to get through the material. If we have to provide detailed instructions to use an intervention we have already failed
- #47 Because this is the last thing you want to see is a frustrated user after you have deployed some content. Collecting some qualitative data early in the design process from stakeholders and end users can help avoid this.
- #48 A key to gathering this observation data is to start collecting it as early as possible. The best way to do that is to build prototypes.
- #49 Next we build a physical prototype. These are great for watching users as we are going to get a first look at how they interact with a design. These are great first pieces of data that will affect how the design goes forward.
- #50 One method that I love in the early design phases is a paper prototype. What I love about the paper prototype is the instant feedback and how quick and easy it is to make. The other great thinga bout this is you can hand the user or stakeholder a marker and some post it notes to help generate the designs.
- #51 Here we see a more advanced version of the paper prototype in action. Notice the mdeia changing as the user “clicks” through the interface. It is a great bit of ffedback to see how the interact with the screen. You can for excessive movement, is there confusion about where to go next, things like that. After tweaking and coming up with a few designs you can move on to the next step
- #52 Once we narrow down the designs using physical prototypes we can build some wire frame prototypes, we will get to an example of this in just a few minutes.
- #53 Digital prototypes are also good because you can expand the reach of a prototype. Using Skype or a GoTo session with a webcam you can watch the person as they work through the content. Watching the face is important because it can tell you a lot about what they are thinking. After gathering data from this wider group of users…
- #54 The prototypes are incomplete, keep it simple as long as possible. Changes are much less expensive to make to prototypes than they are to released designs.
- #55 Prototype example: Palm pilot
- #56 Finally we can collect the data using a refined digital prototype. This is a prototype that is very close to the final product. So lets dive into this a little further….
- #57 Use it yourself, sometimes just getting something in a physical state allows you to see the initial problems. You can gather data from yourself, don’t discount your feelings as you sue the first digital prototypes.
- #58 Workshops
- #59 User Testing
- #60 WE can add context by using qualitative data. Why did so many people click on a certain element? Why was a certain path followed through a set of modules. So we are making a correlation between the design and the data that we are collecting.
- #61 The numbers can only tell us so much – qual = more work, but also more nuanced info
- #65 Available? What’s relevant? Where do they intersect? Let’s start with what’s available…
- #66 Kinds data you can get that can be built into evidence, information, tell stories…
- #77 Heuer – Pscy of Intel Analysis / In sci we called this Multiple Working Hypotheses // Keeps you from being “married” to one idea – pushes you to dig deeper and ask better questions than you would otherwise = more complete, more nuanced understanding. Simple example – Email nudges- did they help? What if we were talking about sales training?
- #78 Is the data diagnostic?
- #79 So lets expand this out to thing about a series of modules that we might have in say an LMS or just out on a website somewhere. Looking at the data that appears..
- #80 We could start to see what people are doing start to see learning paths emerge. Now when we compare this to the behavior change data, we can start to make suggestions to future users. If a certain path shows to provide a good path to expertise for a job type. We can suggest that to people looking to move into that job type.
- #81 How did the expert get here
- #82 A software company was looking to certify users. Now in software, a written test really provides almost no value to actual usage of the software. They need to see the user actually working in the software. So one company started generating xAPI statements from their software. The user clicked a button, interacted with a panel, created something.
- #83 That alloweed them to have user run through a process and then they could compare that to what an expert did on the same process. The number of interactions and the amount of time it took to perform them were tracked. If the user was 40% langer than the expert, a self paced module was assigned to them for the topic. They could go take the module, which can be tracked with xAPI and then retake the test. They could then see if the user could perform the task after the intervention. I know that this was a very quick high level overview of designing for data, let me leave you with one thought
- #84 What suggestions can I make to future users as to the path they take.
- #86 This is an iterative process, did the intervention work? If not why not? What can we do to make it better? Adjustments can be made based on the results being seen from the data generated, did the interventions actually work. This feedback loop is something that we are missing in a lot of current cases.