Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Formative Evaluation for Educational Product Development


Published on

Presented to the Badges for Lifelong Learning grantees of the DML Competition, February 18, 2013.

Published in: Education
  • Great presentation! I'm planning a trial run of our education product in a school. Need to decide what we're going to measure and how to measure it. This is one of the few good resources that I've been able to find.
    Are you sure you want to  Yes  No
    Your message goes here
  • Thanks Matt! Often formative evaluation needs to be done in conjunction with subject matter experts to make sure the content is accurately represented. I think that's the major difference is that content and production need to be closely aligned & coexist. :)
    Are you sure you want to  Yes  No
    Your message goes here
  • Love it! wish I could have been there in person.
    Are you sure you want to  Yes  No
    Your message goes here
  • Great presentation, visually and content-wise. This seems to combine agile and human centered methods of development and design. I've been using a similar method on technology projects for a while, but am finding it's a different animal doing it in education where so many preconceptions exist. :)
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

Formative Evaluation for Educational Product Development

  1. 1. Formative Evaluation for Educational Product Development *Or, how to tell if this thing you made is worth its salt Vanessa Gennarelli Learning Lead, P2PU Master’s Candidate, Harvard University @mozzadrella
  2. 2. Why Should I Test My Product? Can folks use it? Evaluate usability and nomenclature. Do users like it? Measure appeal and engagement. Does this audience get it? Test comprehension and recall.
  3. 3. What is Formative Evaluation? Testing an educational product with users during development. The results inform the direction of the product. Process can be conducted at any time. Speedy stuff--time is of the essence.
  4. 4. Who Uses Formative Evaluation?
  5. 5. Who Uses Formative Evaluation?
  6. 6. Who Uses Formative Evaluation?
  7. 7. That’s peachy, Vanessa.
  8. 8. That’s peachy, Vanessa. Can I do this myself? Do I have time for formative evaluation? Is there an easy way to get started?
  9. 9. The answer is “yes”... And you will crush it.
  10. 10. Methods: Interviews A structured protocol of questions with a target user. Benefits: Learn the vocabulary and logic of your users. Deep investigation of appeal, engagement and comprehension. Limitations: Recruitment, research and analysis is time-intensive.
  11. 11. Methods: Interviews Sample questions: Appeal: What badge would you want to share with a friend? Which would you apply for? Not apply for? Comprehension: [Stop activity]: What just happened? What does this badge mean?
  12. 12. Methods: Think Aloud Protocol As a user walks through your product they narrate what they notice, think, feel, look at, and decide. Benefits: Observe hesitations, hiccups and trouble spots. Observe positive and negative reactions. Limitations: Interviewer needs to stay quiet, can be awkward.
  13. 13. Methods: Think Aloud Protocol Sample Questions: You have completed a project and are applying for a badge. Apply for a badge, describing what you notice and think at each step of the way. Select a badge you’d like to apply for. Describe and explain each action you make.
  14. 14. Example: P2PU Course UX asked folks to create a course 3 respondents Video recording of users--where do they get stuck?
  15. 15. Example: P2PU Course UX Results informed our new Course UX Full report: design-research-course-creation
  16. 16. Methods: Focus Groups A structured protocol with 2+ more targeted users. Benefits: Feedback from several users at once. Limitations: In a group setting, bias can be an issue. Recruiting several users can be difficult.
  17. 17. Methods: Focus Groups Sample questions: Appeal: What did you like most? Least? Engagement: What would you like to know more about? Comprehension: What do you understand the difference between [Badge X] and [Badge Y] to be?
  18. 18. Methods: Questionnaires Series of questions that may have standardized answers. Benefits: Lightweight in regards to collection and analysis. Limitations: Tricky to ask precise and reliable questions. No observational data, questionnaires are all self- reported data.
  19. 19. Methods: Questionnaires Sample Questions Appeal: Rate how much you like/dislike [Badge X] on a 1-5 scale (1=least, 5=most) Comprehension: Rate the clarity of [Badge X] on a 1-5 scale (1=least clear, 5=most clear) What are 3 things you learned about from [X]?
  20. 20. Methods: Click-testing Veryifyapp asks users to click on their preferred design, annotate the screen, or test the labels on your page Benefits: Great for expectation questions “Where would you expect to find...” Testing navigation & nomenclature Inexpensive & fast Limitations: Adult testers only
  21. 21. Example: P2PU Home Page Users were asked to highlight where they could click to create a course 538 viewers, 62 responses
  22. 22. Example: P2PU Home Page Results informed our new home page Full report: user-research-p2puorg-home-page
  23. 23. Sample Project Schedule Identify research areas--what are you evaluating? 1 Decide upon tools and methods. Recruit participants from target user base. NB: You may need a release. Conduct research. Analyze results & present to project team. Target timeframe: 1 week.
  24. 24. Mad Props Karen Brennan, Assistant Professor, Harvard Graduate School 1 of Education Mindy Brooks, Director of Education and Research, Sesame Workshop Brad Felix, Chief Products Officer, Sangari Global Education Allison Gevarter, Interactive Producer, CloudKid Media Ilona E. Holland, Lecturer, Harvard Graduate School of Education
  25. 25. Ask things! Vanessa Gennarelli Learning Lead, P2PU Master’s Candidate, Harvard University @mozzadrella