Measuring Our Impact: The Open.Michigan Initiative


Published on

This presentation was given at the OpenCourseWare Consortium Global Meeting in May, 2011. It describes some of the results from an evaluation project initiated by Open.Michigan in September 2010. Full results can be found at

Published in: Education
1 Like
  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • >2007 pilot; 2008 launch >UMMS (Office of Enabling Tech) and School of Info>2008 launch, UMMS with goal of making first 2 years of medical school content openly licensed.>small staff (currently 4 full time); needed student (volunteer centric approach) support> Large research institution with ~40,000 students and a very distributed academic system across units; 3 campusesUMMS 1,100 students; LSA 19,000
  • >Start using Google Analytics and start doing systematic and consistent surveys and evaluations for our work.  >Phase One: Define the mission and objectives of Open.Michigan >Phase Two: Conduct environmental scan of other Evaluations in Open Education >Phase Three: Complete inventory and review of internal documents Develop analyses such as PEST and SWOT >Phase Four: Formulate and implement strategy; Develop survey tools>Phase Five: Distribute survey Analyze data and synthesize results
  • >Analysis of production, impact, and external knowledge of the open materials created>Other orgs doing similar things (government, institution and nonprofit orgs; OCW & OER)> Qualitative: questionnaires, surveys, interviews for qualitative data, and focusing almost entirely on >Quantitative: web analytics>Analyzed: the production of open material, the use (impact) of open material, and knowledge and opinion of the open materials provided by those outside of the providing organization itself.  Overall:  >>ISKME case study frameworks and “Adoption and Use of Open Textbooks” >>CIC Open Access awareness environmental scan, 2010>>Open Learning Initiative activities>>Open Learning Network activities>>Atkins, Brown et. al. “A Review of the Open Educational Resources (OER) Movement: Achievements, Challenges and New Opportunities” Hewlett report 2007>>Surveys, analytics, follow up and logs.>NSSE report on student engagement>CIBER report on social media use >OER/OCW specific evaluations: >>Tufts: site users who were also faculty members Tufts OCW “positively affects their teaching practices by providing additional teaching materials, by enabling them to integrate Tufts materials into their courses, by increasing their knowledge levels in certain areas and impact how course materials are developed by emphasizing instructional technology.”>>MIT: Self-reports and analyses showed that educators, students, and self-learners alike all use OCW & satisfied by the content available, and the same survey show high impact on their learning goals>>Wikiwijs (Dutch government OER) peer review and recommendation system
  • >Vision document: outlines formalized and explicit mission and goals for Open.Michigan>Priorities for future growth and a three-year plan. >Created in tandem with determining the outcomes that we could measure.  Future Growth:Increase support for OER productionExpand and improve OER offeringsAdd value to content and services
  • > 3 platforms have hosted our content over the past few years: 1) eduCommons: hosted all our OER there2) /education: site-integrated (static Open.Michigan pages) eduCommons instance of OER.3) OERbiteverything is hosted on our Drupal instance including static pages and OER; our wiki and blog are also measureable. >Look at all Google Analytics reports to get full picture of growth. >Referring sites over past 3 yearswikipediayoutubefacebook>First year: local medical gateways and large entities like creative commons and ocwc>Second year: traffic from international sites in china and India as well as social sites like facebook and twitter>Third year: getting more traffic from a variety of sites including ocwc, other oer sites like einztein and project sites
  • 5/3/10-5/3/11: 77,808 visitsover 12,000 are from the Ann Arbor areaover 31,000 are internationally based >Site visits: Coming from 127 countries; China, India and Great Britain being the largest. >Downloads 1231 (last year): syllabi; and pdfs.
  • >Steadily grown in resources: most in SI and UMMS where we have strongest ties.>M1/M2 sequences being published at a regular rate (32% published as of late 2010, must get permission and clear material); now piloting synching audio from lectures to our cleared slides with good results. >dScribes to date: 69 graduate students2 undergraduate students1 librarian 1 staff member 1 visiting scholardScribes have assisted Open.Michigan and 41 faculty in publishing 39 courses
  • >Investment (financial resources, staff, admin support) devoted to OER production and development>Have good horizontal investment and some support from high administration but need more  Examples of support: >SI (we are a preferred employer, invited to participate in local events to recruit students, use their space, heavily recruit from their cohort for employees and volunteers 2008-2011)>Library (coordinate activities, co-facilitate events, participate in local activities with them 2010-2011)>MERLOT (2011) joint collaboration on grant funded projects>Global Reach (share information and facilitate each other’s activities) 2010-2011
  • >Survey results: supportRespondents ambivalent about participating in Open.Michigan efforts (mostly in the neutral or “agree” categories) >Surveys based on environmental scan of other evals, CTools survey, own community interest >CTools survey>>U-M’s Sakai LMS>>2007-2010 had OCW/Open.Michigan questions> Our survey>>all students, all faculty and staff units that support teaching and learning) >>survey about 20 questions, measured sharing habits (do you share, with whom do you share , awareness of and use of OER, social comparison (I would share “if” and support of Open.Michigan)>>Awareness, Value, Use 2010 Results: >Mostly neutral in their responses to OER use and impact (students and instructors)
  • >Administered via email>Staff included: technology, instruction and design  Staff respondents were heavily from the libraries, which may have heard of us more than other staff units across campus Of the surveys, the staff is our least confident survey sample size and response rate.Sharing habits (what, with whom, why)Awareness of Open.Michigan and OERUse of OER (why, how, with whom, create or use?)Impact of OER (in teaching, learning or professional development)Support of OER efforts and Open.MichiganAwareness Students>94% students don’t know about Open.Michigan>84% students never heard of OER >Validity testing: Haven’t heard of OER (1409) vs Heard of/Used/Created OER (256): significantly more students have not even heard of OER than those who have heard of, used or created OER combined.  Faculty>82% faculty don’t know about Open.Michigan>79% faculty never heard of OER>>121 never heard of; 73 heard of/used/created Staff79% staff don’t know about Open.Michigan55% staff never heard of OER
  • >Faculty share lecture slides 79% of the time; syllabi 63% of the time >Staff share reading resources 72% of the time; lecture slides 32% of the time otherwise they share data, administrative information or communications. >Students are mostly sharing on google docs, websites or dropbox: posting to networks (closed) or openly online
  • Lecture slides, syllabi, assignments most popular used/viewed/adapted by faculty and students. Faculty  >Faculty: Viewing how other faculty in my area are approaching materials (32.4%) >Faculty: Encourage colleagues to publish on Open.Michigan (43.9%)> 45.8% use course materials themselves > 37.5% publish their course materials on Open.Michigan >> Used lecture slides from individual lectures in all three categories - viewing, downloading, and adapting.  >> Use of OER as a database of resources  >> Syllabi and assignments viewed more often than they were downloaded or adapted> Assignments were adopted almost as many times (8) as they were viewed (13) Students >Students: previewing prospective courses in depth before I register (35%) >Students: 56.5% would use course material from Open.Michigan > 49.3% would encourage other students to use course material on Open.Michigan > 22.2% would support the publishing process  >>Students heavier on the use/viewing side of OER use and less on the adapting side.  >>Individual lecture slides (also 1st by far), syllabi (2nd by a good margin), and assignments, the fourth most popular option.  >E-textbooks proved to be more popular among students than databases of resources > Most popular OER: individual lecture slides (the most popular among both groups), syllabi and assignments.  
  • - Indeed making an impact but at a mid-level of administrative support and in the broader national/international open education landscape.Still need to work on education about the benefits of OER use and production in the classroom and in learning settings. Still need to work on getting a simple, clear message across to our faculty, students and staff who create learning resources and who use them. Positive results from dScribes, where they use OER more, understand copyright law better, and understand how to use open licenses. They have also reported learning results (being better prepared for courses, tests or learning how to study better through this process) (dScribe exit interviews 07-08 Medical survey).This is what the people want: the set of actionable items. Use words like:Free ShareableAdaptableAttributablePublicNot: Open Educational Resources or “distributable”Staff Staff did not seem to show a preference for one kind of OER use over another. They do, however, appear to view OER much more than they download and/or adapt them.
  • >Overall what we’re finding aligns with the vision statement we created. >Future Growth:Increase support for OER productionExpand and improve OER offeringsAdd value to content and services >Ongoing evaluation: tailored to the interests and needs of our community>Based on the feedback from the surveys: focus on delivering a clear, consistent message and making OER that is directly useful to our community.>Closing the Loop- continue analysis and determine if our assessments are making a difference and making sure we implement changes that are based in community interests and needs.
  • >Awareness of Open.Michigan and OER>Refine survey questions/make simpler >Biggest comment and feedback was “I still don’t know what you do or what OER is!” (mostly from students). > Scope our impact research closer to individual communities and thus focus groups>Not a research unit: a practice and research unit: balance good thought with action. Use of our data  >Google Analytics: developing a module in OERbit to show this information to others>Aggregate survey results: available on request
  • Measuring Our Impact: The Open.Michigan Initiative

    1. 1. Measuring our Impact: <br />The Open.Michigan Initiative<br />CC: BY-NC-SA, Choconancy1Flickr<br />Emily Puckett Rodgers<br />Open Education Coordinator, Open.Michigan<br />OpenCourseWare Consortium Global Meeting 2011<br />Except where otherwise noted, this work is available under a Creative Commons Attribution 3.0 License.<br />Copyright 2011 The Regents of the University of Michigan<br />
    2. 2. Introduction<br />Evaluation team <br /><ul><li>Emily Puckett Rodgers
    3. 3. Ted McCarthy
    4. 4. Pieter Kleymeer</li></ul>Evaluation: started <br />September 2010<br /><br />
    5. 5. Evaluation Design<br />“Through a combination of existing and new qualitative (including surveys, interviews and user feedback) and quantitative (including web analytics, published resources) data, we aim to measure Open.Michigan’s impact on the University of Michigan campus and on the broader open education landscape, as well as its progress toward overall objectives and mission.”<br />Mission and Objectives<br />Environmental Scans<br />Review<br />Implementation<br />Document Study <br />and Review<br />Analysis<br />Organizational <br />analyses<br />Research development <br />and deployment<br />
    6. 6. Environments<br /> Environmental Scan<br /> General evaluations, activities in Open Education <br /> Specific evaluation projects and documentation: <br /><ul><li>Tufts OpenCourseWare (2010)
    7. 7. Free to Learn Guide (2010)
    8. 8. MIT OpenCourseWare (2006)
    9. 9. Oxford’s Technology-Assisted Lifelong Learning (2010)</li></ul>Document Study<br /><ul><li>dScribe data
    10. 10. Wiki
    11. 11. Shared internal documents </li></ul>and spreadsheets <br />
    12. 12. Strategic Vision<br />Open.Michigan’s Strategic Vision<br />Open.Michigan enables University of Michigan faculty, students, staff and others to share their educational resources and research with the world.<br />Open.Michigan’s efforts contribute to two primary goals:<br />to sustain a thriving culture of sharing knowledge at U-M<br />to provide comprehensive public access to all of U-M’s scholarly output<br /><br />
    13. 13. Strategic Plan<br />From the Vision document <br />Measurable Activities<br /><ul><li>Building communities of OER producers and users
    14. 14. Consulting and Outreach Services to facilitate OER production
    15. 15. Development of Processes and Software to support OER production and publishing</li></ul>From the Evaluation plan<br />Guiding Objectives<br /><ul><li>How and Why is our OER being used?
    16. 16. Who is using our OER?
    17. 17. What value does OER bring to U-M?
    18. 18. In what contexts are people using our OER? </li></ul>Next Steps: Goals and Objectives document, Communications Plan<br />
    19. 19. Impact: Growth<br />OERbit<br />static Open.Michigan pages<br />/education OER<br />Open.Michigan platform migrations<br />eduCommons OER<br />9/16/2010<br />8/27/2009<br />4/1/2008<br />4/2011<br />
    20. 20. Impact: Growth<br />Google Analytics<br />
    21. 21. Impact: Contributions<br />Resource facilitation<br /><ul><li>Anishinaabelanguage and literature
    22. 22. African Health OER Network
    23. 23. “Textbook of the Future” discussions
    24. 24. Center for Global Engagement Handbook</li></ul>Resources available<br /><ul><li>~70 courses published
    25. 25. 2 campuses
    26. 26. 12 schools
    27. 27. ~9 OER modules published
    28. 28. ~48 resources deposited in Deep Blue </li></ul><br />
    29. 29. Impact: Contributions<br />Contributions to Open Education<br /><ul><li>OERca, released 2008 (copyright and content clearance tool)
    30. 30. OERbit, released 2011(publishing platform)
    31. 31. dScribe, process for using volunteers and community members to clear content and publish OER
    32. 32. Code of Best Practices in Fair Use for OpenCourseWare (2009)
    33. 33. African Health OER Network </li></li></ul><li>Impact: Investment<br />Institutional Support (U-M)<br />Dean support: <br /><ul><li>Dean Woolliscroft (UMMS)
    34. 34. Dean MacKie-Mason (SI)
    35. 35. Dean Potempa (Nursing, retired)
    36. 36. Paul Courant (MLibrary)</li></ul>Financial support: <br /><ul><li>UMMS
    37. 37. Hewlett Foundation
    38. 38. FAIMER
    39. 39. Open Society Institute
    40. 40. Local grants </li></ul>Institutional support and partnerships: <br /><ul><li>Medical School
    41. 41. School of Information
    42. 42. MLibrary
    43. 43. MERLOT</li></ul>African Health OER Network: <br /><ul><li>Global Reach
    44. 44. African Studies Center
    45. 45. Ghana-Michigan Charter
    46. 46. Ghana Emergency Medicine Collaborative</li></ul>“Having the big block M on a lot of high quality produced teaching modules that are made available to the world… is mission consistent, so what’s the business that we’re in here? We create and distribute knowledge.” -Paul Courant, “Why Open is Important” interview<br />
    47. 47. Impact: Surveys<br />CTools survey<br />40.7% instructors never heard of OCW<br />75.6% students never heard of OCW<br /><br />
    48. 48. Impact: Surveys<br />Open.Michigan and Open Educational Resource survey to U-M<br />Population Size Response Rate Completion Rate<br />39,958 students 27.5% 71%<br />6,149 faculty 30.4% 68%<br />2,268 staff 16.5% 55%<br />Themes: <br /><ul><li>Sharing habits
    49. 49. Awareness of Open.Michigan and OER
    50. 50. Use of OER
    51. 51. Impact of OER
    52. 52. Support of OER efforts and Open.Michigan</li></li></ul><li>Impact: Surveys<br />Sharing<br />"I share learning materials with my colleagues to..." ? (faculty responses)<br />
    53. 53. Impact: Surveys<br />Use <br />“What type of OER have you used?” (student responses)<br />
    54. 54. Impact: Surveys<br />(Some) Results<br />Themes from survey comments <br /><ul><li>Establish Open.Michigan in main channels of U-M communication
    55. 55. Clarify copyright in OER and provide resources
    56. 56. Consistently define Open.Michigan and OER
    57. 57. Describe why/how to use/create OER
    58. 58. Make short-term improvements to OER
    59. 59. Encourage cultural shifts in learning</li></ul>CC: BY Emily Puckett Rodgers<br />Next Steps: participating faculty surveys<br />
    60. 60. Next Steps<br />Evaluation<br /><ul><li>Finish survey analysis
    61. 61. Focus Groups</li></ul>Strategy<br /><ul><li>Goals and Objectives
    62. 62. Communications plan
    63. 63. Advisory committee</li></ul>In the next three years, Open.Michigan will:<br />Producemore and richer content as OER with the various campus units, improve modularity, instructional design, and accessibility of U-M OER<br />Increase the visibility and discoverability of U-M resources through a combination of marketing and metadata<br />Draw participants from more parts of campus to expand its disciplinary coverage<br />Ensure OER production is an embeddedpart of the academic life on campus<br />
    64. 64. Resources<br />Use <br /><ul><li>Evaluation wiki page
    65. 65. Evaluation plan
    66. 66. Survey questions
    67. 67. Strategic planning
    68. 68. Aggregate data
    69. 69. Our Google Analytics data
    70. 70. Aggregate survey results</li></ul><br />Contact<br />Emily Puckett Rodgers<br /><br /><br /><br />
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.