Cooperative Learning Object Exchange (CLOE) A Learning Object Project Case Study Peter Tittenberger, July 2009

Snapshot history 1997?: 3 Ontario institutions join MERLOT as an institutional consortium [Guelph, Waterloo, York]. Each pays 1/3 of the membership fee 1998: 8 Ontario institutions in MERLOT consortium, each contributing share of fee + faculty time to participate 1998-2000: grant from TeleLearning NCE to investigate further community and collaboration models 1999: grant from Industry Canada for students to work on "learnware" 2001-2003: grant from HRDC Office of Learning Technologies to fund learning object development with some investment in infrastructure for CLOE 2004-2005: grant from Inukshuk to collaboratively develop learning objects 2006-2007: UWaterloo continues to subsidize CLOE operation, no success in identifying sources for ongoing funding 2008: operation effectively ceases, although website available until domain registration expires. Via Tom Carey email, July 2009

1997?: 3 Ontario institutions join MERLOT as an institutional consortium [Guelph, Waterloo, York]. Each pays 1/3 of the membership fee

1998: 8 Ontario institutions in MERLOT consortium, each contributing share of fee + faculty time to participate

1998-2000: grant from TeleLearning NCE to investigate further community and collaboration models

1999: grant from Industry Canada for students to work on "learnware"

2001-2003: grant from HRDC Office of Learning Technologies to fund learning object development with some investment in infrastructure for CLOE

2004-2005: grant from Inukshuk to collaboratively develop learning objects

2006-2007: UWaterloo continues to subsidize CLOE operation, no success in identifying sources for ongoing funding

2008: operation effectively ceases, although website available until domain registration expires.

CLOE included: Collaborative design and learning object development Peer Review process Learning object repository Administrative infrastructure

Collaborative design and learning object development

Peer Review process

Learning object repository

Administrative infrastructure

List of CLOE institutions: Brock University Carleton University Conestoga College Durham College University of Guelph Lakehead University La Cité Collégiale Lambton College Laurentian University McMaster University Memorial University of Newfoundland Ministry of Science and Technology, Thailand Niagara College Nipissing University Queen's University Ryerson University Wendy Freeman Seneca College Sheridan College Trent University University of Manitoba University of Ontario Institute of Technology University of Ottawa University of Toronto University of Waterloo University of Western Ontario University of Windsor Wilfrid Laurier University York University

1. Collaborative design and learning object development Grant funded 2004-2005 Inukshuk fund ($2 million) Camp Cloe a one week Instructional Design workshop. CLOE partners attend and work in teams on the collaborative design of LOs and then these LOs are developed over the year.

Grant funded

2004-2005 Inukshuk fund ($2 million)

Camp Cloe

a one week Instructional Design workshop. CLOE partners attend and work in teams on the collaborative design of LOs and then these LOs are developed over the year.

Camp Cloe Instructional Design & Learning Technology Workshop Learning Object Design Process Mock Learning Objects Project Management Understanding Learners Prototyping & User Testing Electronic Prototyping http://web.archive.org/web/20051029195100/tlc.uwaterloo.ca/projects/cloe/CaseStory/

Instructional Design & Learning Technology Workshop

Learning Object Design Process

Mock Learning Objects

Project Management

Understanding Learners

Prototyping & User Testing

Electronic Prototyping

CLOE design and development methodology created a rigorous set of standards for learning object creation.

http://www.brocku.ca/learningobjects/flash_content/index.html

http://www.brocku.ca/learningobjects/flash_content/LO/Stroop_Effect.html

Access Restrictions: These Learning Objects are being developed by ITS (Information Technology Services) in conjunction with CTLET (Centre for Teaching, Learning and Educational Technologies). All objects represent works near completion. As they are currently under development, they are authorized for use by Brock University personnel only, with permission. Completed learning objects are peer reviewed and can be accessed through CLOE (Co-Operative Learning Object Exchange). Please contact CTLET if you would like to apply for full access to CLOE or if you would like more information on Brock University's learning object development. http://www.brocku.ca/learningobjects/flash_content/index.html Last updated: Friday, 01-Oct-2004 16:16:30 EDT

Form to add object to CLOE

2. Peer Review CLOE partners concluded that a hallmark of CLOE should be that all LOs in CLOE should be Peer Reviewed. CLOE created its own Peer Review process modeled after the MERLOT Peer Review process ( http://taste.merlot.org/catalog/peer_review/ ) The CLOE Guidelines to Authors are used by both (a) creators of LOs to inform their design and development process and (b) CLOE Peer Reviewers when conducting a peer review.

The process that the CLOE Editor-in -Chief uses when a new LO is submitted to CLOE:

Guidelines for authors creating los and for peer reviewers evaluating los.

CLOE peer review process created a rigorous methodology and set of standards for learning object submission and review.

3. Learning object repository Stand alone web site. Access restricted. No federated search. D2L arrangement. Reuse procedure (form). Licensing.

Stand alone web site.

Access restricted.

No federated search.

D2L arrangement.

Reuse procedure (form).

Licensing.

CLOE lo ‘borrowing’ form.

4. Administrative infrastructure UWaterloo provided director, admin support and technical components and support Organized and hosted Camp Cloe Memoranda of Understanding (MOUs)

UWaterloo provided director, admin support and technical components and support

Organized and hosted Camp Cloe

Memoranda of Understanding (MOUs)

Memorandum of Understanding When partners join CLOE a high-level official at that institution signs the CLOE Memorandum of Understanding (MOU). The original MOU stated that each partner’s primary contribution was to submit LOs to CLOE and to reuse LOs from CLOE.

When partners join CLOE a high-level official at that institution signs the CLOE Memorandum of Understanding (MOU).

The original MOU stated that each partner’s primary contribution was to submit LOs to CLOE and to reuse LOs from CLOE.

Memorandum of Understanding Potential new partners indicated that they wanted to be full and equal partners in CLOE and yet some of them indicated that based on the size and nature of their institution that they might not submit any LOs to CLOE. To address this issue, CLOE partners have created a new expanded MOU which allows partners to choose their commitment from a list of potential commitments. Each partner is not required to choose a specific number of items from the list but rather to choose the items that they can contribute.

Potential new partners indicated that they wanted to be full and equal partners in CLOE and yet some of them indicated that based on the size and nature of their institution that they might not submit any LOs to CLOE. To address this issue, CLOE partners have created a new expanded MOU which allows partners to choose their commitment from a list of potential commitments. Each partner is not required to choose a specific number of items from the list but rather to choose the items that they can contribute.

Memorandum of Understanding Collaboration in the design of a LO Collaboration in the development and subsequent use of a LO Collaboration in the formative evaluation of the use of a LO Peer review of LOs Collaboration in the adaptation of a LO for use at the partner institution (normally this would not involve modifying the LO but rather documenting the use of the LO in a setting the is different than the setting envisioned by the designers and developers. For example, a LO created for 1st year Psychology being reused in a 2 nd year statistics course) Reusing a LO from CLOE and providing summative evaluation to CLOE and the creators regarding the reuse. List of potential CLOE partner commitments.

Collaboration in the design of a LO

Collaboration in the development and subsequent use of a LO

Collaboration in the formative evaluation of the use of a LO

Peer review of LOs

Collaboration in the adaptation of a LO for use at the partner institution (normally this would not involve modifying the LO but rather documenting the use of the LO in a setting the is different than the setting envisioned by the designers and developers. For example, a LO created for 1st year Psychology being reused in a 2 nd year statistics course)

Reusing a LO from CLOE and providing summative evaluation to CLOE and the creators regarding the reuse.

Empower project leadership to set and achieve goals Select leaders and key staff with requisite experience; clearly communicate mission and goals of the organisation; and create an atmosphere that encourages an entrepreneurial spirit, including a willingness to test new ideas. Craft a strong value proposition Create a resource that offers unique value and continue to add value to the resource based on an understanding of users’ needs. Find creative ways to lower the direct costs of running the project. Secure contributions from the host institution; outsource work through vendors and other external partnerships; work with volunteers. Cultivate sources of revenue to cover both direct costs and ongoing upgrades. Experiment with different revenue models to find the ones that are the best fit for the project; show willingness to try new models; cultivate the ability to identify and communicate the value of the resource to the target audience (of customers, authors, subscribers and so forth). Establish a system of accountability and measurement of the success of the resource and the revenue model. Establish goals and targets and determine the balance between financial and mission-related returns; assess progress towards mission-based and financial goals and targets . Sustaining Digital resources: An On-the-Ground View of Projects Today http://www.ithaka.org/ithaka-s-r/strategy/ithaka-case-studies-in-sustainability/report/SCA_Ithaka_SustainingDigitalResources_Report.pdf Key factors for Sustainability

Empower project leadership to set and achieve goals

Select leaders and key staff with requisite experience; clearly communicate mission and goals of the organisation; and create an atmosphere that encourages an entrepreneurial spirit, including a willingness to test new ideas.

Craft a strong value proposition

Create a resource that offers unique value and continue to add value to the resource based on an understanding of users’ needs.

Find creative ways to lower the direct costs of running the project.

Secure contributions from the host institution; outsource work through vendors and other external partnerships; work with volunteers.

Cultivate sources of revenue to cover both direct costs and ongoing upgrades.

Experiment with different revenue models to find the ones that are the best fit for the project; show willingness to try new models; cultivate the ability to identify and communicate the value of the resource to the target audience (of customers, authors, subscribers and so forth).

Establish a system of accountability and measurement of the success of the resource and the revenue model.

Establish goals and targets and determine the balance between financial and mission-related returns; assess progress towards mission-based and financial goals and targets .

Why did CLOE fail? LO Design and development? Peer review? Repository/reuse? Administrative support?

LO Design and development?

Peer review?

Repository/reuse?

Administrative support?

Why did it fail? LO Design and development Once grant funding ended no financial incentive to collaboratively design and develop. Few universities and colleges had technical capacity built in (most CLOE LO development by students). No demand by faculty to create objects for courses using CLOE methodology.

LO Design and development

Once grant funding ended no financial incentive to collaboratively design and develop.

Few universities and colleges had technical capacity built in (most CLOE LO development by students).

No demand by faculty to create objects for courses using CLOE methodology.

Why did it fail? Peer review No recognition of peer review in promotion, retention, and tenure. No interest by faculty in peer reviewing without recognition Process too elaborate?

Peer review

No recognition of peer review in promotion, retention, and tenure.

No interest by faculty in peer reviewing without recognition

Process too elaborate?

Why did it fail? Repository/reuse Cumbersome registration/paperwork to reuse lo. Low demand for lo reuse D2L repository never becomes operational The context question not an open resource remix/repurpose restricted through no access to source code or license no demand because …

Repository/reuse

Cumbersome registration/paperwork to reuse lo.

Low demand for lo reuse

D2L repository never becomes operational

The context question

not an open resource

remix/repurpose restricted through no access to source code or license

no demand because …

Impact on teaching practices: “ The foremost reason for not using digital resources was that they did not support faculty’s teaching approaches.” Time to identify and adapt resources : “Lack of time was a major constraint, regardless of institution.” Reusing resources in new contexts : “Faculty, including those active and enthusiastic in their use of digital resources, identified many other obstacles to using these resources for teaching, including how to . . . reuse them in new contexts” Opening Up Education http://mitpress.mit.edu/books/chapters/0262033712chap12.pdf

“ The foremost reason for not using digital resources was that they did not support faculty’s teaching approaches.”

Time to identify and adapt resources : “Lack of time was a major constraint, regardless of institution.”

Reusing resources in new contexts : “Faculty, including those active and enthusiastic in their use of digital resources, identified many other obstacles to using these resources for teaching, including how to . . . reuse them in new contexts”

Why did it fail? Administrative support UWaterloo pulls out support after funding ends Unsustainable by volunteers/in kind contributions

Administrative support

UWaterloo pulls out support after funding ends

Unsustainable by volunteers/in kind contributions

Key factors for Sustainability Assume grant funding will not always be available Define sustainability – determine need to generate revenue beyond operating costs Understand, monitor and measure the demand side of the project Consider options for long term governance Develop marketing plans and seek out strategic partnerships Be prepared to change direction in a rapidly changing environment Leaders need to be fully dedicated, fully invested and focused Support a culture of experimentation and innovation Sustainability and Revenue Models for Online Academic Resources http://sca.jiscinvolve.org/files/2008/06/sca_ithaka_sustainability_report-final.pdf

Assume grant funding will not always be available

Define sustainability – determine need to generate revenue beyond operating costs

Understand, monitor and measure the demand side of the project

Consider options for long term governance

Develop marketing plans and seek out strategic partnerships

Be prepared to change direction in a rapidly changing environment

Leaders need to be fully dedicated, fully invested and focused

Support a culture of experimentation and innovation

Empower project leadership to set and achieve goals Select leaders and key staff with requisite experience; clearly communicate mission and goals of the organization; and create an atmosphere that encourages an entrepreneurial spirit, including a willingness to test new ideas. Craft a strong value proposition Create a resource that offers unique value and continue to add value to the resource based on an understanding of users’ needs. Find creative ways to lower the direct costs of running the project. Secure contributions from the host institution; outsource work through vendors and other external partnerships; work with volunteers. Cultivate sources of revenue to cover both direct costs and ongoing upgrades. Experiment with different revenue models to find the ones that are the best fit for the project; show willingness to try new models; cultivate the ability to identify and communicate the value of the resource to the target audience (of customers, authors, subscribers and so forth). Establish a system of accountability and measurement of the success of the resource and the revenue model. Establish goals and targets and determine the balance between financial and mission-related returns; assess progress towards mission-based and financial goals and targets . Sustaining Digital resources: An On-the-Ground View of Projects Today http://www.ithaka.org/ithaka-s-r/strategy/ithaka-case-studies-in-sustainability/report/SCA_Ithaka_SustainingDigitalResources_Report.pdf Key factors for Sustainability

Empower project leadership to set and achieve goals

Select leaders and key staff with requisite experience; clearly communicate mission and goals of the organization; and create an atmosphere that encourages an entrepreneurial spirit, including a willingness to test new ideas.

Craft a strong value proposition

Create a resource that offers unique value and continue to add value to the resource based on an understanding of users’ needs.

Find creative ways to lower the direct costs of running the project.

Secure contributions from the host institution; outsource work through vendors and other external partnerships; work with volunteers.

Cultivate sources of revenue to cover both direct costs and ongoing upgrades.

Experiment with different revenue models to find the ones that are the best fit for the project; show willingness to try new models; cultivate the ability to identify and communicate the value of the resource to the target audience (of customers, authors, subscribers and so forth).

Establish a system of accountability and measurement of the success of the resource and the revenue model.

Establish goals and targets and determine the balance between financial and mission-related returns; assess progress towards mission-based and financial goals and targets .

Thank you.