The document discusses the need for institutions to take a more formal approach to evaluating learning technologies. It introduces some existing evaluation frameworks like the Educause rubric and SECTIONS model. It then outlines UCEM's approach, which involved thoroughly investigating requirements, identifying systems to evaluate, developing a testing plan based on the Educause rubric, testing functionality and data flows, and involving stakeholders before selecting a new assessment platform. Attendees at the talk were asked to provide ideas on evaluation processes and challenges through a Mural board.

1. Making your mind up:
Formalising the evaluation
of learning technologies
Marieke Guy and Tharindu Liyanagunawardena, University College of Estate Management
ALT Conference September 2021

2. ©UCEM
Today’s session
• What just happened?
• Introduction to making
technology procurement
decisions
• Taking a more formal
approach – existing
frameworks
• Mural – time for your
ideas
Marieke Guy Tharindu Liyanagunawardena
Your facilitators

3. ©UCEM
What just happened?
• Covid-19 pandemic
• Pivot to online – emergency remote teaching followed by a year of ‘online
teaching’
• Lots of talk of opportunities and affordances of online
• Made the best of what we already had
• Students initially grateful, then disappointed, but want to keep some things
• Low NSS scores across the sector – 75% overall satisfaction
• Taking stock and re-evaluating
• Better investment in online? OFS Gravity Assist: “a need for greater
investment in the digital campus”
• A lot of big decisions

4. ©UCEM
Introduction to making technology decisions
• Pressures from above
• Pressures from below
• Pressures from the side
• Compare, contrast, consider, evaluate, test, analyse, report, negotiate,
involve (stakeholders), align (with strategy & drivers), compromise, placate,
recommend, decide,…
• Solve problems
• Be pragmatic

5. ©UCEM
Useful existing Frameworks #1
• Educause - The Rubric for E-Learning Tool
Evaluation
• Technology acceptance model
• Jacob Nielson usability testing
• Tony Bates SECTIONS model:
• Students
• Ease of use
• Costs
• Teaching functions (including the affordances
of different media)
• Interaction
• Organisational issues
• Networking
• Security and privacy

6. ©UCEM
Useful existing Frameworks #2
• Change management approaches
• ADKAR (Awareness, Desire, Knowledge, Ability,
Reinforcement)
• McKinsey 7S Model (strategy, structure, systems,
shared values, style, staff, skills)
• Lewin, Bridges, Kotter etc.
• CMALT – constraints and benefits
• Change models help us with understanding potential
reasons for failure

7. ©UCEM
UCEM approach #1
Selection of a new assessment platform:
• Thorough investigation of existing process
• Functional requirements document
• Identification of systems to evaluate
• Systematic testing and evaluation plan
(inspired by Educause rubric)

8. ©UCEM
UCEM approach #2 – Educause rubric
Functionality
• Scale.
• Ease of Use.
• Tech Support / Help
Availability.
• Hypermediality.
Accessibility
• Accessibility Standards.
• User-Focused Participation.
• Required equipment
• Cost of Use.
Technical
• Integration/Embedding within
a Learning Management
System (LMS).
• Desktop/Laptop Operating
Systems and Browser.
Mobile Design
• Access.
• Functionality.
• Offline access
Privacy, Data
Protection, and Rights
• Sign Up / Sign In.
• Data Privacy and Ownership.
• Archiving, Saving, and
Exporting Data.
Social Presence
• Collaboration.
• User Accountability.
• Diffusion.
Teaching Presence
• Facilitation.
• Customization.
• Learning Analytics.
Cognitive Presence
• Enhancement of Cognitive
Task(s).
• Higher-Order Thinking.
• Metacognitive engagement

9. ©UCEM
UCEM approach #3
• Systematic testing on VLE (using testing team and assigned roles)
• Scenarios and tasks (assessment workflow)
• Roles
• Data (flow)
• Environments (browsers and devices)
• Other - documentation
• Completion of decision matrix
• IT software and cloud solution approval process
• Recommendations
• Student consultation panel
• And then the work begins… 

10. ©UCEM
Time for your input…
Using Mural we want you to consider the following questions:
• Do you use existing frameworks? If so please share them
• What works and what doesn’t?
• How can you ensure a thorough evaluation process that is still
time and resource effective?
• Who should be involved in the evaluation process?
• What are the challenges to a more formal process?
• You can carry on adding your ideas after the session
• We will collate responses and write a blog post (UCEM Online
Education)
https://bit.ly/mymu-mural

11. ©UCEM
Any questions?

12. ©UCEM
References
• Reid, P (2018) Identifying, Evaluating, and Adopting New Teaching and
Learning Technologies: https://er.educause.edu/articles/2020/8/identifying-
evaluating-and-adopting-new-teaching-and-learning-technologies
• Anstey, L and Watson, G (2018) A Rubric for Evaluating E-Learning Tools
in Higher Education,Educause Review, September 10,
2018 https://er.educause.edu/articles/2018/9/a-rubric-for-evaluating-e-
learning-tools-in-higher-education
• Technology acceptance
model: https://en.wikipedia.org/wiki/Technology_acceptance_model
• Tony Bates (2015) SECTIONS:
https://teachingcommons.unt.edu/teaching-essentials/teaching-
technology/selecting-educational-technologies-checklist
• UCEM Online Education blog: https://blog.ucem.ac.uk/onlineeducation/

13. ©UCEM Internal Use Only
Thank you
Marieke Guy (@mariekeguy) and Tharindu Liyanagunawardena (@Tharindu_)