Electronic management of assessment slides

1. 19/11/2014 Electronic Management of Assessment:
moving on

2. Electronic management of assessment: moving on 2
Taking part
» Use the chat box to share thoughts and ask questions
» Prefix questions with a “Q”
» Raise your hand if you want to speak
» Technical problems? Send a private message to Phil
(double click on his name)
» Or call 0191 208 5000
» The webinar will be recorded and shared online.

3. Electronic management of assessment: moving on 3
Blackboard
Collaborate
orientation

4. 19/11/2014 Electronic Management of Assessment:
moving on

5. Electronic management of assessment: moving on 5
Overview
»Introductions
»Jisc Electronic management of Assessment (EMA) project and study
»Headline findings so far
»Prioritising challenges to tackle
»Next steps: a service design approach and opportunities to get involved

6. Electronic management of assessment: moving on 6
Introductions
Lisa Gray, Jisc: lisa.gray@jisc.ac.uk
Dr Gill Ferrell, consultant: gill@aspire‐edu.org
Marianne Sheppard, Jisc infoNet:
marianne.sheppard@northumbria.ac.uk
Alex Nisbett, Livework alex@liveworkstudio.com
Phil Swinhoe, Training and administration co‐ordinator, Jisc Netskills

7. Electronic management of assessment: moving on 7
Electronic management of assessment
(EMA) project
Background and context
»Recently completed – a 3 year
programme exploring
technology‐enhanced
assessment and feedback
»4 main themes:
› Feedback and feed forward
› Enhancing student
employability
› Strategies for change
› Electronic management of
assessment (EMA)
CC BY‐NC‐SA 2.0 jackhynes http://bit.ly/jisc‐assess and http://bit.ly/jiscdsaf

8. Electronic management of assessment: moving on 8
Electronic management of assessment
(EMA) project
Background and context
»EMA ‐ benefits and drivers are
clear
»Challenges surfacing relating to
process, practice and
technology
»Working in collaboration with
the Heads of e‐Learning Forum
(HeLF) and UCISA to maximise
the benefits technology can
offer in the management of
assessment
CC BY‐NC‐SA 2.0 ©comedynose via Flickr

9. Electronic management of assessment: moving on 9
The EMA study
Aims
»To gather examples of practices,
policies & process as well as
approaches to implementation
»To identify the challenges around
EMA, in terms of pedagogy/culture,
process or technology and to identify
existing solutions where they can be
found
»To provide interim guidance to
institutions on the state of play and
effective practice with approaches to
implementation
»To make recommendations to
institutions, systems providers and
Jisc on priorities to address
CC BY‐NC‐SA 2.0 ©comedynose via Flickr

10. Electronic management of assessment: moving on 10
Information sources
»HeLF EMA surveys 2011, 2012, 2013
»Jisc assessment and feedback programme (2011 to 2014 involving
over 30 institutions)
»Other Jisc learning and teaching related projects
»Literature review
»Online questions (April 2014) – 90 responses from 70 different
institutions (65 HE and 5 delivering HE in FE) covering all four UK
nations
»Think Tank (May 2014) 32 institutions represented
»Interviews with learning providers

11. Electronic management of assessment: moving on 11
Headline findings
»Nuanced responses but broad trends visible
»Localised initiatives but beginning to scale up
»System integration is a key problem area
»EMA exposes variability in business processes

12. Electronic management of assessment: moving on 12
Lots of localised initiatives
How extensively has your organisation applied EMA?
120
100
80
60
40
20
0
e-Submission e-Marking e-Feedback e-Exams
N/A
Small scale pilots
Large scale pilots
Mandated locally
Mandated organisation wide
Don't know

13. Electronic management of assessment: moving on 13
Loose system integration

14. Title of presentation 00/00/2013 14
Variation in business processes

15. Electronic management of assessment: moving on 15
Pain points

16. Electronic management of assessment: moving on 16
What causes the pain?
»Anonymity: both ends of spectrum motivated by student pressure
»Resistance: many stakeholders take narrow view; little joined‐up
thinking
»Scaling up: may expose technology issues & ‘harden’ staff attitudes
»Feedback: storage & access issues; good practice can be hard to scale
»Practice: may be simple but process workarounds cause frustration &
resentment
»Technology: limited range of core products how hard can it really be?
»Formative approaches: less well developed than summative

17. Electronic management of assessment: moving on 17
A Lifecycle Model

18. Electronic management of assessment: moving on 18
Top 10 Challenges

19. Electronic management of assessment: moving on 19
Top 10 Challenges

20. Electronic management of assessment: moving on 20
Where next?
»Priority area for many institutions
»Student expectations are key
»Continued co‐design with you, Jisc, HeLF & UCISA and engagement
with suppliers
»Second phase of consultation through a service design approach,
moving from challenges to identifying solutions – now till February
2015
»Solution development, including the production of an EMA toolkit –
Summer 2015

21. Electronic management of assessment: moving on 21
Where next?
Phase 1
Mar–July 14
• Landscape review providing initial
identification of challenges and outline of the
state of play with regards to EMA
Phase 2.1
Oct–Aug 15
• Second phase of consultation leading to refined and
prioritised challenges (speaking as a ‘single voice’)
• Moving from challenges to solutions (Oct – Jan)
• Solution development (Feb – July)
Phase 2.2
Oct–Aug 15
• Ongoing development of resources for the ‘toolkit’ based
around the assessment and feedback lifecycle

22. Electronic management of assessment: moving on 22
Moving from challenges to solutions
A service design approach

23. Electronic management of assessment: moving on 23
Service design
What is it?
Service design helps imagine a better future
for customers and organisations alike.

24. Electronic management of assessment: moving on 24
Livework and Jisc
eg. National Monograph Strategy
Re‐configuration of libraries as services
to students and academics

25. Electronic management of assessment: moving on 25
How we approach projects
Our approach is structured around four phases, each
with an activity to verify the chosen direction.
Understand actors and
factors that impact
customers lives.
Validate assumptions
with staff and customers.
Imagine new service
scenarios that add value
to customer
relationships.
Prove what works with
customers and staff.
Design services that
deliver value to our clients
and their customers.
Test frequently and in live
settings.
Help create services that
work in the real world and
deliver value.
Monitor and guide the
implementation.

26. Electronic management of assessment: moving on 26
Moving from challenges to solutions

27. Electronic management of assessment: moving on 27
Two ways of getting involved

28. Electronic management of assessment: moving on 28
Two activities
1. Personas 2. Challenges

29. Electronic management of assessment: moving on
Activity 1: User personas
29
Objectives
To create a rich understanding of the
key service actors (students,
academic staff and administrators).
This will help us in the first workshop
where we imagine solutions to the
challenges.

30. Electronic management of assessment: moving on 30
Activity 1: User personas
How?
1. Choose at least two typologies
(e.g. student, lecturer)
2. Give them a name and define
their role
3. Describe their main
motivations. What do they
want? What do they need to
achieve?
4. Think and describe their
needs.
5. Define the dependencies,
enablers and constraints.

31. Electronic management of assessment: moving on 31
Activity 1: User personas
How?
Visit our blog for the link
Complete a Google form for each.

32. Electronic management of assessment: moving on 32
Activity 2: The challenges
Objectives
To create real, human stories
which bring all of the challenges
to life.
These will help us in the first
workshop where we generate
user‐centred solutions to the
challenges.

33. Electronic management of assessment: moving on 33
Activity 2: The challenges
How?
1. Visit the blog and review each
challenge
2. Select those which you can
think of specific examples,
real stories or anecdotes for.

34. Electronic management of assessment: moving on 34
Activity 2: The challenges
How?
3. Use the form to write your
stories.
4. You will be able to see the
stories others are posting, so
reference or build on them.

35. Electronic management of assessment: moving on 35
Find out more:
»Lisa Gray lisa.gray@jisc.ac.uk
»Landscape report available
from: bit.ly/jisc‐ema
»Join the conversation on the
blog: ema.jiscinvolve.org/
and on twitter #jiscassess
»Join the mailing list:
www.jiscmail.ac.uk/tech‐enhanced‐
assesment
»Guides, case studies, videos
and other resources on
assessment and feedback:
bit.ly/jisc‐assess

36. Any questions?