The document discusses the role of assessment in learning. Traditionally, instructors provided assessment to gauge understanding and adapt learning paths. New technologies focus on flexible delivery but lack assessment. The solution, Compertum, captures learners' response behaviors during assessment to measure certainty without impacting experience. It analyzes response patterns to identify those certain but incorrect, and less certain but correct, enabling targeted training to solidify learning and reduce risks from applying wrong information.

2. What is the
Fundamental
Role of
Learning?

3. Learning
Transformation
Typical learning transformation involves;
1. Taking a candidate lacking in understanding
2. Utilising a range of learning mechanisms to impart knowledge and skills
3. An examination to check the person has learned what is necessary.
Not so long ago this process would only have been delivered using an instructor, who would utilise
the best tool for the job such as information, examples and practice, and perhaps undertaken some
sort of needs assessment using questions to the classroom to gauge initial understanding.

4. Learning
Transformation
The Learning Path
The learning path is a sequence of assessment that a learner requires to master a topic. This
path is usually not so direct as shown in the previous slide, and the outcomes of action cannot
be so assured.
Having an instructor in the loop allows subtle ongoing assessment of the recipient and the
journey to be adapted. The instructor is likely to gauge engagement and how much the method
is resonating with the recipient through understanding their reactions, interaction, and body
language.
Essentially this allows the learning journey to be adapted on the fly.

5. Learning
Transformation
Individuals won’t be simple clones of each other so different learning paths may be required for
different audiences.
Some will come with more initial knowledge, and some will be more confident in what they think
they know – not always being confident in the correct information.
This diagram shows a two dimensional split championed by academic research decades ago,
whereby the amount of knowledge or skill is a separate component to the knowledge certainty
of the topic.
For example, someone with the correct knowledge and who is certain of it is more likely to put it
into practice and perform well, not being swayed by others.

6. What is driving
change within
learning?
What advances are being made in assessment?
The main advancements in learning are generally focussed on delivery – making content more
accessible, flexible, and efficient. This usually means doing away with the human instructor
element.
Technology is focussed on 24/7 global delivery and is made effective through advanced
engagement and remote management.
Solutions in this area include: access from mobile devices, gamification to keep engagement,
virtual experiences, and countless clever ways of using media.

7. What is driving
change within
assessment?
What advances are being made in assessment?
With the vast majority of technical advances focussed on learning, there’s little to announce in
terms of assessment – the only major advancement is simply placing quizzes online and making
them easier to access and work through.
This only addresses drivers to improve cost and flexibility – there is nothing trying to make more
use of assessment or ensure learning is effective and resonating with the learner without the
presence of an instructor.
Has the reason to assess gone away or are there fewer advances to make?
It is our belief assessment still plays a critical role – more than completion of a tick box.
Assessment is a huge missed opportunity, which when done alongside learning development can
help solidify learning, provide guidance through the learning journey and increase effectiveness
of content.

8. Learning
Activity
Instructor Led
?
The Assessment
Through time spent with a student, an instructor can gather far more insight into the learning
journey of the student. The instructor is very likely to know who really “got it”, and who is more
likely to put the learning into practice.
This face to face time, with visibility of a person’s reaction to the situation, their body language,
and how comfortable they are in interactions is arguably the best way of understanding success or
failure.
So, with the driver to remove/shortcut the human element increasing, what are we left with?

10. Our
Journey
Q A
We started off with a straightforward question & answer system, and considered what more can
be done.
This caused us to think about modern technology and how we could capture additional
information about how a candidate responded to a question, not just the answer they provided.
The goal being to attempt to close the gap presented with an instructor no longer in the loop.
With modern web browsers, using built in features of modern tablets, computers or phones, we
developed a solution to capture this data unobtrusively – in other words, whilst the learner is
answering the question, we look at indicators of their certainty, without changing user experience.
These indicators were based around responsiveness, hesitation, revisiting questions, changing of
mind etc., and when the data undergoes combining, normalising, distribution analysis, and
collating by subject – we found we could determine an indication of certainty with no impact to
learner activity.
By interfacing with systems, and utilising existing questions, we also found that the set up and
customisation was low.

11. Our
Journey
So how can this information be applied?
Let’s consider again the two measures from earlier – two measure of correctness and certainty.
For clarity, we’re not talking psychological assessment, we are purely considering response analysis
and statistical analysis.
By finding averages in the population we produce four segments, although of course this isn’t black
and white but shades of grey.
Effective learning is the process of moving towards the top right segment (see graph above), whereby
the knowledge and skills are secured with certainty.
The segments could contain people, but equally a person or group’s understanding of different topics
or subjects can be placed on the same grid.

12. Our
Journey
Top right: People who know the information and are likely to apply it because they exhibit certainty in
their behaviour.
Bottom left: People who typically are only just starting a learning journey – they neither have the
knowledge or certainty.
Bottom right: People who do know their subjects, but are lacking in certainty, therefore may be less likely
to apply the knowledge, or perhaps more likely to follow others’ lead.
Top left: Possibly the most interesting of the 4 segments, and potentially the most harmful people to a
business. Those who are certain in what they know, but what they know is wrong. They may still be
working to outdated information and have not changed their understanding. In this segment, a targeted
activity of unlearning what they currently understand, and then learning the new information is more
applicable. These individuals could present a business risk, while adding the additional risk of influencing
those people who fall in the bottom half of the graph as they have greater confidence but in the wrong
information.

13. The Solution:
Compertum
Please note the data above is real, but has been anonymised.
This above diagram shows an overview of learner results in an aviation related assessment. The dots
represent people, the score can be found along the bottom, with the pass mark drawn in blue and the
behaviour rating on the left.
The individual highlighted, Bob, failed to meet the required score, however, he did exhibit certainty in his
behaviour, a measure not provided in other systems such as eLearning quizzes.
Considering a traditional classroom based situation, an Instructor may have seen that Bob was less open to
absorbing the latest knowledge, perhaps because he didn’t realise it was wrong.
If only the assessment score was considered, a similar failure score could be someone receptive but simply
struggling.
With the use of an instructor, or our solution, the option to undertake a specific unlearning action would be
identifiable, but in a normal assessment this difference would not be detected, potentially leading to
ineffective next steps, or an undetected risk of incorrect knowledge. In the case shown here the assessment
was pre-training, thereby avoiding pitfalls.

14. Compertum
example #2
Here is another example, showing a similar graph with individuals who have all passed the required grade,
but vary in knowledge certainty.
Highlighted are two individuals, both scoring well, but with extremes of certainty.
Without the measure of certainty, these two individuals would potentially be treated in exactly the same
way, with the same level of supervision or responsibility.
Again, an instructor would have noticed lack of certainty of the one individual and potentially the need to
reinforce the knowledge to avoid risk. Perhaps the individual being open to influence from others. Our
software solution Compertum aims to replicate that finding.
Of course, there’s no taking away from the result of the exam, but this gives the opportunity to provide
support, or make more of an opportunity with the strongest individuals.
This could also be applied to an interview scenario, allowing differentiation between high scorers.
So, how do we determine this measure?

15. Compertum
example #3
Here we see a portion of Compertum’s raw timeline view, before conditioning and comparison to the
population, but it gives enough of an indicator.
Here is Sahil, who was rated as passing the exam and with the highest level of certainty on the previous
chart.
The timeline shows different colours with grey corresponding to thinking time, green showing when a
correct answer is given, and red should an incorrect answer be given.
Sahil’s total time here is 3 minutes, so you can see how decisive he was.
If an instructor was watching over his shoulder, or giving these questions face-to-face, his behaviour
would provide a positive measure of certainty as Compertum shows above.

16. Let’s now consider the other person highlighted (Adam).
Again, high scoring, but with the lowest certainty measure.
Compertum
example #3
continued..

17. Here’s Adam’s timeline, with the same colour coding as Sahil’s raw data. It is worth noting here that
Adam’s overall timescale is much longer than Sahil’s at nearly 14 minutes.
In his timeline we see Adam changing his mind, altering previous answers, large amounts of hesitation
and revisiting questions.
If we’d have been looking over Adam’s shoulder, I’m sure we’d be less assured of his certainty in this
topic.
Further to this, we found by looking at his raw data at a subject bias level, we can find actionable data
(see next slide).
Compertum
example #3
continued..

18. Examples
When looking at Adam’s results by subject, there’s clearly an opportunity to focus action.
Here we have grouping by subject and we can see one area is the cause of his problems: ‘Pre-Flight’.
Focussed training here could shortcut traditional lengthy solutions and cut off a risk otherwise
masked.
With so many mechanisms for training available day-to-day, Compertum helps identify best routes
forward, that are the most effective and reduces risk.
We can also apply Compertum to considering the effectiveness of the subject as a whole i.e.
consistent behaviour of the group will indicate ambiguous or less effective training, or even wrong
information being provided.
Conventional systems do not show this level of analysis and often will be masked by average results.

19. Some Applications for Assessments
Compliance & Risk
End of Training
Supervision Requirements
Training Needs Analysis
Adaptive Learning
Recruitment
Progression Planning
We hope you enjoyed the presentation, if assessment is
something you are interested in and would like to find
out more about Compertum please get in touch by
email: contact@compertum.co.uk