Can Confidence Assessment Enhance Traditional Multiple-Choice Testing?

Martin Ebner
Martin EbnerHead of Educational Technology at Graz University of Technology
Can Confidence Assessment
   Enhance Traditional
 Multiple-Choice Testing?


Josef Kolbitsch, Martin Ebner, Walther Nagler, Nicolai Scerbakov
Can Confidence Assessment Enhance Traditional Multiple-Choice Testing?
Technology enhanced Learning




http://flickr.com/photos/powerbooktrance/289992273//
uLearning




http://www.flickr.com/photos/sovietuk/141381675
„Students sometimes accomplish their results
 of a Mutiple-Choice Test by a combination of
    partial knowledge and guesswork.“
Confidence Assessment



        „Students do not only have to choose an
      answer to a multiple-choice question, but also
       have to indicate how confident they feel
               about their given answer.“
additional confidence paramter
Lecture Design


         - lecture „Application of Operating and
           Information Systems“
         - recommended to all students of Graz
           University of Technology
         - 720 students attend
         - 432 did the final exam
Test Design

              100 questions were generated with
               following influence by confidence:
Interpretation of the parameter


                                            Points if Points if
   Confidence        Interpretation
                                            correct incrorrect

     100%      I know it                      10         -5

               I am not sure, but I think
      66%                                      7         -3
               I know it

               I am not sure, but I
      33%                                      3         -2
               assume something

               I did not know it and I
       0%                                      0         0
               have to guess
Test Groups

        - Control group (260 students)
        - Experimental group with Confidence
          (172 students)
        - Experimental group without
          Confidence (172 students)
        - Experimental group scaled (172
          students)
Overview about groups


                              Minimum Maximum   Range of
        Type of Test
                               Score   Score     Points

  Experimental group with
                                -50     100       150
        Confidence

     Experimental group
                                0       100       100
     without Confidence

  Experimental group scaled     0       100       100


       Control group            0       100       100
General findings

                               $!!quot;#


                                ,!quot;#


                                +!quot;#
!quot;#$quot;%&'(quot;)*+),'-./0/)1$*#quot;)




                                *!quot;#


                                )!quot;#

                                                                                      :72;9<2#=90>3.0?.#
                                (!quot;#                                                  :72;#=90>3.0?.#
                                                                                      :72;#=90>3.0?.@#1?/A.3#

                                '!quot;#                                                  =90249A#B49<C#



                                &!quot;#


                                %!quot;#


                                $!quot;#


                                 !quot;#
                                       -./0#   12/03/43#5.67/890#   -.37/0#   -93.#
Is there a difference between the results of the MCT with
confidence assessment and the result of the Control
Group?


                           )!quot;!!#$


                                                /012$345678598$
                           (%quot;!!#$
                                                /012$345678598quot;$:9;<87$

                           (!quot;!!#$              /0124=1$345678598$

                                                3451>4<$?>4=@$
 !quot;#$quot;%&'(quot;)*+),&-.quot;%&/)




                           '%quot;!!#$


                           '!quot;!!#$


                           &%quot;!!#$


                           &!quot;!!#$


                            %quot;!!#$


                            !quot;!!#$
                                     !*&!$#$   &&*'!$#$      '&*(!$#$     (&*)!$#$      )&*%!$#$     %&*+!$#$       +&*,!$#$   ,&*-!$#$   -&*.!$#$   .&*&!!$#$
                                                                                     !quot;#$quot;%&'(quot;)*+)0'123-3),$*#quot;)
Is there a difference between the scaled results of MCT
with confidence assessment and the result of the Control
Group?


                           +!quot;!!#$


                                               ./01$234567487$
                           *!quot;!!#$
                                               ./01$234567487quot;$98:;76$

                                               ./013<0$234567487$
                           )!quot;!!#$
                                               2340=3;$>=3<?$
 !quot;#$quot;%&'(quot;)*+),&-.quot;%&/)




                           (!quot;!!#$



                           '!quot;!!#$



                           &!quot;!!#$



                           %!quot;!!#$



                            !quot;!!#$
                                     !,&!$#$                        &%,(!$#$             (%,*!$#$             *%,-!$#$   -%,%!!$#$
                                                                               !quot;#$quot;%&'(quot;)*+)0'123-3),$*#quot;)
Is there a relationship between the confidence parameter
and the correctness of the answer?

                                      $!!quot;#


                                       ,!quot;#


                                       +!quot;#


                                       *!quot;#
   !quot;#$quot;%&'(quot;)*+)/%$*##quot;$&),%-.quot;#-)
       !quot;#$quot;%&'(quot;)*+),%-.quot;#-)




                                       )!quot;#


                                       (!quot;#


                                       '!quot;#


                                       &!quot;#


                                       %!quot;#                                             -./012/32#456728#


                                       $!quot;#                                             9/3.::23;#</8=2:8#


                                        !quot;#
                                              $!!quot;#   ))quot;#                       &&quot;#            !quot;#
                                                             0*%12quot;%$quot;)3'45quot;-)
Discussion

      - Result of Experimental Group with
        Confidence are slightly worse than
        those of Control Group
      - Result of Experimental Group scaled are
        slightly better than those of Control
        Group
      - Relationship between confidence
        parameter and percentage of wrong
        answers are highly relevant
„By adding a
  confidence
   parameter
     to a MCT
    the result
  of the MCT
deteriorates“
SOCIAL LEARNING

Computer and Information Services
  Graz University of Technology

            Graz University of Technology




           Martin Ebner
    http://elearning.tugraz.at
  http://elearningblog.tugraz.at
1 of 19

More Related Content

Similar to Can Confidence Assessment Enhance Traditional Multiple-Choice Testing?

The Project TrapThe Project Trap
The Project TrapMichael Griffin
261 views75 slides
Munne ki laashMunne ki laash
Munne ki laashsunninews92
24 views20 slides

Similar to Can Confidence Assessment Enhance Traditional Multiple-Choice Testing?(20)

The Project TrapThe Project Trap
The Project Trap
Michael Griffin261 views
Device deploymentDevice deployment
Device deployment
Angelo van der Sijpt3.6K views
Munne ki laashMunne ki laash
Munne ki laash
sunninews9224 views
All about Apache ACEAll about Apache ACE
All about Apache ACE
OSGi User Group France1.6K views
Connectix webserverConnectix webserver
Connectix webserver
steveheer110 views
Outsourcing 3.0: the agile wayOutsourcing 3.0: the agile way
Outsourcing 3.0: the agile way
Alexey Krivitsky1.7K views
Methods of proper work outMethods of proper work out
Methods of proper work out
navedgaur786239 views
sam pressosam presso
sam presso
Choeffner293 views
07 samyagan07 samyagan
07 samyagan
guestdeb794238 views
Technology That WorksTechnology That Works
Technology That Works
Tolbert Elementary361 views
WALA Tutorial at PLDI 2010WALA Tutorial at PLDI 2010
WALA Tutorial at PLDI 2010
Julian Dolby3.4K views
Keynote - SBIKeynote - SBI
Keynote - SBI
smeech377 views
Dangerous GoodsDangerous Goods
Dangerous Goods
globexwwc546 views

More from Martin Ebner(20)

Maker EducationMaker Education
Maker Education
Martin Ebner306 views
School Start Screening ToolSchool Start Screening Tool
School Start Screening Tool
Martin Ebner339 views
MOOC map (Version 3)MOOC map (Version 3)
MOOC map (Version 3)
Martin Ebner276 views
ReDesign your lecture Canvas [eng]ReDesign your lecture Canvas [eng]
ReDesign your lecture Canvas [eng]
Martin Ebner217 views
ReDesign your lecture Canvas [de]ReDesign your lecture Canvas [de]
ReDesign your lecture Canvas [de]
Martin Ebner171 views
MOOC-Maker Canvas [eng]MOOC-Maker Canvas [eng]
MOOC-Maker Canvas [eng]
Martin Ebner684 views
MOOC-Maker Canvas [de]MOOC-Maker Canvas [de]
MOOC-Maker Canvas [de]
Martin Ebner1.3K views

Recently uploaded(20)

ICS3211_lecture 08_2023.pdfICS3211_lecture 08_2023.pdf
ICS3211_lecture 08_2023.pdf
Vanessa Camilleri68 views
Psychology KS5Psychology KS5
Psychology KS5
WestHatch53 views
Azure DevOps Pipeline setup for Mule APIs #36Azure DevOps Pipeline setup for Mule APIs #36
Azure DevOps Pipeline setup for Mule APIs #36
MysoreMuleSoftMeetup75 views
Dance KS5 BreakdownDance KS5 Breakdown
Dance KS5 Breakdown
WestHatch52 views
BYSC infopack.pdfBYSC infopack.pdf
BYSC infopack.pdf
Fundacja Rozwoju Społeczeństwa Przedsiębiorczego144 views
Nico Baumbach IMR Media ComponentNico Baumbach IMR Media Component
Nico Baumbach IMR Media Component
InMediaRes1186 views
2022 CAPE Merit List 2023 2022 CAPE Merit List 2023
2022 CAPE Merit List 2023
Caribbean Examinations Council3K views
Chemistry of sex hormones.pptxChemistry of sex hormones.pptx
Chemistry of sex hormones.pptx
RAJ K. MAURYA97 views
discussion post.pdfdiscussion post.pdf
discussion post.pdf
jessemercerail70 views
Education and Diversity.pptxEducation and Diversity.pptx
Education and Diversity.pptx
DrHafizKosar56 views
Psychology KS4Psychology KS4
Psychology KS4
WestHatch52 views
Sociology KS5Sociology KS5
Sociology KS5
WestHatch50 views
Drama KS5 BreakdownDrama KS5 Breakdown
Drama KS5 Breakdown
WestHatch50 views
ACTIVITY BOOK key water sports.pptxACTIVITY BOOK key water sports.pptx
ACTIVITY BOOK key water sports.pptx
Mar Caston Palacio132 views

Can Confidence Assessment Enhance Traditional Multiple-Choice Testing?

  • 1. Can Confidence Assessment Enhance Traditional Multiple-Choice Testing? Josef Kolbitsch, Martin Ebner, Walther Nagler, Nicolai Scerbakov
  • 5. „Students sometimes accomplish their results of a Mutiple-Choice Test by a combination of partial knowledge and guesswork.“
  • 6. Confidence Assessment „Students do not only have to choose an answer to a multiple-choice question, but also have to indicate how confident they feel about their given answer.“
  • 8. Lecture Design - lecture „Application of Operating and Information Systems“ - recommended to all students of Graz University of Technology - 720 students attend - 432 did the final exam
  • 9. Test Design 100 questions were generated with following influence by confidence:
  • 10. Interpretation of the parameter Points if Points if Confidence Interpretation correct incrorrect 100% I know it 10 -5 I am not sure, but I think 66% 7 -3 I know it I am not sure, but I 33% 3 -2 assume something I did not know it and I 0% 0 0 have to guess
  • 11. Test Groups - Control group (260 students) - Experimental group with Confidence (172 students) - Experimental group without Confidence (172 students) - Experimental group scaled (172 students)
  • 12. Overview about groups Minimum Maximum Range of Type of Test Score Score Points Experimental group with -50 100 150 Confidence Experimental group 0 100 100 without Confidence Experimental group scaled 0 100 100 Control group 0 100 100
  • 13. General findings $!!quot;# ,!quot;# +!quot;# !quot;#$quot;%&'(quot;)*+),'-./0/)1$*#quot;) *!quot;# )!quot;# :72;9<2#=90>3.0?.# (!quot;# :72;#=90>3.0?.# :72;#=90>3.0?.@#1?/A.3# '!quot;# =90249A#B49<C# &!quot;# %!quot;# $!quot;# !quot;# -./0# 12/03/43#5.67/890# -.37/0# -93.#
  • 14. Is there a difference between the results of the MCT with confidence assessment and the result of the Control Group? )!quot;!!#$ /012$345678598$ (%quot;!!#$ /012$345678598quot;$:9;<87$ (!quot;!!#$ /0124=1$345678598$ 3451>4<$?>4=@$ !quot;#$quot;%&'(quot;)*+),&-.quot;%&/) '%quot;!!#$ '!quot;!!#$ &%quot;!!#$ &!quot;!!#$ %quot;!!#$ !quot;!!#$ !*&!$#$ &&*'!$#$ '&*(!$#$ (&*)!$#$ )&*%!$#$ %&*+!$#$ +&*,!$#$ ,&*-!$#$ -&*.!$#$ .&*&!!$#$ !quot;#$quot;%&'(quot;)*+)0'123-3),$*#quot;)
  • 15. Is there a difference between the scaled results of MCT with confidence assessment and the result of the Control Group? +!quot;!!#$ ./01$234567487$ *!quot;!!#$ ./01$234567487quot;$98:;76$ ./013<0$234567487$ )!quot;!!#$ 2340=3;$>=3<?$ !quot;#$quot;%&'(quot;)*+),&-.quot;%&/) (!quot;!!#$ '!quot;!!#$ &!quot;!!#$ %!quot;!!#$ !quot;!!#$ !,&!$#$ &%,(!$#$ (%,*!$#$ *%,-!$#$ -%,%!!$#$ !quot;#$quot;%&'(quot;)*+)0'123-3),$*#quot;)
  • 16. Is there a relationship between the confidence parameter and the correctness of the answer? $!!quot;# ,!quot;# +!quot;# *!quot;# !quot;#$quot;%&'(quot;)*+)/%$*##quot;$&),%-.quot;#-) !quot;#$quot;%&'(quot;)*+),%-.quot;#-) )!quot;# (!quot;# '!quot;# &!quot;# %!quot;# -./012/32#456728# $!quot;# 9/3.::23;#</8=2:8# !quot;# $!!quot;# ))quot;# &&quot;# !quot;# 0*%12quot;%$quot;)3'45quot;-)
  • 17. Discussion - Result of Experimental Group with Confidence are slightly worse than those of Control Group - Result of Experimental Group scaled are slightly better than those of Control Group - Relationship between confidence parameter and percentage of wrong answers are highly relevant
  • 18. „By adding a confidence parameter to a MCT the result of the MCT deteriorates“
  • 19. SOCIAL LEARNING Computer and Information Services Graz University of Technology Graz University of Technology Martin Ebner http://elearning.tugraz.at http://elearningblog.tugraz.at