Using technology to create new types of questions is important, but also exploring new types of test representations or analyzing the most adequate technological device to support a specific assessment activity based on tests. In this presentation Patricia will explain her PhD research focused on Computing-based Testing (CBT). She will introduce her theoretical approaches but also exemplary implementations and experiments to illustrate what new CBT scenarios means. After introducing the conceptual domain, the first scenario shows how question-items can be enriched with specific multimedia information (web maps). The second scenario illustrates another particular case of the CBT domain where a test can be augmented with real physical elements (assessment in real situ) using mobile devices. And finally, the third scenario describes an implementation that combines advanced question-items and tests using 3D virtual worlds (assessment in virtual situ).

1. Using technology to support
assessment in the XXI century
Patricia Santos Rodriguez
Research Associate in Technology Enhanced
Learning – BRILLE – FP7 IP Learning Layers
Year: 2013

2. Outline
1. Background
2. PhD Thesis
3. Implementations and Experiments
4. Conclusions and future work
Presentation - Patricia Santos 2

3. Background
Professional and Research
experience

4. 1 Background
• Patricia Santos has got a:
• Computer Engineering degree (2007)
• Master of Information, Communication and Media Technologies (2008)
• PhD in Information, Communication and Media Technologies (2011), focused on
the research areas of TEL and Human Computer Interaction (HCI)
• In 2011 Patricia presented her PhD thesis Computing-based Testing: conceptual
model, implementations and experiments extending IMS QTI, obtaining the
qualification of Summa cum Laude.
 Award: Best Phd Thesis (ICT) Transfer of knowledge Entity: Awards Social Council -UPF
Year: 2012
• Since 2007, collaborating with the GTI research group (http://gti.upf.edu) within the
Department of Information and Communication Technologies at the Universitat Pompeu
Fabra.
Presentation - Patricia Santos 4

5. 1 Background
Professional Experience
• Involved in European and national R&D projects in the field of TEL. Main projects:
• TENCompetence: developing and using infrastructure to support individuals, groups and
organizations in lifelong competence development.
• Assessment methods for competence development.
• Learn3 project: this project organizes the spectrum of TEL along three main lines towards:
3D Learning, Learning 3.0, 3rd place Learning. The aim was to study and propose new
learning scenarios that can have a profound impact on the learning experience.
• Enriched questions, assessment in real situ and assessment in virtual situ
5
Presentation - Patricia Santos

6. PhD Thesis
“Computing-based Testing”

7. 2 PhD Thesis
Context and motivation
Technology Enhanced Learning
Computing-Based Testing
Different researchers claim that the use of ICT enables the design of new
assessment solutions, allowing the creation of more complex scenarios
based on tests
CHALLENGE: Creating advanced assessment scenarios should
involve the use of technology to perform test-activities which would
be impossible or very expensive to reproduce using traditional
methods of assessment
GLOBAL OBJECTIVE: To propose and evaluate a conceptual framework and associated
models for the Computing-based Testing domain to facilitate the design and
implementation of advanced CBT scenarios and tools, extending the scope of the IMS
Question and Test Interoperability (an educational technology standard)
Presentation - Patricia Santos 7

8. 2 PhD Thesis
Context, objectives and contributions
CONTEXT InnovativeTechnology
Assessment Technology Supporting Assessment
Education CAA Specifications
Patterns & Models
Objectives
To create and evaluate
To model the Computing-based Testing advanced CBT implementations To analyze the suitability of
conceptual domain for facilitating the and scenarios for supporting the IMS QTI for computationally
sharing of information, the design and assessment of higher order representing advanced CBT
the implementation of advanced CBT skills (extending the IMS QTI scenarios
scenarios and systems standard)
Contributions
2. New scenarios of CBT using 3. Extension paths for QTI to
1. Modelling of the ICTs for assessing higher-order
Conceptual Domain for support advanced CBT
skills scenarios. Demonstration of
Computing-based Testing
its interoperability with other
languages and systems
Main Contribution: Santos, P. Hernández-Leo, D. Perez-Sanagustin, M. & Blat, (2012) Modelling the
Computing Based Testing domain extending IMS QTI: a Framework, models and exemplary implementations.
Computers and Human Behaviour, 28 (5), 1648-1662. (ISI-JCR Q1) Impact factor: 1.865
Presentation - Patricia Santos 8

9. 2 PhD Thesis: Modeling the CBT domain
CBT Framework
• The aim of the CBT framework is to facilitate a comprehensive representation of the
elements and the relationships which have to be considered for designing advanced CBT
scenarios
PhD Presentation - Patricia Santos 9

10. 2 PhD Thesis: Modeling the CBT domain
CBT Models
• Models for computationally represent the aproaches proposed
PhD Presentation - Patricia Santos 10

11. 2 PhD Thesis
Computing-based Testing
Simple Tests
Assessment in real situ: physical world as
interactive context of tests and questions
Assessment in virtual situ: 3D worlds as
interactive context of tests and questions
Enriched questions and interactions
Presentation - Patricia Santos 11

12. Examples
Implementations & Experiments

13. 3 Implementations and Experiments
Research Experience
• Especially focused on requirements analysis, design, implementation and evaluation of
research pilots and experiments in real educational and lifelong learning contexts.
• Formal educational contexts: experiments with secondary and university teachers and
students.
• Informal educational contexts: adult lifelong learners (with special needs and
cognitive disabilities) and work-based contexts.
13
Presentation - Patricia Santos

14. 3 Implementations and Experiments
Research Methodology
Multidisciplinary problem domain:
• Hybrid RM: Software Engineering RM + Social Context
• Four Phases: Informational, Propositional, Analytical, Evaluation
• Evaluation phase:
• Mixed Method: Analysis of Quantitative + Qualitative data
• Usability tests, Contextual inquiry, Co-Design, Questionnaires, Focus groups...
• Triangulation: comparative analysis from differerent data sources
• Case Study: experiments grouped and analyzed following the case study method according
to specific research questions
14
Presentation - Patricia Santos

15. 3 Enriched representation of questions
QTIMaps: Educational context
• Assessment of geography information and spatial skills
Teachers
CBT Framework
CBT PIMs
Assessment at classroom GTI team
Participatory design
(Schuler & Namioka, 1993)
“ identify, localize and analyze
geographical elements at different
QTIMaps PSM
scales” and “search, select, understand
and relate (...) implementation
gMapsQTItest cartographic information
from different sources: books, media,
information technologies “
Geography & History K-12 spanish curriculum
Presentation - Patricia Santos 15

16. 3 Enriched representation of questions
QTIMaps - Implementation
• The implementation, (gMapsQTItest, 2010) enables us to test the educational benefits
and usability of using web maps as interactive contexts for representing closed
questions
edit publish
QTI-
Questionnaire
GoogleMaps
Editor
Teacher
interpret
save
newAPIS Renderize Demo
Data-Base QTI engine & compute
Visualize &
interact
Middleware
GoogleMaps
API
Students
PhD Presentation - Patricia Santos 16
Presentation - Patricia Santos

17. 3 Enriched representation of questions
QTIMaps – Case study
• Main experiment: IES DUC de Montblanc case, 23
students between 14 and 16 years and their teacher.
Design a gMapsQTI test with 8 questions/interactions.
Authentic assessment context
• Smaller experiment: Science Week at UPF (mini case),
23 students from different schools visited an exhibition.
Test with 8 questions/Interactions.
• Data: questionnaires, observations, automatic data &
interviews
Findings
1. The students are interested in web map applications using
them to explore geographical information
2. The approach seems to provide significant learning benefits
if it is used as formative or summative assessment tool
3. Students learn quickly how to use the different interactions.
The difficulties were on the content of the questions and
not related to the interactions.
4.QTIMaps PSM is useful for implementing assessment
activities for assessing higher-order skills in Geography
Presentation - Patricia Santos 17

18. 3 Assessment in real situ
QuesTInSitu – Educational context
• Assessing the students’ skills in situ (situated learning)
Teachers
CBT Framework
CBT PIMs
Assessment in (real) situ GTI team
Participatory design
(Schuler & Namioka, 1993)
“Students need to be located in situ in
the correct place because they will need
QuesTInSitu PSM
to interact with the environment
(observing, touching, talking with people,
QuesTInSitu implementation
etc.) in order to understand the question
and answer it.“
Presentation - Patricia Santos 18

19. 3 Assessment in situ
QuesTInSitu system and mobile app
• The interaction context of the test had to be the city a natural park... The questions have to
be associated to real positions and distributed as a route in the physical environment
“Assessment in situ” refers to a type of
activity where the questions of a test
have to be answered in front of a related
real location (in situ) *.
Teachers can create geolocated QTI
questions and tests (routes), and
students answer the tests using a
mobile device with GPS and 3G.
* Main Contribution: Santos, P., Pérez-Sanagustín, M., Hernández-Leo, D. & Blat, J. (2011). QuesTInSitu: From tests to routes
for assessment in situ activities. Computers & Education, 57 (4), 2517-2534. (ISI-JCR Q1) Impact factor: 2.617
Presentation - Patricia Santos 19

20. 3 Various experiments in diverse subject matters
and educational levels
Assessing Botany in situ A literature adventure
Discovering Barcelona!
Barca
The teacher created a route A group ofBarca
senior
Bachelor students
where the university learners (~65 years old),
explored the city observing
students had to answer members of a literature
and interacting with the
questions in situ observing group, created two routes
architecture and the street
the Barcelona botany with the aim of proposing
furniture with the aim of
garden, and finding, questions about facts of
putting in practice
touching and measuring a literature novel set in a
urbanism and history
specific plants. district of Barcelona.
skills.
Presentation - Patricia Santos 20

21. 3 Assessment in real situ
QuesTInSitu – Case study
Examples (videos of the pilots):
(1) Learning History Art in situ , a m-learning activity with 62 secondary students and 2
teachers http://bit.ly/GUW5vf (starting at 2′ 30”, in Catalan language)
(1) A literature adventure in situ, design and implementation of m-learning routes with older
learners http://www.youtube.com/watch?v=bBKvz9hYQ-k (english annotations)
Findings
1. QuesTInSitu supports successfully an assessment in situ activity thanks to
functionalities such as the monitoring service or the provision of feedback in real
time/situ.
2. Educational benefits: dynamic and practical activity; put in practice of transversal
(collaborative and communicative) and specific skills (botany, art, history,
geography...).
3. The GPS location-based system makes the process of finding locations faster and
helps the students in their exploratory tasks
4. The assessment in situ activity had a notable positive impact in the
attitude/motivation of the students and teachers
PhD Presentation - Patricia Santos 21
Presentation - Patricia Santos

22. 3 Assessment in virtual situ
Wonderland-QTI – Educational context
• This case shows how a virtual world is an adequate interaction context to create advanced
tests and question-items to assess Literature and language skills
GAST team
CBT Framework
CBT PIMs
GTI team
Assessment in (virtual) situ Participatory design
(Schuler & Namioka, 1993)
“When students learn a new
Wonderland-QTI PSMthey have to read and
language,
write correctly, but also they have
Wonderland-QTI third party implementation
to know how to respond to real
circumstances (not only talking but
also doing actions)”
Presentation - Patricia Santos 22

23. 3 Assessment in virtual situ
Wonderland-QTI - Implementation
• Wonderland-QTI checks the students’ actions in the
virtual world
• Tests are represented by the interaction context of the
virtual world
• Students interact with the virtual world in order to answer
the questions
• The Wonderland-QTI PSM captures the elements for
implementing assessment in virtual situ activities
PhD Presentation - Patricia Santos 23
Presentation - Patricia Santos

24. Conclusions &
Future work

25. 4 Conclusions and Future Work
Future Research Directions
Modeling the CBT domain
• Models to capture the main characteristics that advanced Question-Items,
Test and assessment activity dimensions.
• These models are extensions of the QTI Information Model
Advanced CBT implementations and scenarios
• The scenarios and systems implemented in these experiments represent
exemplary contributions for the CBT field and QTI
Future research directions
•Designing and implementing other advanced CBT scenarios and systems.
•Including CBT patterns in an authoring tool to guide teachers to select and
design adequate tests, question-items and activities considering their needs
•To study the inclusion of CBT activities in a flow of blended learning activities.
Presentation - Patricia Santos 25

26. 4 Conclusions and Future Work
New version of QuesTInSitu
• QueTInSitu: The Game: a new version of the QuesTInSitu application has been
developed as a feasible approach to computationally represent and support the
creation of gamified in situ learning activities that consider the puzzle-based
game metaphor. (Melero, Santos et al, submitted)
Video: http://www.youtube.com/watch?v=YkvoXSh9UZU
Presentation - Patricia Santos 26

27. 4 Conclusions and Future Work
Related publications, projects and awards
• Publications
 3 international journal papers (ETS11, C&E11, CHB12)
 2 book chapters (Investigations of e-learning patterns-IGI,Practical design patterns for teaching and
learning with technology)
 International conference papers (ICALT09, INCOS09, INCOS10, IE2010, EC-TEL-TEFA10, IE10, E-
iEd11, MUE12...)
 Posters (IICIDU11, EC-TEL12)
• Projects
European projects: TENCompetence
Spanish Ministry of Education and Science projects: Learn3
QTIMaps selected as one of its top best practices for assessment : ICOPER project
• Awards
IMS award was received in the category of “Most Innovative New Realization of Standards” - by the
contribution done in the TENCompetence Learning Design Toolkit
BDigital Awards for Digital Innovation 2011 - QuesTInSitu nomination
 CAA 2010: Best Paper Award
• Workshops & conferences (invited)
V Jornadas e-Learning Castilla y León 2011 - QuesTInSitu
 Jornada Aprenentatge 3.0 2011 - QuesTInSItu workshop with teachers
27
Presentation - Patricia Santos

28. Questions?
Thank you!
Patricia Santos Rodríguez
@: patricia.santos.rdz@gmail.com
@: Patricia.Santosrodriguez@uwe.ac.uk
LinkedIn: http://www.linkedin.com/pub/patricia-santos-
rodriguez/54/207/a97