This document summarizes a seminar on research methodologies in technology enhanced learning. It discusses various topics related to doctoral research including defining research questions, literature reviews, methodologies, evaluation processes, research communities and paradigms. Examples from past student theses are provided to demonstrate mixed methods approaches, iterative design processes, and lessons learned regarding defining problems, collecting and analyzing data, and engaging with peer review.

1. Research methodologies in
Technology Enhanced Learning
Prof. Yannis Dimitriadis
GSIC/EMIC research group
University of Valladolid
May 12, 2015
Univ. of Patras,
Patras, Greece
Doctoral student seminar

2. Seminar overview
2
 Doctoral student seminar
 Research issues in doctoral theses and
publications in the e-learning community
 Revision of GSIC/EMIC artifacts
 Theses, publications, etc.
 Lifecycle (proposals, submissions, reviews, etc.).
 Discussion of methodological issues:
 multi-disciplinary field
 evaluation/validation process
 Analysis of a sample of participants’ artifacts
 Lessons learnt and recommendations

3. Theory and practice in research
 Research as
– Art and/or systematic approach
– Tools, instruments, theory to support research
– Good practices
– High dependence on context of research field
 Research communities and Kuhn’s theory on
scientific revolutions
– Paradigm, textbooks and puzzles
 Let’s try to connect theory and practice for
novel research practitioners
3

4. Research communities
 Local vs. global level
– The need for a productive interaction
 Individual vs. global perspective
– The researcher as an individual
 The role of peer review in research communities
– Why citations are so important!
 Types of research artifacts and forae
– Journals, conferences, workshops
– Proposals and final artifacts
– Science in action according to Latour
4

5. An exercise with the TEL field
 A matter of terminology?
– “TEL”, “New technologies in education” , “e-
learning”
– Where is the focus?
– What are the instruments for research?
 Define a concept map for TEL field and
community
– Actors, trends, methods
– Static and dynamic view
– Think of an example and associated literature
– Connect with your own (even limited) experience
5

6. Few GSIC/EMIC examples
 Material available at Dropbox:
– https://www.dropbox.com/sh/rfp8z0itsdunich/AABezAXWd-
ipGc2yCCvjjW_-a?dl=0
 The life of a paper: From ICALT 2004 to ETS 2005
– Seminar
 Davinia Hernández (2007)
– Thesis (proposed and submitted)
 Luis Pablo Prieto (2012)
– Thesis (submitted)
 Juan Alberto Muñoz (2015)
– Thesis (submitted)
6

7. The life of a paper:
From ICALT 2004 to ETS 2005
 Bad reviews are still bad, even if they are
positive and lead to a best paper award:
– Reviewer2: Fine
– Reviewer3: Very interesting, thank you!
 Some conference papers may lead to journal
special issues
– No guarantee for an automatic paper “upgrade”
 A good review process is probably the best
research dialogue on a topic
7

8. The life of a paper:
From ICALT 2004 to ETS 2005
 Some interesting reviewer notes:
– “but this proposal has not been implemented yet
and, consequently, has not been empirically
evaluated”
– “The theoretical foundations and related theories
need to be weaved into their discussion…”
– “They make a number of claims but …. they do
not demonstrate the feasibility and utility of their
approach from the point of view of educators”
8

9. D. Hernández (2004-2007)
Thesis diagram
9

10. L.P. Prieto (2009-2012)
Thesis diagram
10

11. L.P. Prieto
Engineering method + mixed methods
11

12. L.P. Prieto
Iterations and connections
12

13. L.P. Prieto
The EREM framework and instrument
13

14. L.P. Prieto
RQ1: Orchestration Framework
14http://prezi.com/aa2vighak7hh/orchestration-in-tel-cscl-as-easy-as-53/

15. L.P. Prieto
RQ1 Framework: Engineering method
15

16. L.P. Prieto
RQ1: EREM diagram
16

17. L.P. Prieto
RQ1: Anticipated reduction diagram
17

18. L.P. Prieto
RQ1: Partial conclusions
18

19. L.P. Prieto
RQ1: Findings
19

20. L.P. Prieto
RQ1: Global conclusions
20

21. L.P. Prieto
RQ2 Glue!PS: The proposal
21

22. L.P. Prieto
RQ2 Glue!PS: Engineering method
22

23. L.P. Prieto
RQ2 Glue!PS: Anticipated reduction
23

24. L.P. Prieto
RQ2 Glue!PS: Partial conclusions
24

25. L.P. Prieto
RQ2 Glue!PS: EREM diagram
25

26. L.P. Prieto
RQ2 Glue!PS: Mixed methods diagram
26

27. L.P. Prieto
RQ2 Glue!PS: Example of data sources
27

28. L.P. Prieto
RQ2 Glue!PS: Some findings
28

29. L.P. Prieto
RQ2 Glue!PS: Partial conclusions
29

30. L.P. Prieto
RQ2 Glue!PS: Global conclusions
30

31. M.J. Rodríguez (2010-2014)
Thesis diagram
31

32. M.J. Rodríguez
Evolution of DBR (Design Based research)
32

33. M.J. Rodríguez
Diagram of exploratory phase
33

34. M.J. Rodríguez
Data sources in exploratory phase
34

35. M.J. Rodríguez
Evaluation design (EREM diagram)
35

36. J.A. Muñoz:
Some notes on methodology (I)
 Pay special attention to Section 1.2 of thesis:
– It contains a very comprehensive account of the main
research issues
– “Those were my assumptions about the world and the
knowledge. In other words, those were my ontological and
epistemological believes.”
– “I considered problematic to assume the objectivity and
the independence of the researcher and the phenomenon
explored”
– Finally, the research approach followed … is interpretive,
since I consider it matches well with the phenomenon
investigated (the orchestration problem for teachers … )
and its context (persons and organizations of persons)36

37. J.A. Muñoz:
Some notes on methodology (II)
 EREM - anticipatory data reduction schemes
– “We considered EREM very appropriate due to the
guidance and help that the framework provides to novel
evaluators … Examples are the different conceptual tools,
such as an evaluation design diagram, or a multimedia
collaborative report”
– “anticipatory data reduction process to explore the
research question, creating a schema of “research
question – issue – topics – informative questions”. Thus,
we defined issues as the main conceptual organizers of
the evaluation process. Such issues were divided into
more concrete topics to help us understand the different
dimensions within the issues. ... each topic was explored
through various informative questions” 37

38. J.A. Muñoz:
Some notes on methodology (III)
 Anticipatory data reduction and coding issues
– anticipatory data reduction schema was used as an
initial category tree for the process of coding, therefore
using a dominantly deductive a-priory approach for the
creation of an initial set of codes
– we consider a single-coder sufficient, given the
interpretive nature of the study, the dominantly deductive
approach (with a predetermined and agreed initial
category tree), and the fact that we do not aim to measure,
quantify, obtain statistical significant results or find
relations between variables, but rather make sense of the
participants perceptions and experiences
38

39. J.A. Muñoz:
Some notes on methodology (IV)
 Interpretive research and its strategies
– in interpretive research, the notions of reliability,
internal/external validity and objectivity used in positivist
approaches to assure the quality of a research are
replaced with the notions of dependability, credibility,
transferability and confirmability
– The strategies to comply with these criteria from an
interpretive perspective are very different from the ones
used to ensure quality in a positivist research, and include
(among others) long permanence in the field, use of
multiple data gathering techniques, use of deep
descriptions, member checking, etc.
39

40. J.A. Muñoz:
Some notes on methodology (V)
 Some strategies:
– prolonged engagement during months of work with the different
teachers and persistent observation in the field;
– member checking, obtaining feedback from the informants about the
data and the interpretations;
– acknowledgement of participant opinions, by interviewing the
teachers and by analyzing teachers’ and students’ reflections;
– integration of the thorough collaborative observation reports in a
single portfolio, thus enabling a thick description of the phenomenon
under scrutiny, reported in detail to the whole evaluation team;
– peer review within the evaluation team to avoid bias;
– exploration of the systems in different educational contexts;
– triangulation of data source, methods and researchers to cross-
check data and interpretations.
40

41. Some lessons learnt (I)
 Think well of the publication forum
 Consider the community paradigm
 Define well and clearly the research question
 Think of originality, feasibility, relevance
 Do not “forget” the literature
 Provide sound methodological “position”
 Synthesize results (tables, diagrams)
 Try out peer review
 Check for plagiarism and acknowledgements41

42. Some lessons learnt (II)
 Use visual research instruments (thesis
diagram, phases of DBR, etc.)
 Define approach and techniques
 Use them well (e.g. triangulate)
 A system by itself is not enough
 Collect empirical data and synthesize well
 Interact with the community, use social
events
 The “hill metaphor” and “Ithaki”
42