Presentation at the Designing Human Technologies 5.0 doctoral course, 8 May 2017, Valgeranna, Estonia.

1. Presenting constructive
design research as a
doctoral dissertation
Hans Põldoja

2. Hans Põldoja
Head of studies, lecturer of educational technology
Tallinn University, School of Digital Technologies
Education:
Aalto University, School of Arts, Design and Architecture (2016)
Tallinn Pedagogical University (2003)
hans.poldoja@tlu.ee
http://www.hanspoldoja.net

3. Põldoja, H. (2016). The Structure and Components
for the Open Education Ecosystem: Constructive
Design Research of Online Learning Tools. Helsinki:
Aalto University.

4. 4727 days

5. Five design cases

6. PILOT

7. LeMill

8. EduFeedr

9. LeContract
Age: 26
Education: Master student
Occupation: librarian
Maria
Maria has studied information science and now she is doing her
Masterʼs studies in interactive media. At the same time she has a full
time job as a school librarian. Therefore she is interested in combining
school assignments with her work as much as possible. At the same
time she is a self-directed learner who likes to go in depth in topics that
are interesting for her.
Goals:
Personalization: “It is hard to have a full time job and be a master
student at the same time. If possible, then I try to choose assignments
that can be connected with my work.”
Scaffolding: “I feel that often it is difficult to specify all the resources
and actions that I have to make in order to achieve my learning
objectives. Good examples from other learners help me to refine my
contract.”
Awareness: “It was good that we had to review our learning
contracts. This way I was constantly aware of my objectives and
thinking about the strategy to achieve my goals.”
Photo by Alessandro Valli,
taken from http://www.flickr.com/photos/liquene/4435467897/

10. DigiMina

11. How to connect these five cases?

12. Open education ecosystem

14. Formulating and reformulating
research questions…

15. Research questions
• What are the main design challenges related to the open
education ecosystem?
• What are the design patterns used in designing online
learning tools and services for the open education ecosystem?
• What kind of structure and components are needed to create
the open education ecosystem?

16. Research design and methodology

17. Methodological considerations

18. (Leinonen, Toikkanen, & Silfvast, 2008)

19. Interaction design methods
• Personas
• Scenario-based design
• Concept mapping
• User stories
• Paper prototyping
• Wireframes
• …

20. Contextual Inquiry
Personas
Participatory Design
Scenario-based
design
Participatory design
sessions
Concept mapping
Product Design
User stories
Paper prototyping
Information
architecture
High-fidelity
prototyping
Software Prototype
As Hypothesis
Agile sprints
Software prototype
(adapted from Leinonen et al., 2008)

21. Cooper, A., Reimann, R., Cronin, D, & Noessel,
C. (2014). About Face: The Essentials of
Interaction Design. Indianapolis, IN: John Wiley
& Sons, Inc.

22. Age: 58
Education: PhD
Occupation: professor
Jakob
Jakob is a professor of industrial design. He has been working in the
university for almost 30 years. As a designer and researcher he has
participated in many R&D projects but his real love is teaching.
In last few years Jakob has decided to spend less time on projects and
travelling. This means that he has more time for preparing his courses
and trying out different things. He has experimented with blogs and
social software. In his last course he decided to try personal learning
contracts. It took quite a lot of time but he hopes that learning
contracts will help him to give a more personal learning experience for
his students.
Goals:
Adjusting the course: “I would like to know what the students expect
to learn in my course. Then I can adjust the course assignments
according to their needs.”
Personalized support: “I want to give more personalized guidance
and support for my students.”
Fair grading: “I donʼt like grading but in some courses I am required to
do it. I have found that learning contracts help me on fair grading. I rely
on learning contracts when I negotiate grades with my students.”
Photo by Jesse Courtemanche,
taken from http://www.flickr.com/photos/jesse757/4170721132/

23. Carroll, J. M. (2000). Making Use: Scenario-Based
Design of Human-Computer Interactions.
Cambridge, MA: The MIT Press.

24. Scenario 1: First experience with EduFeedr
John is teaching an open online course where he has more than 30 participants.
All the participants have their individual blogs where they publish the weekly
assignment. John is using a feed reader to follow all the student blogs. He is
also trying to comment all the posts that have an inspiring ideas.
In the middle of the course John notices that it becomes increasingly
complicated to manage the course. Several participants are not able to keep up
with the tempo of the course. In the feed reader it is not easy to see how far
different participants have proceeded with the course.
One day John reads about new feed reader EduFeedr that has special features to
support online courses. It an online feed reader similar to Google Reader. John
creates an account and starts exploring the possibilities. He can easily import
all the feeds from his current feed reader.
After importing the feeds he notices that the students’ posts are somehow
grouped by the assignments. This way it is easy to see how far the participants
have proceeded with their work.
It is possible to browse students posts by a tag cloud. Among other tags there
is a tag "urgent". John clicks on the tag and finds out that a few students who
needed fast feedback to proceed with their home task have used that tag.
There is also an image that displays the social network between the student
blogs. John can see which blogs are more actively linked and commented.
John is impressed by these possibilities. He decides to get a cup of coffee and
explore the other features of EduFeedr.

26. Novak, J. D. (2010). Learning, Creating, and Using
Knowledge: Concept Maps as Facilitative Tools in
Schools and Corporations. New York, NY:
Routledge.

28. Cohn, M. (2004). User stories applied: For agile
software development. Boston, MA: Addison-
Wesley.

30. Snyder, C. (2003). Paper prototyping: The fast and
easy way to design and refine user interfaces. San
Francisco, CA: Morgan Kaufmann.

32. Logo
Languages
Search
Content Methods Tools Community
Log in / out links
Featured content cover
image
Featured content cover
image
Featured content cover
image
Featured content cover
image
Title Title Title Title
Featured contentNew content
Footer
My content: drafts, published,
collaborations, collections
Browse content

33. Koskinen, I., Zimmerman, J., Binder, T., Redström, J.,
& Wensveen, S. (2011). Design Research Through
Practice: From the Lab, Field, and Showroom.
Waltham, MA: Morgan Kaufmann.

34. Constructive design research
• New knowledge is developed through constructing actual
design artifacts such as products, systems, spaces or media
• Addresses limitations of earlier approaches such as user-
centered design
• Mockups and prototypes that help people to open up their
imagination

35. Fallman, D. (2008). The Interaction Design Research
Triangle of Design Practice, Design Studies, and
Design Exploration. Design Issues, 24(3), 4–18.
http://doi.org/10.1162/desi.2008.24.3.4
Design Issues: Volume 24, Number 3 Summer 20084
The Interaction Design Research
Triangle of Design Practice, Design
Studies, and Design Exploration
Daniel Fallman
1. Introduction
Interaction design takes a holistic view of the relationship between
designed artifacts, those that are exposed to these artifacts, and the
social, cultural, and business context in which the meeting takes
place. While there is no commonly agreed definition of interaction
design, its core can be found in an orientation towards shaping
digital artifacts—products, services, and spaces—with particular
attention paid to the qualities of the user experience.1
To be able to
deal with user experience—including physical, sensual, cognitive,
emotional, and aesthetical issues; the relationship between form,
function, and content; as well as fuzzy concepts such as fun and
playability—a number of recent efforts have been made in the direc-
tion of establishing a better understanding of the role of the user
experience in interactive systems design.2
Unlike the Human-Computer Interaction (HCI) community
for instance, interaction design fully recognizes itself as a “design
discipline” in that its ultimate objective is to create new and change
existing interactive systems for the better.3
There is a current plethora
of departments, groups, and multidisciplinary labs dealing with
interaction design that have their origins in such diverse places as
computer science, HCI, anthropology, industrial design, informat-
ics, and applied physics and electronics. Adding to the disciplinary
confusion, each group typically also is configured as a multidisci-
plinary team.
Since the field of interaction design currently is growing
rapidly in scope as well as importance,4
both within academia and
industry, there is an increasing need to also expand, further develop,
and professionalize interaction design research. Refined models of
interaction design research; embracing both what it currently is as
well as pointing toward what it could be, arguably would be very
useful tools in this process.
In this paper, we will introduce a model of interaction
design research that has evolved at the Umeå Institute of Design,
Umeå University, in Sweden in recent years, and which currently
is guiding our interaction design research efforts as well as our
Ph.D. education. Thinking about interaction design research in the
way proposed by the model has helped us to keep up what we see
1 Jonas Löwgren, “How Far beyond
Human-Computer Interaction Is
Interaction Design?” Digital Creativity
13:3 (2002): 186–192; and Terry
Winograd, “From Computing Machinery
to Interaction Design” in Beyond
Calculation: The Next Fifty Years of
Computing, Peter J. Denning and Robert
Metcalfe, eds. (New York: Springer-
Verlag, 1997), 149–162.
2 Lauralee Alben, “Quality of Experience:
Defining the Criteria for Effective
Interaction Design,” Interactions 3:
3 (1996): 11; Jodi Forlizzi and Katja
Battarbee, “Understanding Experience
in Interactive Systems,” Proceedings of
the Conference on Designing Interactive
Systems (2004); and John McCarthy and
Peter Wright, Technology as Experience
(Cambridge, MA: MIT Press, 2004).
3 Daniel Fallman, “Design-Oriented
Human-Computer Interaction,”
Proceedings of Human Factors in
Computing Systems Conference (2003):
225–132.
4 John Zimmerman, Jodi Forlizzi, and
Shelley Evenson, “Taxonomy for
Extracting Design Knowledge from
Research Conducted during Design
Cases,” Proceedings of Futureground
(2004).
© 2008 Massachusetts Institute of Technology
Design Issues: Volume 24, Number 3 Summer 2008

36. (Fallman, 2008)
Design Issues: Volume 24, Number 3 Summer 2008 5
Science, Anthropology, Sociology, Philosophy, and so on. The basic
structure of our model is visualized as a triangle.
Commercial design
organizations
Other
disciplines
Philosophy
Idealistic, Societal, and
Subversive
Design critique, Art,
Humanities
Cumulative, Distancing,
and Describing
Context driven,
particular, and synthetic
Design Studies
Design
Exploration
Design Practice
The model of interaction design research
in its most basic form.

37. (based on Fallman, 2008)
Design Practice
Design
Exploration
Design Studies
Tools
Educational practices
Openness
1
2
3
4
5
1
2
3
4
5
PILOT
LeMill
EduFeedr
LeContract
DigiMina
Design cases

38. (based on Fallman, 2008)
Design Practice
Design
Exploration
Design Studies
Tools
Design patterns
Digital ecosystem
Educational practices
Openness
1
2
3
4
5
1
2
3
4
5
PILOT
LeMill
EduFeedr
LeContract
DigiMina
Design cases

39. Results

40. Design challenges
• 9 design challenges for the open educational resources
• 8 design challenges for blog-based open online courses
• 5 design challenges for assessment and recognition of
competencies
• Classified as pedagogical (8) socio-cultural (6) and technical
(8)

41. Challenge 1: Digital learning resources are mainly used
for individual learning and for presentations
In many cases digital learning resources are used by
students for individual learning (reading, looking,
playing, quizzes) or by teachers in their classes
(presentations). It is a challenge to design OER tools
and services that guide teachers away from the
acquisition of knowledge paradigm to the participation
and knowledge creation paradigms (Paavola et al., 2004).

42. Challenge 4: Lack of collaboration and peer production of
learning materials
European teachers are not used to sharing their learning
resources with other teachers. Often teachers think that
their resources are not good enough for sharing in
public. Also, teachers are worried about copyright
issues. Some teachers would need external motivation to
share their resources. Publishing a learning resource in
the repository is an extra step that is often missed
because of lack of time. There is always a threshold for
joining an online community and starting to collaborate
with other people. Most of the learning object
repositories are designed for searching and publishing
resources, not for collaboration.

43. Challenge 7: Providing localization and reusability while
retaining authentic context
Localizing learning resources does not mean simply
translating the content from one language to another. It
is important that the learning resources provide
authentic context for the target group. In the PILOT
project, it was a challenge to design a template
structure that would allow flexibility in localization,
so that the teacher could decide which textual content
and media elements should be edited or replaced in the
localization process. From the technical perspective,
localization is also related to versioning of learning
resources.

44. Design patterns
• 12 design patterns for collaborative authoring of open
educational resources (LeMill and PILOT)
• 12 design patterns for blog-based open online courses
(EduFeedr and LeContract)

45. Alexander, C., Ishikawa, S., & Silverstein, M.
(1977). A Pattern Language: Towns, Buildings,
Construction. New York, NY: Oxford University
Press.

46. (Alexander et al., 1977)

47. (Alexander et al., 1977)

48. Approaches for identifying design patterns
• Inductive (from specifics to generalizations) and deductive (from
generalizations to specifics) approaches (Baggetun, Rusman, &
Poggi, 2004)
• Using IMS Learning Design to detect patterns in existing courses
(Brouns et al., 2005)
• Framework for identifying design patterns for e-learning systems
(Retalis, Georgiakakis, & Dimitriadis, 2007)
• Layer-based architectural approach (Gibbons, 2014)

49. Patterns for collaborative
authoring of OER’s

50. Pattern 1: Authoring template
This pattern deals with providing a clear structure for creating
new learning resources.
It may be difficult to start creating a new learning resource from
the scratch. Having a certain predefined structure for new learning
resources would help teachers to get started. A large collection of
peer produced learning resources would benefit from having a consistent
structure and layout. Consistent structure contributes to the quality
of learning resources. On the other hand, it is important to achieve
balance between predefined structure and flexibility for the authors.
Therefore: The learning resource authoring tool should provide a
set of pedagogical templates that scaffold teachers and content
producers in creating new resources. LeMill provided six pedagogical
templates for creating learning resources: web page, presentation,
exercise, lesson plan, school project, and PILOT. Web page is a generic
template while other templates provide a more predefined structure.
Authoring templates consist of different types of sections that are
called blocks in LeMill. For example, web pages in LeMill consist of
text blocks, media pieces and embed blocks. The exercise template has
additional blocks for various question types. Templates may also
scaffold the use of new pedagogical methods, such as the PILOT template
in LeMill.
This is a central design pattern, that is related to a number of
smaller design patterns. Learning resources based on authoring
templates have a DRAFT (2) status, support EMBEDDING (3) and LINKEDNESS
(4), are published under a SINGLE LICENSE (5), and could be developed
into TRANSLATIONS (6) or ADAPTATIONS (7). Two special types of
authoring templates are METHOD DESCRIPTIONS (8) and TOOL DESCRIPTIONS
(9). As a central design pattern, authoring template is addressing a
number of design challenges: (C3) assuring the quality of
collaboratively created open educational resources; (C4) lack of
collaboration and peer production of learning materials, (C5) lack of
reuse, revising and remixing, and (C2) scaffolding the use of new
pedagogical methods.
Short description
Conflicting forces
Recommended
configuration
Related patterns
and design
challenges

51. Design challenges and
patterns for collaborative
authoring of OER’s
C1 Digital learning resources are
mainly used for individual
learning and for presentations
C2 Scaffolding the use of new
pedagogical methods
C3 Assuring the quality of
collaboratively created open
educational resources
C4 Lack of collaboration and
peer production of learning
materials
C5 Lack of reuse, revising and
remixing
C6 Multilingualism
C7 Providing localization and
reusability while retaining
authentic context
C8 Limited findability and poor
usability
C9 Poor use of the underlying
principles of the Web
P1 Authoring template
P2 Draft
P3 Embedding
P4 Linkedness
P5 Single license
P6 Translations
P7 Adaptations
P8 Method descriptions
P9 Tool descriptions
P10 Collection
P11 Teaching and learning story
P12 Featured resources
Design challenges Design patterns

52. Patterns for blog-based
open online courses

53. Design challenges and
patterns for blog-based
open online courses
C10 Supporting learners with
setting up their personal learning
goals and strategies
C11 Keeping the learner
motivation throughout the course
C12 The danger of over-
scripting
C13 Establishing and keeping
the community gravity
C14 The fragmentation of
discussions in blog-based
courses
C15 Lack of coordination
structures for managing blog-
based courses
C16 Lack of awareness support
mechanisms
C17 Commenting and versioning
of learning contracts
P13 Being open for lurking
P14 Open enrollment
P18 Blogroll
P19 Course tag
P23 Learning analytics
visualizations
P15 Nicknames
P16 About page
P17 Personal learning contract
P20 Aggregated discussions
P21 Reflective assignments
P22 Summary posts
P24 Open badges for
assessment
Design challenges Design patterns

54. Open education ecosystem
For this dissertation, the open education ecosystem is defined
as a learning ecosystem that consists of tools, services,
resources and stakeholders who share a common set of values.
The core value that defines the extent of the open education
ecosystem is openness.

55. (based on Gütl & Chang, 2008)
Open education
ecosystem

56. Learning stakeholders of
the open education
ecosystem

57. Learning utilities of the
open education
ecosystem

58. Discussion

59. Discussion
• Theoretical and practical implications: design of open
educational tools and services
• (In)validity and (un)reliability: strict replication of research
procedures
• Limitations of the study: more validation is needed
• Recommendations for further research: combining personal
learning contracts, self- and peer-assessment, and open badges

60. Fallman, D., & Stolterman, E. (2010). Establishing
criteria of rigour and relevance in interaction design
research. Digital Creativity, 21(4), 265–272. http://
doi.org/10.1080/14626268.2010.548869
Establishing criteria of
rigour and relevance in
interaction design
research
Daniel Fallmana
and Erik Stoltermanb
a
Department of Informatics, Umea˚ University, Sweden
b
School of Informatics and Computing, Indiana University, USA
dfallman@informatik.umu.se; estolter@indiana.edu
Abstract
Interaction design research is a unique discipline embra-
cing practicing professionals, design educators and aca-
demic researchers, but attracting attention from people
with different backgrounds and interests has lead to
the question of what constitutes ‘good research’. What
is rigorous and relevant interaction design research
and how do we recognise and evaluate it? Most
current attempts at dealing with these issues tend to be
on loan from other disciplines, and may overlook,
conceal, or knowingly exclude some of what makes
interaction design research such a unique field. Our
primary contribution is that what may be perceived as
three different design research activities—design prac-
tice, design exploration and design studies—have their
own purposes, intended outcomes, and internal logic.
Each approach must be examined in its own right and
the notions of rigour and relevance have to be based
on a firm understanding of the particular purpose of
each approach.
Keywords: rigour and relevance, interaction design
research, design practice, design exploration, design
studies
1. Introduction
Interaction design research has rapidly evolved
into a unique, thriving discipline embracing
practising professionals, design educators and aca-
demic researchers. As with many evolving disci-
plines, attracting attention and effort from a large
number of people with different backgrounds,
interests and ways of seeing also tends to cause
what can perhaps best be described as ‘disciplinary
anxiety’. Where are we going, what is the core of
the field, what are relevant research questions,
what are appropriate methods? Questions like
these inevitably lead to the more general subject:
what constitutes ‘good research’ and how do we
recognise and evaluate it? What is rigorous and
relevant interaction design research?
This paper attempts to commence such a dis-
cussion by looking in some detail at the concepts
of rigour and relevance.
1.1. Disciplinary anxiety
Disciplinary anxiety can be experienced in a field
when heterogeneous ways of doing research lead
to diverse assumptions about what constitutes
legitimate research. We find this to be the case in
interaction design research today, where notions
of legitimacy are being thrown around without
being paid enough attention. Often, these notions
tend implicitly or explicitly to be on loan from
Digital Creativity
2010, Vol. 21, No. 4, pp. 265–272
ISSN 1462-6268 # 2010 Taylor & Francis
DOI: 10.1080/14626268.2010.548869
http://www.informaworld.com
Downloadedby[Aalto-yliopistonkirjasto]at03:0523January2016

61. References
• Baggetun, R., Rusman, E., & Poggi, C. (2004). Design Patterns For Collaborative Learning: From Practice To Theory And
Back. In L. Cantoni & C. McLoughlin (Eds.), Proceedings of EdMedia World Conference on Educational Media and
Technology (pp. 2493–2498). AACE.
• Brouns, F., Koper, R., Manderveld, J., Van Bruggen, J., Sloep, P., van Rosmalen, P., et al. (2005). A first exploration of an
inductive analysis approach for detecting learning design patterns. Journal of Interactive Media in Education, 2005(3), 1–
10.
• Gibbons, A. S. (2014). An Architectural Approach to Instructional Design. New York, NY: Routledge.
• Gütl, C., & Chang, V. (2008). Ecosystem-based Theoretical Models for Learning in Environments of the 21st Century.
International Journal of Emerging Technologies in Learning, 3(3), 50–60. http://doi.org/10.3991/ijet.v3i1.742
• Leinonen, T., Toikkanen, T., & Silfvast, K. (2008). Software as Hypothesis: Research-Based Design Methodology. In
Proceedings of the Tenth Anniversary Conference on Participatory Design 2008 (pp. 61–70). Indianapolis, IN: Indiana
University.
• Retalis, S., Georgiakakis, P., & Dimitriadis, Y. (2007). Eliciting design patterns for e- learning systems. Computer Science
Education, 16(2), 105–118. http://doi.org/10.1080/08993400600773323

62. This work is licensed under the Creative Commons Attribution-ShareAlike 4.0
International License. To view a copy of this license, visit
http://creativecommons.org/licenses/by-sa/4.0/
Hans Põldoja
hans.poldoja@tlu.ee
Tallinn University
School of Digital Technologies
http://www.hanspoldoja.net
https://www.slideshare.net/hanspoldoja