Field studies as evaluation method for socio-technical interventions in Technology-Enhanced Learning. Much research in TEL is design work – i.e., the research team designs an intervention that is intended to support learning. This intervention needs to be evaluated to show the extent to which this goal has been reached; and to gain additional insights that are sought for. Field studies are one main type of evaluations. They are challenging to set up; and in case of a bad study design cannot be easily repeated due to the effort and cost of running a field study. The goal of this lecture and workshop is To provide a blueprint for field studies as evaluation method for socio-technical interventions in technology enhanced learning To present a hierarchical principle of evaluating learning interventions– based on Kirkpatrick & Kirkpatrick: Usage/observable activities – Learning – Impact on task/work performance – Impact on organization (in workplace learning/applicable to settings in which individual learning impacts a wider social entity) To have students plan a field study for their own PhD in rough lines individually To discuss their plans with peers and the lecturer, as well as other senior researchers who may be present – i.e., students will get feedback on their own plan The blueprint for field studies is to evaluate in a hierarchy of research questions/evaluation level: First, one assesses the observable (learning) activities that are carried out – in particular how and whether participants adhered to the prescribed intervention; this helps understand the success of the intervention and it is possible to identify problems. Second, one assesses concrete learning outcomes – insights that are generated. Thirdly, one assesses a change in behaviour, and fourthly a change in performance. In parallel, a mix of qualitative and quantitative methods should be used – this allows on the one hand statistical comparison (pre/post; between groups). On the other hand, one can get in depth explanatory insights.

1. 1
S C I E N C E n P A S S I O N n T E C H N O L O G Y
u www.tugraz.at
Shamelessly Off-Topic: Please participate in
study on “Reflecting on Time Management”:
http://tiny.cc/timeTool
What‘s in it for you? Reflect on your time management – and if you
have no need for THAT, do share your secret with everyone this week!

2. 2
S C I E N C E n P A S S I O N n T E C H N O L O G Y
u www.tugraz.at
Field Studies in Technology-
Enhanced Learning
June 3, 2019
Viktoria Pammer-Schindler, Institute for Interactive Systems and
Data Science @ TU Graz

3. 33
What Field Studies are for
- Positioning field studies within research process and
methodological universe
Viktoria Pammer-Schindler, Institute for Interactive Systems and Data Science

4. 44
I have developed this beautiful <thing>
(=artefact)...
Viktoria Pammer-Schindler, Institute for Interactive Systems and Data Science
What will it do to real
users in their real
environment?

5. 55
Question applicable IF
Methodology is to develop a solution for something,
and to combine research and impact on practice
§ Design Science Research
§ Action Research
Perspective on technology is to understand
technology in combination with people and their social
context/activities
§ Activity theory
§ Socio-technical systems theory
Viktoria Pammer-Schindler, Institute for Interactive Systems and Data Science

6. 66
Field Studies
Answer the question
§ Does the developed technology (=IT artefact)
§ WORK (=large parts of our discussion today will be
figuring out what this exactly means)
§ As part of a socio-technical intervention
§ In the selected environment
Viktoria Pammer-Schindler, Institute for Interactive Systems and Data Science

7. 77
When NOT to do a field study
Your technology is immature – field studies do not replace
good practice SW engineering like testing and using usability
heuristics
§ Unusable
§ Buggy
Your research question
§ Is technological – how fast/correct is the algorithm, etc.
§ In general isn‘t about people acting in complex
contexts
Viktoria Pammer-Schindler, Institute for Interactive Systems and Data Science

8. 88
Design Science Research – The Briefest of
Introductions
Viktoria Pammer-Schindler, Institute for Interactive Systems and Data Science

9. 99
Design Science Research Principles
Input to design:
§ Address a problem of practical relevance
§ Existing background knowledge
Design
§ Evaluate the artefact you design rigorously
§ Depending on intended contribution: analytical, case study,
experiment, field study, simulation
Output of design:
§ Contribute to environment of relevance - solves problem
§ Contribute to background knowledge
Viktoria Pammer-Schindler, Institute for Interactive Systems and Data Science

10. 1010
Activity Theory – The Briefest of Introductions
Descriptive theoretical framework that provides a structure to think about
human activities
Activity as central unit of analysis
Subject – an individual or group which have an
Object – an “objectified motive”, a goal – it motivates
the activity, and gives it direction
§ Subject and object uniquely identify the activity and
cannot be analysed on their own.

11. 1111
Tools mediate activities
§ Tools change the essential qualities of the activity
(mediation)

12. 1212
Subjects act in a social context
Subjects act within a community with shared objects.
§ The relationship between the subject and the
community is mediated by rules.
§ The relationship
between the
object and the
community is
mediated by
a division of labour,
and work structure.

13. 1313
A Process View: Field studies are part of a
design-oriented research process
Understand Design Evaluate
Viktoria Pammer-Schindler, Institute for Interactive Systems and Data Science
... which is in reality of course iterative and not linear

14. 1414
Understand – WHAT?
§ Domain
§ Target users and their context – people, technology,
organisations; activities
§ Domain knowledge
§ Background
§ Learning
§ Human-computer interaction / Interactive systems design
§ Related work: Which similar interventions have already
been experimented with
§ Methods
Viktoria Pammer-Schindler, Institute for Interactive Systems and Data Science
Understand

15. 1515
Design – WHAT?
Socio-technical intervention
§ Technology
§ Keep an eye open for interfaces, transitions to other
technologies in users’ environments
§ Practice
§ How should technology usage be embedded in target
users’ work practice? Who else needs to be informed/
work differently?
Viktoria Pammer-Schindler, Institute for Interactive Systems and Data Science
Design

16. 1616
§ Reaction
§ Learning
§ Behaviour
§ Results / Impact
Based on Kirkpatrick & Kirkpatrick’s (2010) four levels of evaluating
trainings in organisations
Ø It’s a hierarchical model – higher level categories necessitate “good”
results on the lower levels
Ø It allows for debugging and the identification of preliminary results
Ø Different levels can use evaluation criteria from different fields of
knowledge: HCI, learning sciences, organisational sciences
Viktoria Pammer-Schindler, Institute for Interactive Systems and Data Science
EvaluateEvaluate – WHAT?
Does the intervention “WORK”?
- Dimensions of evaluation

17. 1717
Reaction – How do users react to the
intervention?
§ Sympathy – Do you like it?
§ Usability – observations, scales such as system usability scale (SUS),
open questions
§ Perceived usefulness
§ Usage - Self-reports, log data
Rationale:
§ If users don’t use /comply with your intervention, it cannot have an
effect!
§ Helps interpret “weird” effects / results at other levels
§ Ease of use and perceived usefulness predict technology acceptance
(Venkatesh et al., 2003)
Viktoria Pammer-Schindler, Institute for Interactive Systems and Data Science

18. 1818
Learning
§ Assess gained knowledge and competencies
§ Ideal: good quality tests are available in the domain of learning
§ Observe
§ Self-reports, can also be self-reports of changes
Rationale:
§ Gained knowledge / competencies may be the goal of your
intervention
§ Without knowledge gain / insights, change of behaviour is unlikely (but
it is not strictly a pre-requisite)
§ Helps identify disconnect between conscious learning and
unconscious learning (behaviour)
Viktoria Pammer-Schindler, Institute for Interactive Systems and Data Science

19. 1919
Behaviour
§ Measure relevant aspects of behaviour
§ Ideal: Quantitative tools for measuring behaviour
§ Same as for learning: Observe, self-reports
Rationale:
§ Only changed behaviour can lead to a change in results of target
activity.
Challenge:
§ Behaviour is not easily changed in short-term interventions
Viktoria Pammer-Schindler, Institute for Interactive Systems and Data Science

20. 2020
Results / Impact
§ Measure results and impact of learning
§ Individual performance – e.g., individual sales, athlete
performance / victories
§ Organisational performance – e.g., mortality rates, project profit
margins
Rationale
§ Evaluates whether the right „things“ have been learned (w.r.t. context)
§ Helps identify the value of learning – towards learner, and other
stakeholders (who pay for instance)
Challenge:
§ Performance is typically caused by a multitude of contributing factors –
causality is hard to ascertain, especially in pre/post designs
Viktoria Pammer-Schindler, Institute for Interactive Systems and Data Science

21. 2121
Evaluate – HOW?
Methods
Mixed methods
§ Observe in person, and by collecting interaction data
§ Ask study participants, and stakeholders around study participants
§ Access existing data
§ Quantative data
§ self-assessments, knowledge tests, assessments by others,
organisational performance data
§ Qualitative data
§ open questions and free-form observations, interviews, field studies
Compare: Typical: within subjects – pre/post; between subjects –
intervention and control group
Viktoria Pammer-Schindler, Institute for Interactive Systems and Data Science

22. 2222
Recommended Reading / Literature
Underlying Presentation
Design Science Research
§ Hevner (2007) – A Three Cycle View of Design Science Research. Scandinavian Journal of
Information Systems, Vol. 19/2, p.87-92.
§ Hevner et al. (2004) – Design Science in Information Systems Research. MIS Quarterly, Vol 28/1, p.
75-105.
Activity Theory
§ Brief reading: https://viktoriapammer.wordpress.com/2015/08/19/activity-theory-101/
§ Kaptelinin & Nardi (2009) – Acting with Technology. Activity Theory and Interaction Design. MIT
Press.
Evaluation levels
§ Kirkpatrick & Kirkpatrick (2010) - Evaluating Trainings Programs - The Four Levels. 3rd Edition, Read
How You Want, 2010.
§ Example paper using Kirkpatrick & Kirkpatrick: Fessl et al. (2017) – In-app Reflection Guidance:
Lessons Learned across Four Field Trials at the Workplace. IEEE TLT, Vol. 10/4, p. 488-501.
Technology Acceptance Model
§ Venkatesh et al. (2003) - User Acceptance of Information Technology: Toward a Unified View. MIS
Quarterly, Vol. 27/3, p.425-478.
Viktoria Pammer-Schindler, Institute for Interactive Systems and Data Science

23. 2323
Do It Yourself: Design Your Own Field
Study
Viktoria Pammer-Schindler, Institute for Interactive Systems and Data Science

24. 2424
Step-by-Step
Reflect the research questions of your own PhD:
1. Is a field study relevant for you?
2. What problem of relevance do you address? Can you
argue the relevance well – do you have solid
understanding of your environment of relevance?
3. What theories / background knowledge do you use?
4. What is your expected contribution to these theories /
background knowledge?
5. How could you evaluate your artefact at the levels of
reaction, learning, behaviour (if applicable), results/
impact (if applicable)?
Viktoria Pammer-Schindler, Institute for Interactive Systems and Data Science