The document summarizes research on the Live Interest Meter (LIM) app, which collects feedback from audiences during presentations to support reflective learning. The LIM app gathers, aggregates, and visualizes audience feedback on an "interest meter." Experimental tests of the app at project meetings and university lectures showed it was technically satisfactory but acceptance was limited. A user study provided guidance to refine the use case and guide further development, showing openness to using quantified feedback for reflection but concerns about distraction. Ongoing work aims to improve usability, add reflection support after presentations, and evolve the approach to be more collaborative.

1. Live Interest Meter - Learning from Quantified
Feedback in Mass Lectures
INFORMATIK
FZI FORSCHUNGSZENTRUM
V. Rivera-Pelayo, J. Munk, V. Zacharias, and S. Braun
FZI Research Center for Information Technology, Karlsruhe, Germany
LAK Conference 2013 – Leuven, Belgium
10th April 2013

2. Agenda
 Introduction
 Use Case and Motivation
 Live Interest Meter App
 Tests and User Study
 Conclusions
 Ongoing Work
12.04.2013 © FZI Forschungszentrum Informatik 2

3. Use Case and Motivation
Holding a good
presentation is hard!
12.04.2013 © FZI Forschungszentrum Informatik 3

4. Use Case and Motivation
How to get better at it?
• With practice
• Get feedback
Holding a good
• Learn from the feedback!
presentation is hard!
12.04.2013 © FZI Forschungszentrum Informatik 4

5. Use Case and Motivation
 LA for workplace learning
 Capturing of feedback for reflective learning
 Inspired by Quantified Self approaches
 Mass lectures, virtual courses and conferences
12.04.2013 © FZI Forschungszentrum Informatik 5

6. Use Case and Motivation
 LA for workplace learning
 Capturing of feedback for reflective learning
 Inspired by Quantified Self approaches
 Mass lectures, virtual courses and conferences
12.04.2013 © FZI Forschungszentrum Informatik 6

7. Live Interest meter - LIM App
 Gathering, aggregation and visualization of feedback
 Protagonist: the meter
 General
performance
 Speech speed
 Comprehension
12.04.2013 © FZI Forschungszentrum Informatik 7

8. LIM App: Evolution graph
 Available for Android and for Browsers (JavaScript)
 Browser to facilitate the configuration of the presenter
12.04.2013 © FZI Forschungszentrum Informatik 8

9. LIM App: Polls and Questions
 Polling and anonymous rated questions
12.04.2013 © FZI Forschungszentrum Informatik 9

10. LIM App: JavaScript Version
12.04.2013 © FZI Forschungszentrum Informatik 10

11. Experimental tests
 (1) Project meeting
 10 participants used LIM
 Discussion driven
12.04.2013 © FZI Forschungszentrum Informatik 11

12. Experimental tests
 (1) Project meeting
 10 participants used LIM
 Discussion driven
 (2) Lecture at university
Technical results very satisfactory
 15 participants
 Interactive session Acceptance results rather limited
12.04.2013 © FZI Forschungszentrum Informatik 12

13. Experimental tests
 Project meeting
 10 participants used LIM
 Discussion driven
 Lecture at university
Technical results very satisfactory
 15 participants
 Interactive session Acceptance results rather limited
Presenter‘s role well defined Suitable for big audiences
12.04.2013 © FZI Forschungszentrum Informatik 13

14. User Study
 Main goals
 Refine use case
 Guide further development
 In which scenarios and how can the quantification of feedback performed
by the LIM App support reflective learning?
 Which features are more appreciated by users, both presenters and
audience members?
12.04.2013 © FZI Forschungszentrum Informatik 14

15. User Study
 20 qualitative interviews
 Willing to substitute traditional questionnaires for captured feedback
 Anonymity appreciated (for feedback and questions)
  Polls but  Chat
 Reflection needs time – preferred after the session
 Online survey
 1 month (May-June 2012)
 120 participants Presenter
 87 valid responses Audience 44,82%
55,18%
12.04.2013 © FZI Forschungszentrum Informatik 15

16. User Study – Reaction to feedback
What do you consider realistic regarding
how you can react to the feedback (as presenter)?
12.04.2013 © FZI Forschungszentrum Informatik 16

17. User Study – Data collection
64,37 %
Data collected live and continuously is
better and more significant than only periodically collected data.
12.04.2013 © FZI Forschungszentrum Informatik 17

18. User Study – Information needed
Which information
would the presenters
like to know about
and which information
does the audience
want to evaluate?
12.04.2013 © FZI Forschungszentrum Informatik 18

19. User Study – Characteristics of the LIM App
How participants evaluated the LIM App features
12.04.2013 © FZI Forschungszentrum Informatik 19

20. Conclusion
78,16 % found the idea behind 57,47 % would like to test the
the LIM App very positive LIM App
 Readiness of presenters to learn retrospectively
 Motivate students to use the application
 Main concern is distraction
Applying Learning Analytics to informal workplace learning and
data gathered during the learning process
12.04.2013 © FZI Forschungszentrum Informatik 20

21. Towards a second prototype
 Usability improvements
 Support for reflection after the presentation
 From presenter to collaborative approach e.g. topics
 Mock-ups to adapt the results to real features
12.04.2013 © FZI Forschungszentrum Informatik 21

22. Any questions?
THANK YOU!
rivera@fzi.de
@veronicarp
vriverapelayo
12.04.2013 © FZI Forschungszentrum Informatik 22

23. Theoretical Model & Related Work
12.04.2013 © FZI Forschungszentrum Informatik 23

24. Theoretical background
[1] Applying Quantified Self Approaches to Support Reflective Learning. Verónica Rivera-Pelayo, Valentin Zacharias, Lars Müller,
Simone Braun. Learning Analytics and Knowledge 2012 (LAK 2012), Vancouver, Canada
[2] A Framework for Applying Quantified Self Approaches to Support Reflective Learning. Verónica Rivera-Pelayo, Valentin
Zacharias, Lars Müller, Simone Braun. IADIS International Conference on Mobile Learning (Mlearning 2012), Berlin, Germany
12.04.2013 © FZI Forschungszentrum Informatik 24

25. Related Work
 Clickers and feedback tools
 Polling
 Student outcomes and comprehension
 Student attendance and interest on the course
 Reflective Learning in TEL
 Student’s perspective
 Data from LMS, blogs or software used
 LIM App
 Improve lecturer/presenter’s professional performance
 Students give feedback and contribute to it
 Enrich lecture
 Feedback contextualized by polls, questions and chat
 Gathering of data during the lecture
4/12/2013 © FZI Forschungszentrum Informatik 25

26. ARS and Clickers
 [1] M. Akbari, G. Böhm, and U. Schroeder. Enabling communication and feedback in mass
lectures. In ICALT, pages 254{258, 2010.
 [2] M. Bonn, S. Dieter, and H. Schmeck. Kooperationstools der Notebook Universität Karlsruhe
(TH). In Mobiles Lernen und Forschen, pages 63-71. Klaus David, Lutz Wegener (Hrsg.),
November 2003.
 [3] J. E. Caldwell. Clickers in the large classroom: Current research and Best-Practice tips. CBE
Life SciEduc, 6(1):9-20, Mar. 2007.
 [4] D. Duncan and E. Mazur. Clickers in the Classroom: How to Enhance Science Teaching Using
Classroom Response Systems. Pearson Education, 2005.
 [5] J. Hadersberger, A. Pohl, and F. Bry. Discerning actuality in backstage - comprehensible
contextual aging. In EC-TEL, volume 7563 of Lecture Notes in Computer Science, pages 126-
139. Springer, 2012.
 [6] D. Kundisch, P. Herrmann, M. Whittaker, M. Beutner, G. Fels, J. Magenheim, W. Reinhardt, M.
Sievers, and A. Zoyke. Designing a Web-Based Application to Support Peer Instruction for Very
Large Groups. In ICIS '12, Research in Progress, Orlando, USA, December 2012.
 [7] G. Rubner. mbclick - an electronic voting system that returns individual feedback. In WMUTE,
pages 221-222. IEEE, 2012.
 [8] A. Wessels, S. Fries, H. Horz, N. Scheele, and W. Effelsberg. Interactive lectures: Effective
teaching and learning in lectures using wireless networks. Comput. Hum. Behav., 23(5):2524-
2537, Sept. 2007.
4/12/2013 © FZI Forschungszentrum Informatik 26

27. Reflective learning in TEL
 [1] R. Ferguson, S. B. Shum, and R. D. Crick. EnquiryBlogger: using widgets to support
awareness and reflection in a PLE Setting. In ARPLE11, PLE Conference 2011, Southampton,
UK, 11-13 July, 2011.
 [2] S. Govaerts, K. Verbert, E. Duval, and A. Pardo. The student activity meter for awareness and
self-reflection. In CHI '12 Extended Abstracts on Human Factors in Computing Systems, pages
869-884, New York, NY, USA, 2012. ACM.
 [3] J. L. Santos, S. Govaerts, K. Verbert, and E. Duval. Goal-oriented visualizations of activity
tracking: a case study with engineering students. In 2nd International Conference on Learning
Analytics and Knowledge, LAK '12, pages 143-152, New York, NY, USA, 2012. ACM.
4/12/2013 © FZI Forschungszentrum Informatik 27