Remote Collaborative Multi-User Informal Learning Experiences: Design and Evaluation


Published on

Arroyo, E.; Righi, V.; Tarrago, R.; Santos, P.; Hernández-Leo, D.; Blat, J.; Remote Collaborative Multi-user Informal Learning Experiences: Design and Evaluation. In: Proceedings of 6th European Conference on Technology Enhanced Learning, EC-TEL 2011, Palermo, Italy, September 2011, pp. 43-56.

This paper presents a customizable system used to develop a
collaborative multi-user problem solving game. It addresses the increasing
demand for appealing informal learning experiences in museum-like settings.
The system facilitates remote collaboration by allowing groups of learners to
communicate through a videoconferencing system and by allowing them to
simultaneously interact through a shared multi-touch interactive surface. A user
study with 20 user groups indicates that the game facilitates collaboration
between local and remote groups of learners. The videoconference and multitouch
surface acted as communication channels, attracted students’ interest,
facilitated engagement, and promoted inter- and intra-group collaboration—
favoring intra-group collaboration. Our findings suggest that augmenting
videoconferencing systems with a shared multitouch space offers new
possibilities and scenarios for remote collaborative environments and
collaborative learning.

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Two remote museums, a shared installation.This project was developed within the context of two Spanish museums that exhibitedan art collection simultaneously in Figueres and Barcelona cities.The unique characteristics of this use-case directed the application to become an additional channel for museumvisitors to explore the exhibition and increase the sense of connectedness between twospaces. All the while, allowing museum visitors to interact and collaborate.
  • in Figueres and Barcelona cities.Toy museum
  • Arts Santa Mònica, housed in an exceptional building on the Ramblain Barcelona, is a space of convergence and crossover between the different disciplines of contemporary artistic creation and science, thought and communication.Attentive to the cultural mutations and social changes accompanying the transition to the knowledge society, and working in conjunction with universities and academic institutions, research institutes and centres for the production and presentation of art, science and communication, Arts Santa Mònica generates ideas, projects, research and materials that stimulate dialogue betweenthe local and all it has to offer and the global dimension of society today.
  • Located in Figueres, Spain,It provides an introduction to new points of view and serves as a means to reflect on a very familiar subject matter for both children and adults: toys and games.The toys are grouped together by subject area and also include photographs from different time periods of children and their toys so that visitors can see how children played in years past. Each toy also includes an explanatory note (name, date, manufacturer, origin, etc.) and, in some cases, a note written by well-known authors reflecting on specific toys and games.A visit to our Museum allows visitors to be exposed to numerous considerations, encouraging them to reflect on them during or after their visit:· The past, present and the future based on the evolution of games and toys (before and after industrialisation, with the appearance of new materials, depending on society’s evolution and life conditions, etc.);· Facts and phenomena related to the environment and science, for example, in the “Optical Illusions” section or by seeing the multiple applications of scientific developments to toys; and· The aesthetic traits of the toys and games themselves, bearing in mind the different periods, countries of origin or if the toys and games were aimed at boys or girls.
  • The exhibition is fruit of the generous donation by Philippe and DorothéeSelz to the Museu del Joguet de Catalunya of the collection of a series of collections that their parents, Françoise and Guy Selz, put together over half a century (1925-1976) and that they themselves have gone on adding to.
  • For examples, some of these objects include: OCELLSAquestgrup de set ocellsestanfets de fustapintadaamb colors vius, ratlles, punts, senefesÖCada un delsocellsest‡ colïlocat a sobredíunapetitaplataformaquadrada, fetqueindicaunacondiciÛest‡ticai, per tant, probablementdecorativa. Elsocellsestanfets a Bombai, a la Õndia.DIABLEEl diableest‡ realitzatambcartrÛi paper maixÈsobrefilferroiest‡ fet a Oaxaca, un dels 31 estats de MËxic. Com jasabeu, un diableÈs un esperitsobrenaturalque, origin‡riament, designava un esperit molt savi, perÚ no pas malËfic. Per influËnciareligiosa (sobretotpeljudaisme, cristianismeiislam), el diableÈsentËs com un ‡ngelqueesvarebelïlar contra DÈuivaseguir a Lluciferi, per aixÚ, va ser expulsatdelscels cap a l'Infern. ElsdiablessÛnesperitsquehanesdevingutmalignesique se suposaquemiren de temptarals humans. A Oaxaca el diableÈsunafiguraben present, jaque la confessiÛ de femajorit‡riaÈs el catolicisme.
  • Working with real stake holders, That is working with museum curators, with administrators provide real constrains and requirements
  • Open-ended scenarios Stakeholders did not agree on a single scenario. Envisioned scenarios included treasure hunting activities, collaborative puzzles, interactivevideoconferencing and performances, collaborative music composition, collaborative tagging, expertise sharing, collaboratively exploring high definitions images, amongothersHowever, despite this open-endedness, we were restricted by fixed contentFixed content: The toy museum has their main focus on toys, and they wanted to center the installation around that topic. In particular, a collections of toys which is so big, that it is rarely presented entirely and they need to rotate the exhibition. Also having another remote museum showcasing the objects in the installation, helps the reach a bigger exposure.The Toy Museum of Catalonia requested for the experience to be a game (one major design constrain) as they receive frequentguided school visits and students had expressed their interest in playing while at themuseum (expected from a toy museum).Thus, it was necessary to develop an agile flexible system to provide the means for enacting different types of interactive learning games and experiences withdifferent user flows and interface behaviors. Finally, coordinated activities in both museums, groups of students visiting the Arts Santa Monica museum in Barcelona and the Toy Museum of Catalonia in Figueres are invited to play a game with other groups of students also visiting a museum in a different city.All stakeholders agreed that incorporating multi-touch surfaces and videoconferencing could enhance the visitor and learner Experience.
  • The goal of this experience is to facilitate collaborative problem solving while working in remote environments.The project also seeks to support one-to-one and many-to-many communication to allow more than one learner to participate andinteract in informal learning experiences between remote and co-located learnersCollaboration and cooperationthe primary goal is to facilitate collaborative activities within a group of co-located students, while also facilitating remote collaboration and coordination between groups of students in different locationsIntra-group collaboration refers to discussion and interaction elicited to solve a problem task, in which co-located members are involved. At the other hand, intergroup collaboration refers to problem solving activities that involve members belonging to remote groups.Physical and verbal collaborationIn collaborative environments individual participation can take place both through verbal communication and through physical actions [8]. Our system supports both physical and verbal intra- and inter-group communication and collaborationEquality of CollaborationSimultaneous interaction can prevent turns-taking behavior among participants and at the same time can promote equitable interaction [22]; however it can also result in parallel working and distribution of tasks among users [22]. In a remote environment parallel working and taking-turn behavior might constrain reciprocity and communication, therefore limiting collaboration among remote members. We adopted several design criteria to avoid limiting collaboration and promote an equal participation among remote partners. A multi-touch table allows for direct manipulation of digital content while
  • to facilitate intra- and inter- group collaboration,the system provides students with a shared workspace where simultaneous and multiuserinteractions are allowed for both co-located and remote participants
  • Interaction type:
  • evaluation consisted of 10 one-shot experimental sessions in which 20 groups (with 2 to 3 persons per group) of 39 first-year university students (14 women, 25 men) played all game levels (lasting on avg. 5.4 min.) in a controlled environment. The stations were set up in two remote rooms at a university campus. Participants in each room were given few minutes to familiarize with the multi-touch table and to introduce themselves to the remote team through videoconference while one researcher explained the main goal of the game to both teams. The researchers remained in the room throughout the session, but did not interact with participants unless in response to specific problems related to technical difficulties. All sessions were video recorded and the actions of each participant were analyzed via videocoding---------------------------------------------------mh...good point! He revisado el documento excel con todos los datos. Las secciones fueron 10 (== 20 grupos), estoy segura de esto! Pero, no me acuerdo por cual razón, tenemos la análisis de solo 9 grupos (tot personas 39!). Quizá seria mejor cambiar la slide y poner 9 experimental sessions y 18 grupos, o si no cambiar el numero de personas a 45..
  • explore the efficiency of the design criteria in promoting collaboration and discussion among participants. The user study provided quantitative and qualitative data to assess how the collaboration features in game lead to different collaborative behaviors, and to assess user’s perception and satisfaction with the game.Intra and Inter group collaboration were measured by calculating the time (in seconds) passed by each group in interacting with the co-located or remote partners (e.g. discussing where to move a piece; making suggestions, indicating on the table the correct place etc…).Values represents the score for the entire group, computed as the sum of the contributions of each member.We are interested in exploring three aspects of collaboration: 1) to what extent thegame promoted inter- and intra-group collaboration among the different stages of thegame, 2) the existence of a preferred type of collaboration, i.e. a tendency of users incollaborating more with the co-located members (intra-group collaboration) than withthe remote ones (inter-group collaboration), and 3) to what extent the game promotedequal inter-group participation, i.e. if both groups equally collaborated with the otherteam or if the process of collaboration was mostly initiated by one of the two teams.
  • We collected information about overall user satisfaction using a follow-up questionnaire divided into four main categories based on the following criteria: perceived inter- and intra-group communication, perceived inter- and intra-group collaboration, overall satisfaction of the experience. Overall satisfaction was assessed using close and open qualitative questions to elicit positive and negative aspects
  • Analysis of collaborative patterns in users’ activities showed that collaborative engagement of participants decreased as the game progressed: on average, participants collaborated for 23.3%, 16.7%, and 13.5% of the total time for level one, two and three respectively. HP: A possible explanation of this phenomenon could be related with the fact that at the beginning of the game users are more willing to collaborate between themselves, moreover they are invited to talk and negotiate in order to select a unique piece. As the game progressed, this initial stimulus decreases and so does the collaboration. **Vertical axis represents the percentage of the time passed collaborating to respect of the level completion time
  • This indicates that participants were more engaged in talking and interacting with the members of the same team than in discussing with the remote team, that was true for both level 2 and 3, where inter and intra group collaboration are elicited.**Vertical axis represents the percentage of the time passed collaborating in one of the two modalities (inter- intra-), to respect of the total time passed collaborated.
  • During the entire game, users preferred verbal communication as channel for collaboration rather than physical interaction (e.g. deictic gestures) on the surface (p <.001). But in level 1, when coordination and cooperative gestures are required, participants used more deictic gestures (physical collaboration) to communicate with the remote partners with respect to the other levels. **Vertical axis represents the percentage of the time passed collaborating in one of the two modalities (physical, verbal), to respect of the total time passed collaborated.
  • Collaboration was mostly balanced between the two groups. However, in level 1, when coordinated gestures are needed, one team had a clear predominating behavior, indicating to the other team the piece to selectWe used the Gini Coefficient to express the equitable nature of collaboration among remote teams. It uses the contribution of each group to compute the deviation with respect of equal participation between the two groups.
  • Users perceived to have collaborated more with the members of the same team than with the remote partner;, this validates our observation.Feedback regarding the positive aspects indicates that participants appreciated the collaborative nature of the game and the type of interaction allowed by the multi-touch interface.Analysis of the negative aspects of the game revealed several problems regarding the design of the game, such as the lack of clarity in the goal of the game, the lack of competition, and the game subject, the easiness of the game etcc... Other complains were addressed to the setup of the system, participants were annoyed by the fact that they had to split the attention between the two screens, they mention that sometimes they were not aware of the suggestions displayed on the vertical screen.
  • Eduactiional evaluation. Testing with real children in a natrural environment, would allow us to assess the educational impact of the application. We believe that it is necessary to go beyond lab settings and conduct evaluation on real, live environments. This will provide the test fire, and would allow us to really find out if the system satisfies the overall objectives.
  • We identified several problems. For instance, the system would have to adapt to situations in which only one user interacts by himeself on the space. As there is only one user the COLLLABOATIVE goals of the application NO LONGER are suitable, so the system should also offer a enjoyable experience to this type of user. Additionally, a similar situation occurs when there is a GROUP of participants interacting with the system, they will no longer have somebody to talk to on the other side In fact, they would not be able to use the app, as it REQUIRES both remote groups to participate. This could be addressed by adding a SOLO mode, or by simulating user interactions on the REMOTE station. Future work includes exploring other scenarios, other installation, or other settings. Specially scenarios where COOPERATIVE INTERACTION ON THE SHARED SURFACE makes sense. The Application is generic ENOUGH as to take an XML configuration file to change the behaviors, levels. All that is required is changing a few images to generate a brand new application. Some other interaction and usability aspects to improve include. Increasing remote awareness- Face to face interactions. and the use of shared immersive workspaces
  • Increase awareness support. Such as showing the other participant’s actions so that it is in a clear and interpretable by the end-users. At the moment weonly use highlight to show some other user is interacting with an object. But users have shown that they need further visual feedback.---------Pauchet. Ref aqui:
  • Facilitate face to face interaction. One of the draebacks in thesystem is that people DO NOT REALLY SEE each other. This is not a new problem, as many other similar systems share the same situation. Tele-graffiti LivePaperAgora Others….Commercial video-conferencing systemsThis problem has to be address, however there is a technical limitation in the placement of the camera. A it would need to be behind the screen. DoubleDigitalDesk
  • Thus, the future is to have immersive face to face interactive spaces where people can collaborate and communicate seamlessly. For instance, The Microsoft concept video illustrates this point showing their vision for supporting for face to face interaction in immersive and ubiquitous environments.
  • We are already tackling some of these problems in our research work.
  • Remote Collaborative Multi-User Informal Learning Experiences: Design and Evaluation

    1. 1. Remote Collaborative Multi-User InformalLearningExperiences: Design and EvaluationProceedings of 6th European Conference on Technology Enhanced Learning, EC-TEL 2011, Palermo, Italy, September 2011, pp. 43-56<br />Ernesto Arroyo, Valeria Righi, Roger Tarrago, Patricia Santos,Davinia Hernandez-Leo,Josep Blat<br />Interactive Technologies Group GTI<br /><br />UniversitatPompeuFabra<br />Barcelona, Spain<br />
    2. 2. CONTEXT<br />Two Remote Museums, one Shared Installation<br />
    3. 3.
    4. 4.
    5. 5.
    6. 6. A SHARED installation<br />LA COL·LECCIÓ DE COL·LECCIONS DE GUY SELZ <br />
    7. 7. Large collection of toys and objects<br /><ul><li>From around the World
    8. 8. Descriptions with:
    9. 9. Origin
    10. 10. Uses
    11. 11. Importance</li></li></ul><li>
    12. 12. Constrains<br />Real Stakeholders, Real Requirements<br />
    13. 13. Requirements<br /><ul><li>Open-ended scenarios (customizable)
    14. 14. Fixed-educational content
    15. 15. Sharing between remote locations
    16. 16. Game like interaction (playing at the museum)
    17. 17. Support informal learning
    18. 18. Support formative feedback
    19. 19. Guided School visits
    20. 20. Coordinated Activities in remote museums</li></li></ul><li>Objective<br />
    21. 21. Objective<br /><ul><li>Support and promote inter and intra group collaborative learning on a remote environment
    22. 22. Collaboration and cooperation
    23. 23. Remote and local collaboration
    24. 24. Intra- and Inter- group
    25. 25. Physical and verbal collaboration
    26. 26. Intra- and Inter- group
    27. 27. Equality of collaboration
    28. 28. Balanced or equal participation</li></li></ul><li>Approach<br />
    29. 29. MULTI-TOUCH<br />APPROACH:<br />
    30. 30. Multi-user<br />APPROACH<br />
    31. 31. VIDEO-CONFERENCING<br />APPROACH<br />
    32. 32. APPROACH: Interactive Experience<br />Multi-touch Interaction + Videoconferencing + Multi-user<br />Remote Game over a cooperative shared space<br />Common museum installation<br />Internet<br />Figueres Museum<br />Barcelona Museum<br />
    33. 33. Approach<br /><ul><li>Customizable games
    34. 34. Collaboration and cooperation
    35. 35. Remote and local collaboration
    36. 36. Intra- and Inter- group
    37. 37. Physical and verbal collaboration
    38. 38. Intra- and Inter- group
    39. 39. Equality of collaboration
    40. 40. Balanced or equal participation
    41. 41. Meaningful and interesting</li></ul>Goal<br />Solution<br /><ul><li>XML representation of game customizable aspects
    42. 42. Shared workspace
    43. 43. Simultaneous remote and co-located interaction
    44. 44. cooperative manipulation
    45. 45. multi-touch surface
    46. 46. Videoconference
    47. 47. remote awareness
    48. 48. Game levels
    49. 49. discovering, exploring
    50. 50. special challenges</li></li></ul><li>GAME<br />Description<br />
    51. 51. Game Description<br /><ul><li>Level 1 : Exploration</li></ul>Collaborative selection<br /><ul><li>Level 2: Geographic categorization Quiz</li></ul>Simultaneous remote and co-located interaction<br /><ul><li>Level 3: Functionality categorization Quiz</li></ul>Simultaneous remote and co-located interaction<br />
    52. 52.
    53. 53. LEVEl 0<br />Game Status: Player Selection<br />
    54. 54.
    55. 55.
    56. 56. LEVEL 1<br />Game Status: Collaborative Exploration<br />
    57. 57.
    58. 58. LEVEl 2<br />Game Status: Categorization<br />
    59. 59.
    60. 60. Level 3<br />Game Status: Categorization<br />
    61. 61.
    62. 62. Video<br />Game<br />
    63. 63. video<br />Game<br />
    64. 64. Evaluation <br />Procedure and Measures<br />
    65. 65. Procedure<br />Method:9 one-shot experimental sessions (avg. 5.4 minutes)<br />Participants:18 groups - 39 students<br /> (2-3 pl. per group)<br />Context:Set-up in two distant rooms in a <br /> university campus<br />
    66. 66. Quantitative video analysis<br /><ul><li>Intra- and inter- group collaboration</li></ul>Helping and discussing with the members of the same/remote team<br /><ul><li>Physical and Verbal collaboration</li></ul>Physical: helping moving a piece, or touching the surface to indicate the place where to move the piece<br />Verbal: suggesting, verbally, the place where to move the piece<br /><ul><li>Equality of inter-group collaboration</li></ul>Did teams equally collaborate with the remote team? <br />Time collaborating? <br />
    67. 67. Questionnaire: quantitative and qualitative analysis<br /><ul><li>Perceived inter- / intra- communication and collaboration and satisfaction</li></ul> closed+ open comments<br /><ul><li>Overall Satisfaction</li></ul>“I enjoyed playing this game”<br /><ul><li>Perceived collaboration</li></ul> “I actively worked with the players of my/remote group while I was playing the game”<br />“I played the game mostly on my own”<br /><ul><li>Negative &Positive factors</li></ul>“List 5 aspects of the experience, that you liked/disliked<br />
    68. 68. Evaluation<br />Results<br />
    69. 69. Overall Collaboration<br />Collaboration decreased as the game progressed <br />% Total Time Collaborating<br />Game Levels<br />
    70. 70. Inter-Intra-group Collaboration<br />Participants engaged significantly more inintra-group collaboration than inter-group <br />% Relative Time Collaborating<br />*p<0.001<br />2 3<br />Game Levels<br />p>0.05<br />
    71. 71. Physical – Verbal Collaboration<br />Verbal collaboration is preferred than the physical one<br /><ul><li>When cooperative interaction is required (level 1), participants used more deictic gestures to communicate with the remote partners</li></ul>% Relative Time Collaborating<br />*p<0.001<br />Game Levels<br />p>0.05<br />
    72. 72. Equity of inter-group collaboration<br />Collaboration was mostly balanced between the two groups<br /><ul><li>When negotiation is required (level 1), one team tended to predominate on the other one, by indicating the piece to select </li></ul>unequal<br />Gini Coefficient<br />equal<br />Game Levels<br />
    73. 73. Follow-up survey<br /><ul><li>Our participants were overall satisfied of the experience (M. 3.8 /5)
    74. 74. Perceived Collaboration: users perceived to have collaborated more with the members of the same team than with the remote partners
    75. 75. Positive Aspects: </li></ul>multi-touch interaction, the collaborative nature of the game, <br />videoconference with the remote group<br /><ul><li>Negative Aspects: </li></ul>split-screen setup, “easiness” of the game, lack of competition<br />
    76. 76. Resuming…<br />
    77. 77. Recapitulation<br /><ul><li>Multi-touch game to increase collaborative problem solving between remote museums’ visitors
    78. 78. Shared workspace + Videoconference = increase remote awareness
    79. 79. Simultaneous interaction avoids taking-turn between players and facilitate collaboration
    80. 80. Players play more with the co-located partners using verbal communication
    81. 81. Design of the installation?
    82. 82. Design of the game?</li></li></ul><li>FUTURE WORK<br />
    83. 83. Future Work - Evaluation<br /><ul><li>Evaluation in-the-wild
    84. 84. Real environments
    85. 85. Usability evaluation
    86. 86. Children’s feedback
    87. 87. Analysis of educational benefits</li></li></ul><li>Future Work - Interaction<br /><ul><li>Problems, system to adapt game depending:
    88. 88. One-sided participation
    89. 89. Single user interaction
    90. 90. New user scenarios
    91. 91. Extensible application
    92. 92. Generic approach based on XML configuration of behaviors (e.g., levels) and images to show
    93. 93. Interaction and ease-of-use
    94. 94. Increased remote awareness
    95. 95. …</li></li></ul><li>Pauchet<br />
    96. 96.
    97. 97. Microsoft concept video<br />F2F interaction in immersive and ubiquitious environments<br />
    98. 98. ONGOING WORK<br />Future and <br />
    99. 99. Simultaneous face-to face interaction<br /><ul><li>Collaborative interaction over 3D shared spaces
    100. 100. Multi-touch
    101. 101. Real-time
    102. 102. Immersive
    103. 103. Occlusion free</li></li></ul><li>Cooperative Face-to-Face Interaction<br /><ul><li>Collaborative interaction over 3D shared spaces
    104. 104. Multi-touch
    105. 105. Real-time
    106. 106. Immersive
    107. 107. Occlusion free
    108. 108. Cooperative</li></li></ul><li>Cooperative Face-to-Face Interaction<br />Investigating interaction methods on that facilitate cooperation on remotely shared 3D environments<br />
    109. 109. ICT DepartmentUniversitatPompeuFabra<br />C/Tànger, 122-140, E-08018 BarcelonaTel : (34) 93 542 2500 y (34) 93 542 2201Fax : (34) 93 542 2202<br />davinia.hernandez@upf.edugti.infoat upf.es<br />Thank you!<br />