PolyCAFe and Social Learning Support for CSCL in LTfLL

1,615 views

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,615
On SlideShare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
14
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

PolyCAFe and Social Learning Support for CSCL in LTfLL

  1. 1. PolyCAFe and Social Learning Support for CSCL in LTfLL Stefan Trausan-Matu Traian Rebedea Mihai Dascalu Vlad Posea “Politehnica" University of Bucharest Computer Science Department9th December 2010
  2. 2. General data 2 9 December 2010
  3. 3. The LTfLL - EU FP7 STREP Project (2008-2011)• Open Universiteit Nederland (coordinator)• The University of Manchester• Open University UK• Universiteit Utrecht• Eberhard Karls Universität Tübingen• Wirtschaftsuniversität Wien• Université Pierre-Mendès France, Grenoble• Politehnica University of Bucharest (PUB)• Institute for parallel processing of the Bulgarian Academy• Aurus Kennis- en Trainingssystemen BV• BIT MEDIA E-learning solution GMBH and CO KG 3 9 December 2010
  4. 4. Past International Research ProjectsRelated to LTfLL The cognitive paradigm (e.g. Web semantic) Portable AI Lab (PAIL) – IDSIA Lugano (1994-1995) PeKADS - EU Copernicus (1994-1996) LARFLAST – EU Copernicus (1998-2001) IKF – EU EUREKA (2002-2003) Towntology – EU COST Action (2005-2008) The socio-cultural paradigm (e.g. Web2.0) VMT –USA NSF (2005-2007) EU-NCIT – EU FP6 (2005-2008) Cooper – EU FP6 (2005-2007) 4 9 December 2010
  5. 5. The Outcomes of the LTfLL ProjectPrototypes of next-generation services built onadvanced research on the application oflanguage technologies in education. http://www.ltfll-project.org 5 9 December 2010
  6. 6. LTfLL Services 6 9 December 2010
  7. 7. LTfLL Development Cycles 7 9 December 2010
  8. 8. Workpackages Work WP1 WP2 WP3 WP4 WP5 WP6 WP7 WP8 WP9 TOTAL per package Beneficiary OUNL 19 7 10 11 11 3 2 63 UU 3 5 5 2 18 7 3 1 44 UTU 7 6 18 6 3 1 41 WUW 10 3 4 7 4 3 1 32 UPMF 1 1 19 3 2 1 27 PUB-NCIT 1 3 19 17 8 3 1 52 AURUS 3 6 1 6 3 1 20 KTS UNIMAN 5 12 6 14 3 1 41 IPP-BAS 3 6 11 15 9 3 1 48 BIT 15 3 5 6 1 30 MEDIA OU 8 3 6 3 1 1 22 TOTAL 19 58 51 68 53 56 70 33 12 420 8 9 December 2010
  9. 9. PUB Implementation Team Prof.dr.ing. Stefan Trausan-Matu Prof.dr.ing. Valentin Cristea As.drd.ing. Traian Rebedea As.drd.ing. Vlad Posea As.drd.ing. Mihai Dascalu As.drd.ing. Costin Chiru As.drd.ing. Dan Mihaila Ing. Alexandru Gartner Ing. Erol ChioascaStudents which helped at the implementation: Dan Banica Mihai Nicolae Iulia Pasov Ionela Voinescu Iulia Moscalenco Oana Mihai Alexandru Georgescu 9 9 December 2010
  10. 10. PUB Publications on LTfLL 29 papers, 12 in Proceedings ISI, 23 in international databases, 2 NLPSL Workshops with Proceedings (Eds. S. Trausan-Matu, P. Dessus) 4 book chapters (including published at Springer and Hershey) 10 9 December 2010
  11. 11. The Problems Solved by PolyCAFe(a Polyphony-Based System for Collaboration Analysis and Feedback Generation) 11 9 December 2010
  12. 12. Chat Conversations with MultipleParticipants Multiple participants (≥3), conferencing style Particular features – multiple, parallel discussion chains !!! There is a need for Determining important utterances Contributions of the participants Degree of collaboration - inter-animation analysis 12 9 December 2010
  13. 13. Example: CSCL assignment Students had to debate in chat sessions in groups ranging from 3 to 8 In the first part of the conversation, each student had to defend a technology by presenting its features and advantages and criticize the others by invoking their flaws and drawbacks In the final part of the chat, they had to discuss on how they could integrate all these technologies in a single online collaboration platform 13 9 December 2010
  14. 14. CSCL assignment: Problems How to assist teachers in evaluating students’work in chats? Offer assistance to students Abstraction tools Automatic feedback 14 9 December 2010
  15. 15. Experiments with Chat-basedCSCL K-12 students solving mathematics problems bothindividually and collaboratively in the VMT project atDrexel University, Philadelphia, USA Computer Science students at Bucharest “Politehnica”University, Romania at Human-Computer Interaction course in Romanian and French – role playing and debate Natural Language Processing - role playing and debate Algorithm Design – problem solving 15 9 December 2010
  16. 16. The VMT chat environment 16 9 December 2010
  17. 17. VMT Referencing facility 17 9 December 2010
  18. 18. Theoretical Basis of PolyCAFe 18 9 December 2010
  19. 19. Paradigms about KnowledgeCognitive Socio-culturalNewell, Simon Vygotsky, Bakhtin“Knowledge is in the head” “Knowledge is in the community”Artificial Intelligence, Theory of Activity,Natural Language Processing Collaborative systemsOntologies FolksonomiesSemantic Web Social Web (Web2.0)Intelligent Tutoring Systems Computer-Supported Collaborative Learning 19 9 December 2010
  20. 20. Computer Supported CollaborativeLearning A new paradigm in learning with computers (Koshmann, 1999): Knowledge is constructed socially (Vygotsky) Induced by the spread of forums, chats, blogs, wikis and folksonomies learning in (on-line) virtual teams and/or communities 20 9 December 2010
  21. 21. Dialogism – Mikhail Bakhtin• Basis for the CSCL paradigm (Koschman, 1999)• “… Any true understanding is dialogic in nature” (Voloshinov-Bakhtin, 1973)• Opposed to de Saussure ideas, which are the basis for Natural Language Processing Polyphony Inter-animation of voices 21 9 December 2010
  22. 22. The Polyphonic Model of CSCL(Trausan-Matu, Stahl and Zemel, 2005,http://mathforum.org/wikis/uploads/Stefan_Interanimation.doc) A polyphony of voices characterizes any linguistic phenomenon (Bakhtin) including CSCL chats Inter-animation (Bakhtin, Wegerif) may be detected in interactions and it may be used for analyzing collaboration and assessing learners Integrating NLP techniques with polyphony identification and Social Network Analysis may provide a way for analyzing the contributions of each participant and their collaboration. Inter-animation and polyphony appears also in non-verbal interactions Consider threads (voices which last) rather than analyzing pairs of utterances 22 9 December 2010
  23. 23. Polyphonic analysis(Trausan-Matu & Stahl, 2007, http://gerrystahl.net/vmtwiki/stefan.pdf) 23 9 December 2010
  24. 24. 24 9 December 2010 24
  25. 25. Words, voices and threads Different positions assigned to participants – different voices Additional voices – frequent concepts – repeated words become voices, stronger or weaker Voices continue and influence each other through explicit or implicit links. Voices correspond to chains or threads of utterances: repeated words lexical chains co-references reasoning or argumentation rhetorical schemas 25 9 December 2010
  26. 26. Analysis Units in the PolyphonicModelWordsUtterancesPairs of utterances (links)ThreadsVoicesParticipants 26 9 December 2010
  27. 27. Units of Interaction Echoes of voices Polyphonic-contrapuntal weaving Inter-animation Links between utterances and between words Links may be: implicit explicit 27 9 December 2010
  28. 28. Inter-animation Patterns(Trausan-Matu, Stahl & Sarmiento, 2007)Longitudinal Adjacency pairs Repetitions Elaboration Cumulative talk (collaborative utterances) RepairTransversal Convergence Differential Dissonance 28 9 December 2010
  29. 29. Implementation Details of PolyCAFe 29 9 December 2010
  30. 30. Architecture of PolyCAFE 30 9 December 2010 30
  31. 31. 31 9 December 2010
  32. 32. NLP pipe spelling correction, stemmer, tokenizer, Named Entity Recognizer, POS tagger and parser, and NP-chunker. Stanford NLP software (http://nlp.stanford.edu/software) Spellchecker : Jazzy http://www.ibm.com/developerworks/java/library/j- jazzy/ Alternative NLP pipes are under development, GATE (http://gate.ac.uk) LingPipe (http://aliasi.com/lingpipe/). 32 9 December 2010
  33. 33. Functional architecture Surface Analysis Readability & Page essay grading Morphological Analysis Social Network and POS Tagging Analysis Semantic Evaluation (LSA) Grading Process 33 9 December 2010
  34. 34. Tagged LSA• Corpus of chats focused on collaborative technologies• POS Tagging• Stemming• Segmentation – Participants – Fixed Window• Cosine similarity Term-Doc Matrix + Tf – Idf 34 9 December 2010
  35. 35. Vector space visualization Radial Model Physical Model 35 9 December 2010
  36. 36. Social Network Analysis• Degree• Centrality – Closeness – Graph – Eigen Value• User Ranking – Google Page Ranking 36 9 December 2010
  37. 37. Utterance evaluation Social • Degree • Semantic similarity Qualitative • Predefined topics • Overall discourse • NLP Pipe Quantitative • No of occurrences   mark(u) =  ∑ words ( stem ) × (1 + log( no _ occurences ))  × emphasis (u ) × social (u ) length   remaining  emphasis (u ) = Sim (u , whole _ document ) × Sim (u , predefined _ keywords ) social (u ) = ∏ (1 + log( f (u )) all social factors f ( quantitative and qualitative ) 37 9 December 2010
  38. 38. Collaboration (1)• Utterance graph – Explicit links – Attenuation – Implicit links – Trust• Social cohesion• Quantitative collaboration ∑ all links l with different speakers attenuatio n(l) * trust(l) quantitati ve collaborat ion = total number of links (implicit/ explicit) 38 9 December 2010
  39. 39. Collaboration (2) Qualitative - Gain based collaboration = ECHO Personal – individual knowledge building Links to previous utterances with same speaker Collaborative – collaborative knowledge building Links to previous utterances with different speaker personal gain(u) = ∑ ((mark(v) + gain(v) ) * similarity(u, v) * attenuation(l) * trust(l)) link l exists between u and v, v is an earlier utterance and u and v have same speakercollaborative gain(u) = ∑ ((mark(v) + gain(v) ) * similarity(u, v) * attenuatio n(l) * trust(l)) link l exists between u and v, v is an earlier utterance and u and v have different speakers 39 9 December 2010
  40. 40. Collaboration (3) 40 9 December 2010
  41. 41. Services & Widgets PolyCAFe is an online platform: Web services (Java and PHP-based) Web widgets using W3C Widgets1.0 standard The widgets can be integrated into any web platform that has a W3C widget container (e.g. Wookie) There are services for maintenance tasks and analysis tasks (process discussion, search, etc.) The widgets have been deployed in Elgg (PLE) 41 9 December 2010
  42. 42. Presentation of Widgets• Conversation Feedback (content + collaboration)• Conversation Visualization (collaboration)• Utterance Feedback (content + collaboration)• Participant Feedback (content + collaboration)• Search Conversation (content)• Prepare chat/forum assignment• Add conversations for assignment 42 9 December 2010
  43. 43. Conversation Feedback 43 9 December 2010
  44. 44. Conversation Visualization 44 9 December 2010
  45. 45. 45 9 December 2010
  46. 46. 46 9 December 2010
  47. 47. Search Conversation 47 9 December 2010
  48. 48. Validation and Verification of PolyCAFe 48 9 December 2010
  49. 49. Validation Experiment 1.0• Validation of PolyCAFe 1.0 – 9x students – 5x tutors, 1x teacher• Two chats with 4-5 students in a team• All the students used PolyCAFe for the first time• Only two tutors have previously used the system• Students used the automatic feedback to get insight about their discussion• Tutors used PolyCAFe to offer manual feedback 49 9 December 2010
  50. 50. Validation Results – Tutors The tutors validated all the instruments with suggestions to change part of them Quiz with 35 questions – all of them passed with an average score between 3.50-5.00 / 5.00 The system is relevant and useful for their activity The time for providing final feedback to the students is definitively reduced (between 30- 50%) The quality of the feedback is improved 50 9 December 2010
  51. 51. Validation Results – Tutors (2) Category Average Perc. agree Pedagogic effectiveness 4.11 83% Efficiency 5.00 100% Cognitive load 4.60 100% Usability 4.36 93% Satisfaction 4.57 91% Total 4.53 93%1 – Strongly disagree; 2 – Disagree; 3 – Neutral; 4 – Agree; 5 – Strongly agree 51 9 December 2010
  52. 52. Validation Results – Students The students have validated most of the instruments Quiz with 32 questions – 5 where not validated; all others have scores over 3.66/5.00 In the focus group, they reported that several misleads have been found using this widgets These errors or misleads were reported to be only minor without influencing the overall feedback It has been suggested to try and fix them in order to gain the full trust of the users 52 9 December 2010
  53. 53. Validation Results – Students (2) For example, the students did not validate: “Overall, I believe that the support for my learning PolyCAFe (Chat Analysis and Feedback Service) provides is close enough to the current support provided by humans. ” Average = 3.11 Perc. agree = 33% However, we do not want to provide a substitute for human evaluation! 53 9 December 2010
  54. 54. Validation Results – Students (3) Category Average Perc. agree Pedagogic effectiveness 3.94 77% Efficiency 4.22 78% Cognitive load 3.56 56% Usability 4.11 81% Satisfaction 3.89 72% Total 3.94 73%1 – Strongly disagree; 2 – Disagree; 3 – Neutral; 4 – Agree; 5 – Strongly agree 54 9 December 2010
  55. 55. Validation Results - Widgets Learners TutorsValidation Statement Agreement AgreementPolyCAFe feedback is useful 100% 100%PolyCAFe feedback is relevant 63% 80%Conversation feedback is useful 78% 80%Conversation visualisation is 89% 100%usefulUtterance feedback is useful 83% 100%Participant feedback is useful 78% 100%Search conversation is useful 61% 100% 55 9 December 2010
  56. 56. Validation Experiment 2.0Validation of PolyCAFe 1.5 25x students in experimental group 10x students in control group 6x tutors in experimental group2x 7 chats with 5 students in a teamMost of the students used PolyCAFe for the first timeAll the tutors have previously used the systemExperiment is still under-wayPreliminary results are very encouraging: very highcorrelation between ranking of participants by the systemand by the participants themselves 56 9 December 2010
  57. 57. Verification Accuracy of system ranking of participants Accuracy of grading utterances Accuracy of determining implicit links, collaboration areas, discussion threads Accuracy for determining speech acts Accuracy for determining Model of Inquiry classes 57 9 December 2010
  58. 58. Example – Speech ActsSpeech act - label Precision Recall Continuation 93% 92% Statement 94% 93% Greeting 100% 80% Accept 92% 80% Partial accept 71% 55% Agreement 90% 51% Understanding 96% 58% Negative 97% 78% Reject 73% 82% Partial reject 35% 27%Action directive 75% 90% Info request 100% 71% Thanks 100% 100% Maybe 100% 69% Conventional 66% 50%Personal opinion 100% 36% Sorry 66% 75% 58 9 December 2010
  59. 59. Conclusions and Future Work(PolyCAFe) The system is working well The system passed the validation with students and tutors The visualization widget proved to be the most useful The interface might be improved 59 9 December 2010
  60. 60. Transferability Issues• Domain – The topic of the conversation should be easily solved using discussions, no graphics or formulas• Language – Need for the components of the NLP pipe – Corpus for training the LSA – Maybe, a domain ontology• Activity – Collaborative activity – Teams of 4-15 students (in the current design) 60 9 December 2010
  61. 61. Learning Support Based on SocialNetworks Search and Recommendation 61 9 December 2010
  62. 62. Learning Scenario Learner just uses social networking web sites He connects to friends and possible tutors Our application indexes the user’s friends and resources and their peers friends and resources Based on the data acquired we offer search and recommendation services 62 9 December 2010
  63. 63. Tag and Social Network-based Search Learner Community Knowledge = = =Web 2.0 user Social networking application Relevant content 63 9 December 2010
  64. 64. Find Relevant ResourcesUnderstand the learner’s request and lead him to aresource that is both relevant and trusted as it isrecommended by a member of the community 64 9 December 2010
  65. 65. Find Relevant PeersWe search for the right person to offer feedbackand guidance 65 9 December 2010
  66. 66. Semantic Indexing of Learning Objects• Crawled dataconverted in semanticformats •Vocabularies used: SIOC, Neuman’s tagging ontology, SCOT, MOAT, SKOS•*using Amazon EC2for crawling 66 9 December 2010
  67. 67. Search and RecommendationsSearch• Search algorithm based on PageRank R (v ) R(u ) = c ∑• Folkrank - a user is important if he annotates important (v) v∈B ( u ) N resources with relevant (well ranked) tags.• Fokrank works on a folksonomy - hypergraph - F=(V,E) where V=U ∪R ∪T and E⊂UxRxTRecommendation• Uses clustering of resources, users and tags to suggest relevant resources 67 9 December 2010
  68. 68. Architecture of the Application 68 9 December 2010
  69. 69. Using the Learning Services in a PLE Search for Search for resources users username Social networking platform RecommendedCheck the readingslearner’s Relevant tagsprofile and their importance 69 9 December 2010
  70. 70. Conclusions and Future Work(Search and recommendations service) We use social data to provide feedback for learners We offer relevant resources to the user from his social network We help learners to find peers or tutors The work was embedded in Moodle/WebCT and validated in Bucharest and Utrecht We’re centralizing the results 70 9 December 2010

×