Your SlideShare is downloading. ×
Cross-System Personalization for College Students
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.


Saving this for later?

Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime - even offline.

Text the download link to your phone

Standard text messaging rates apply

Cross-System Personalization for College Students


Published on

An invited talk presented at UbiqUM workshop on 6/25/2012

An invited talk presented at UbiqUM workshop on 6/25/2012

Published in: Education, Sports

  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. Cross-SystemPersonalization forCollege StudentsPeter Brusilovsky withSergey SosnovskyMichael YudelsonShaghayegh SahebiChirayu WongchokprasittiSharon Hsiao
  • 2. User-Adaptive Systems Then… Collects information about individual user User Modeling side Adaptive System User Model Adaptation side Provides adaptation effect Classic loop user modeling - adaptation in adaptive systemsUniversity of Pittsburgh - PAWS Lab
  • 3. … and nowUniversity of Pittsburgh - PAWS Lab
  • 4. Personalization Challenges in the NewContext•  How an adaptive system can benefit from information about users collected by other systems? –  What is the framework for UM integration? –  Will it improve cold-start situation? –  Will it improve parallel use of multiple systems•  Can we do it for different types of user models? –  Knowledge model –  Interest modelUniversity of Pittsburgh - PAWS Lab 4
  • 5. Concept-Level Knowledge Model Concept 4 Concept 1 3 10 Concept N Concept 2 0 7 2 4 Concept 5 Concept 3 University of Pittsburgh - PAWS Lab
  • 6. Cross-System Knowledge Modeling Concept 4 Concept 1 no yes Concept N no Concept 2 yes no yes Concept 5 Concept 3 Concept 4 Concept 1 no yes Concept N no Concept 2 yes no yes Concept 5 Concept 3 Concept 4 Concept 1 no yes Concept N no Concept 2 yes no yes Concept 5 Concept 3 University of Pittsburgh - PAWS Lab
  • 7. The Approach: Ontology-Based Cross-System Personalization Connect DM (ontologies) Missing linksUniversity of Pittsburgh - PAWS Lab
  • 8. Main Stages of Our Work•  Centralized user modeling (1990-1998)•  Multi-system personalization based on single domain model: ADAPT2 (2003-2007)•  Cross-domain mapping for cold start (2007) –  C to Java•  Single domain guided evidence mapping (2008-2010) –  Topic to concept mapping for Java –  Constraints to concepts mapping for SQL•  Single domain automatic mapping (2010-2012)University of Pittsburgh - PAWS Lab 8
  • 9. How we started – from C to Java•  Manual vs. Automatic ontology mapping•  Knowledge mapping using ontology mapping Java•  Compare predicted and C demonstrated knowledge•  Automatic mapping is comparable with manual•  Overall gain for translated knowledge is not high UM of C UM of•  We got concerned about knowledge Java model to model mapping knowledge•  Started exploring evidence mappingUniversity of Pittsburgh - PAWS Lab
  • 10. SEDONA: UM exchange with ontologyservers Concept 4 Concept 1 no yes Concept N no Concept 2 yes no yes Concept 3 Concept 5 Ontology A Concept 4 Concept 1 no yes Concept N no Concept 2 yes no yes Concept 5 Concept 3 Ontology B Concept 4 Concept 1 no yes Concept N no Concept 2 yes no yes Concept 5 Concept 3 University of Pittsburgh - PAWS Lab
  • 11. SEDONA: UM Exchange•  Ontology server is an exchange point for concept- level overlay student models that are based on the stored ontology•  Each UM server or adaptive system that can deduce student knowledge in terms of this ontology reports it to the server•  Each adaptive system that need to know the level of student knowledge for concepts of this ontology can query the ontology serverUniversity of Pittsburgh - PAWS Lab
  • 12. Lightweight event-based centralizeduser modeling Concept 4 Concept 1 no yes Concept N no Concept 2 yes no yes Concept 5 Concept 3 Concept 4 Concept 1 no yes Concept N Concept 4 no Central UM Concept 1 no Concept 2 yes Concept N yes no no Concept 2 yes yes Concept 5 no Concept 3 yes Concept 5 Concept 3 Concept 4 Concept 1 no yes Concept N no Concept 2 yes no yes Concept 5 Concept 3 University of Pittsburgh - PAWS Lab
  • 13. Goal: True Integration•  Student side: – Use systems in parallel (any order, any combination) – No extra overhead (single sign-on, single place to access)•  System side: – Integrated environment > (system1 + system2) – Each system should try to increase the quality of user modeling and adaptation University of Pittsburgh - PAWS Lab
  • 14. Java Problets: The Interface HelpQuestion textSampleprogram System s feedbackStudent s answer
  • 15. Java Problets: Domain Model•  Problets implement traditional overlay user modeling to adapt to student s performancel  The domain model of a problet is a concept map enhanced with learning objectives, that combine pedagogical and domain knowledge
  • 16. QuizJET (1):System Description•  QuizJet (Java Evaluation Toolkit) is a system for authoring and delivery of online self-assessment quizzes for Java programming language•  A typical QuizJET problem is a sample program (consisting of one or several classes), that a student needs to evaluate and provide an answer a follow-up question•  QuizJET generates problems by substituting a numerical value in the program template with a randomized parameter•  Upon receiving a student s answer QuizJET provides a feedback indicating the correctness of the answer and the right answer (if the student s attempt was not successful)
  • 17. QuizJET (2):Student Interface•  Students can access QuizJET problems through the KnowledgeTree portal ProblemsTopics in the classes course Problem Activities textavailable for the current topic QuizJET s feedback
  • 18. QuizJET (3): Domain Model•  Java Ontology specifies about 500 classes connected with 3 types of relations: subClassOf, partOf/hasPart, and related•  About 300 classes are available for indexing•  A class can play one of two roles in the problem index: prerequisite or outcome University of Pittsburgh - PAWS Lab
  • 19. Domain Model Integration•  Main problem: different modeling paradigms –  A learning objective models application of a concepts in the certain context –  Extra classes from the Java ontology have been used for context modeling –  Weights are assigned to prevent too aggressive propagation of classes responsible for context modeling•  Example: –  This learning objective models a situation when the conditional part of the if-else statement is a relational expression evaluated into true value
  • 20. Evidence-based UM integration inCUMULATEUniversity of Pittsburgh - PAWS Lab
  • 21. SQL-ExploratoriumUniversity of Pittsburgh - PAWS Lab
  • 22. SQL-Tutor
  • 23. Goal: Integrated Environment
  • 24. SQL Explorer: SQL OntologyUniversity of Pittsburgh - PAWS Lab
  • 25. SQL-Tutor: ConstraintsUniversity of Pittsburgh - PAWS Lab
  • 26. Domain Model Mapping•  Constraints and Concepts are too difficult to map them automatically •  A typical constraint models syntactic or semantic relation between several concepts •  Manual connect constraint to concepts with some degree (small-1, medium-2, or large-3)University of Pittsburgh - PAWS Lab
  • 27. Evidence-Based Modeling •  Solution to SQL-Tutor problem, triggers a number of constraints satisfied and or violated •  Mapping model calculates knowledge update for every concepts related to every triggered constrained: •  The updates are reported to SQL- Exploratorium s user modeling serverUniversity of Pittsburgh - PAWS Lab
  • 28. Architecture
  • 29. Evaluation •  University of Pittsburgh, 2 courses: undergraduate and graduate •  ½ of semester •  42 students tried SQL-KnoT, 18 – SQL- Tutor •  Out of 103 sessions of using SQL-KnoT 66 co-located with SQL-Tutor usageUniversity of Pittsburgh - PAWS Lab
  • 30. Results•  Questionnaire (21 students) – I1 / I2: Overall, I like the interface of SQL- KnoT/SQL-Tutor. – U1 / U2: SQL-KnoT/SQL-Tutor is a useful learning tool. – C1 / C2: SQL-KnoT/SQL-Tutor problems challenged me intellectually.
  • 31. What was presented in the past UbiqUM•  An example of semantic integration of two working adaptive systems relaying on very different domain models•  Students used the systems together during single sessions and liked the opportunity•  More evaluation is needed to verify the effect of integration of user modeling accuracy and adaptation•  It is interesting to evaluate the combined adaption (adaptive navigation from SQL-Exploratorium followed by intelligent coaching from SQL-Tutor) University of Pittsburgh - PAWS Lab
  • 32. Discussion+ Experts only need to produce relations b/w KIs – the rest is automatic+ Relations can be removed (strength=0)-  Cannot add relationsUniversity of Pittsburgh - PAWS Lab
  • 33. References: Past UbiqUM papersSosnovsky, S., Mitrovic, A., Lee, D. H., Brusilovsky, P., Yudelson, M., Brusilovsky, V., and Sharma, D. (2008) Towards integration of adaptive educational systems: mapping domain models to ontologies. Proceedings of 6th International Workshop on Ontologies and Semantic Web for E- Learning (SWEL2008) in conjunction with ITS2008, Montreal, Canada, June 23, 2008.Sosnovsky, S., Mitrovic, A., Lee, D. H., Brusilovsky, P., and Yudelson, M. (2008) Ontology-based integration of adaptive educational systems. Proceedings of 16th International Conference on Computers in Education (ICCE’2008), Taipei, Taiwan, October, 27-31, 2008, pp. 11-18.Brusilovsky, P., Sosnovsky, S., Yudelson, M., Kumar, A., and Hsiao, I.-H. (2008) User Model Integration in a Distributed Adaptive E-Learning Systems. Proceedings of Workshop on User Model Integration at the 5th International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems (AH2008), Hannover, Germany, July 29, 2008.Brusilovsky, P., Mitrovic, A., Sosnovsky, S., Mathews, M., Yudelson, M., Lee, D., and Zadorozhny, V. (2009) Database exploratorium: a semantically integrated adaptive educational system. In: Proceedings of Ubiquitous User Modeling Workshop at the 17th International Conference on User Modeling, Adaptation, and Personalization (UMAP 2009), Trento, Italy, June 22, 2009Sosnovsky, S., Brusilovsky, P., Yudelson, M., Mitrovic, A., Mathews, M., and Kumar, A. (2009) Semantic Integration of Adaptive Educational Systems. In: T. Kuflik, S. Berkovsky, F. Carmagnola, D. Heckmann and A. Krüger (eds.): Advances in Ubiquitous User Modelling. Lecture Notes in Computer Science, Vol. 5830, pp. 134-158University of Pittsburgh - PAWS Lab 33
  • 34. Evaluating and improving mapping:SQL Exploratorium and SQL Tutor•  Authoring constraint mapping is time consuming•  How we can evaluate weights?•  How we can improve mapping? Constraints Concepts w=1 Join 207 "You need to specify the join w=2/3 condition in FROM!" FROM Clause w=2/3 Attribute 147 "You have used some names w=2/3 in the WHERE clause that are not Table w=1/3 from this database." DatabaseUniversity of Pittsburgh - PAWS Lab 34
  • 35. SQL KnoT and SQL-Tutor (2)•  6 experts (2 teachers, 2 GSA, 2 practitioners)•  1012 constraint-concept relations: strong (1/1), medium (2/3), weak (1/3)•  Usage log of 3544 SQL-Tutor problem-solving attempts of 38 users•  Dataset specific subset –  282 constraints, 576 relations, 61 conceptsUniversity of Pittsburgh - PAWS Lab
  • 36. Fitting The Source(Constraint) Model•  Experts only need to produce relations b/w KIs – the rest is automaticUniversity of Pittsburgh - PAWS Lab 36
  • 37. What was not presented at UbiqUM•  Sosnovsky, S., Dolog, P., Henze, N., Brusilovsky, P., and Nejdl, W. (2007) Translation of overlay models of student knowledge for relative domains based on domain ontology mapping. 13th International Conference on Artificial Intelligent in Education, AI-ED 2007, Marina Del Rey, CA, July 9-13, 2007, IOS, pp. 289-296•  Yudelson, M., Brusilovsky, P., Mitrovic, A., and Mathews, M. (2010) Using Numeric Optimization To Refine Semantic User Model Integration Of Adaptive Educational Systems. Proceedings of the Third International Conference on Educational Data Mining (EDM 2010), Pittsburgh, PA, June 11-13, 2010, pp. 221-230.•  Sosnovsky, S. (2011) Ontology-based Open-Corpus Personalization for e-Learning. PhD Thesis, University of PittsburghUniversity of Pittsburgh - PAWS Lab
  • 38. What Happened with auto-mapping?University of Pittsburgh - PAWS Lab 38 Sergey Sosnovsky PhD Thesis
  • 39. OOPS Interface: Reading Phase Feedback/exit buttons Navigation links to the next and the previous topics content of the chosen topicUniversity of Pittsburgh - PAWS Lab 39 Sergey Sosnovsky PhD Thesis
  • 40. Cross System Interest Modeling•  CoMeT: a social system for sharing information about research colloquia in Pittsburgh•  Models user research interests by observing bookmarking and sharing behavior•  Cold start problem – can’t recommend with no bookmarks•  Can we seed user profiles using other systems that represent user research interests? –  Paper bookmarking systems – CuteULikeUniversity of Pittsburgh - PAWS Lab
  • 41. CoMeT: Collaborative Management ofTalks (try of Pittsburgh - PAWS Lab 41
  • 42. Traditional Interest Modeling withKeywords•  Document model –  a bag of words represented as a vector in keywords vector space with TF.IDF weighting scheme Keywords W W W W W W 1 2 3 4 5 6 D1 0 1 0 0 0 0 D2 .5 0 0 .5 0 0 Talks/Papers D3 .12 .13 0 .25 .5 0 D4 .25 0 .25 0 .25 .25University of Pittsburgh - PAWS Lab
  • 43. Recommending Talks to Users •  User model –  A combination of vectors of “interesting documents” –  Possibly weighted by the rare of interest •  K-nearest neighbor method –  recommend top K closest documents to user profile UP: User Profiles U: User Profiles in D: Documents in in Keywords Space Talks/Papers Space Keywords Space w1 w w3 W W w3 D1 D2 D3 D4 2 1 2 U1 1 0 1 U1 1 0 0 0 D1 0 1 0 D2 0 0 .5 user U2 .25 0. .37user U2 .25 0 .5 .25 s 5s D3 0 1 0 U3 0 . .37 U3 0 .5 .25 .25 25 D4 0 0 .5 Keywords Documents Keywords University of Pittsburgh - PAWS Lab 43
  • 44. Recommendation with AdditionalSources of Information•  Sources of information about user interests: –  Standard information: Keywords of bookmarked talks in CoMeT –  Tags of talks in CoMeT –  Keywords of bookmarked papers from CiteULike –  Tags of papers in CiteULike (CUL)•  Explore the impact of additional sources•  Also explore different models for fusion of tags and keywordsUniversity of Pittsburgh - PAWS Lab 44
  • 45. Document Representation Models•  Control condition: Keywords Only (KO) –  Keywords extracted from documents’ titles and abstracts•  Keywords+n*Tags (KnT) –  Keywords extracted from documents’ titles and abstracts + tags assigned to documents•  Keywords Concatenated by Tags (KCT) –  Keywords extracted from documents’ titles and abstracts + tags assigned to documentsUniversity of Pittsburgh - PAWS Lab 45
  • 46. Keywords+n*Tags (KnT) Model•  Each document: a bag of words containing : –  document’s abstract, title and tags•  Tags: regular keywords –  Each tag appears n times•  Merge CUL and CoMeT data in this model: same as KO Common Tag Keywords Keywords & Tags s D3 W3 W4 W3=T1 W1 W2 T3 T4 /T1 /T2 W4=T2 Keywords: w1, w2, w3, w2 n=2 D1 0 1 1 0 0 0 D2 1 0 3 5 0 0 Tags: Talks/Papers T1, T3 D3 1 2 3 0 1 0 D4 2 0 5 0 2 1University of Pittsburgh - PAWS Lab 46
  • 47. Keywords Concatenated by Tags (KCT)Model•  Tags: a separated source of information•  Each document: a bag of keywords and a bag of tags –  Concatenating keywords and tags vectors –  TF.IDF weightening scheme Keywords Tags D3 W1 W2 W3 W4 T1 T2 T3 T4 Keywords: W3=T1 w1, w2, w3, w2 W4=T2 D1 0 1 1 0 0 0 0 0 Talks/Papers D2 1 0 3 1 0 2 0 0 Tags: T1, T3 D3 1 2 1 0 1 0 1 0 D4 2 3 3 0 1 0 2 1University of Pittsburgh - PAWS Lab 47
  • 48. Merging CUL and CoMeT Data D: Merged Documents’ Matrix Dc: CUL Papers’ Matrix Dt: CoMeT Talks’ Matrix W1 w2 W3 T1 T2 w1 w T1 T2 C1 0 0 1 0 0 2 W W T1 P1 1 0 0 0 2 3 K C2 0 0 0 .5 0k P2 .25 0 .5 .25 e C1 0 1 0 + P1 1 0 0 0 0 C2 0 0 .5 e P2 .25 0 0 .5 .25 P3 0 .5 .25 .25 P3 0 .5 0 .25 .25 m+i l+j l+m+i+j-o-p k- the number of CiteULike papers m- the number of keywords used in CiteULike papers i- the number of tags used in CiteULike papers e- total number of talks in CoMeT l- total number of keywords in CoMeT j- total number of tags in CoMeT o- the number of common keywords between two CoMeT and CiteULike systems P- the number of common tags between two CoMeT and CiteULike systems University of Pittsburgh - PAWS Lab 48
  • 49. Experimental Results•  User study: –  8 real users of both CoMeT and CiteULike systems•  Questionnaire for each recommended talk: –  Is this talk related to your interest? (yes/no question) –  How interesting this talk to you? (in 5-point scale) –  If the talk is related to your interests, how novel is this talk to you? (in 5-step scale)•  Measures: –  Relevance: precision by yes/no answers –  Interest: nDCG by 5-point scale –  Novelty: averaged the novelty ratings (Non-relevant = zero novelty)University of Pittsburgh - PAWS Lab 49
  • 50. Precision and Novelty for differentnumber of recommendationsUniversity of Pittsburgh - PAWS Lab 50
  • 51. Conclusion•  Including another reliable user profile →  increase precision of recommendations;•  Using CiteULike data for all models –  Increased Relevance of recommended documents –  Decreased novelty for KO model •  CiteULike: adding, reviewing and rating related papers to their research field •  CoMeT: information about talks happening within a specific time given on a particular date users bookmark a more novel, less relevant talk•  Adding tags –  Increased novelty of recommendations (both using CoMeT and CUL data) –  Increased relatedness in larger number of recommendations•  Injection of keywords from another source of data: more reliable than including tags for relevance•  Including tags from various sources of information: more reliable for interestingness or noveltyUniversity of Pittsburgh - PAWS Lab 51
  • 52. Back to the start One user, many models of the same userUniversity of Pittsburgh - PAWS Lab
  • 53. Let’s look from the other side User Model User Model Adaptive System User Model User Model Many users, many models of different users University of Pittsburgh - PAWS Lab
  • 54. Treemap Group UM for JavaProgrammingUniversity of Pittsburgh - PAWS Lab
  • 55. The more the students compared to their peers, the higher post-quiz scores they received (r= 0.34 p=0.004)University of Pittsburgh - PAWS Lab
  • 56. Progressor for Java ProgrammingUniversity of Pittsburgh - PAWS Lab
  • 57. Progressor+ for rich contentUniversity of Pittsburgh - PAWS Lab
  • 58. And some more references •  Sahebi, S., Wongchokprasitti, C., and Brusilovsky, P. (2010) Recommending research colloquia: a study of several sources for user profiling. In: Proceedings of the 1st International Workshop on Information Heterogeneity and Fusion in Recommender Systems (HetRec 2010) at the 2010 ACM conference on Recommender systems, RecSys 10, Barcelona, Spain, ACM, pp. 32-38 •  Brusilovsky, P., Hsiao, I.-H., and Folajimi, Y. (2011) QuizMap: Open Social Student Modeling and Adaptive Navigation Support with TreeMaps. Proceedings of 6th European Conference on Technology Enhanced Learning (ECTEL 2011), Palermo, Italy, Sptember 20-23, 2011, Springer-Verlag, pp. 71-82. •  Hsiao, I.-H., Bakalov, F., Brusilovsky, P., and König-Ries, B. (2011) Open Social Student Modeling: Visualizing Student Models with Parallel Introspective Views. In: Proceedings of 19th International Conference on User Modeling, Adaptation, and Personalization, UMAP 2011, Girona, Spain, Springer-Verlag, pp. 171-182. •  Hsiao, I.-H., Guerra, J., Parra, D., Bakalov, F., König-Ries, B., and Brusilovsky, P. (2012) Comparative Social Visualization for Personalized E-learning. Proceedings of the Working Conference on Advanced Visual Interfaces, AVI 2012, Capri, Italy, ACM Press, pp. 303-307. 58University of Pittsburgh - PAWS Lab