Recomendação de Conteúdo para Redes Sociais Educativas

Content Recommenda,on Based on Data Mining in Adap,ve Social Networks Marcel Pinheiro Caraciolo mpc@cin.ufpe.brOrientador: Germano Crispim gcv@cin.ufpe.br

Who is Marcel ?Marcel Pinheiro Caraciolo - @marcelcaraciolo Sergipano, porém Recifense. Cientista Chefe da Orygens (http://orygens.com) Mestrando em Ciência da Computação no CIN/UFPE na área de mineração de dados Membro e Moderador da Celúla de Usuários Python de Pernambuco (PUG-PE) Minhas áreas de interesse: Computação móvel e Computação inteligente Meus blogs: http://www.mobideia.com (sobre Mobilidade desde 2006) http://aimotion.blogspot.com (sobre I.A. desde 2009) Jovem Aprendiz ainda nas artes pythonicas.... (desde 2007) 2

The Problem Provide features that can enhance online social learning environment One social software in special are recommender systems Several approaches have been applied to web-based educational systems [1] [2] But only a few recommender systems use data mining and explanations in the recommendations

Objec,ves Design a methodology for social recommender systems that incorporate different knowledge sources It can be structured such as messages, personal profile, timeline, questions , answers It can be non-structured study plans, disciplines, classes watched, relationships educational resourcesents in a social envi- 01 g a recommender en- (%)*++%&$%, 01 -&./&% 01different recommenda- ation called Crab [15]. !"#$%&" (%)*++%&$23*&4 al domains, including 56786- suggest online learn- 599786-eir preferences, knowl-terests. The system is 8&;*,+23*&ational social network 9/<"%,/&. =%)>&/?#%4dents, helping them to !"#$%&":47*. The innovative aspect ing on social networks ions at the given rec- components of Architecture proposal for the recom- It uses several Figure 1: the social networkudent more confidence mender system arning understanding. s use data mining and [5]. 3. CONTENT RECOMMENDATION IN ED-st we describe the re- UCATIONAL SOCIAL NETWORKS

Objec,vesDesign an improved explanation of the recommendation to the user Generally in the currently available systems, the recommendation only comes with an overall score Increase the recommendation acceptance rate in order to enhance the student experience in social networks You answered questions about X,Y, Z. You answered questions about X,Y, Z. You have A,B,C friends in common Your friend asked this question. Your friends P,X and Y follows him. You both live at the same state Pernambuco. You have X,Y, Z followers in common You are both interested in portuguese, maths and english.You have X,Y, Z concursos in common You have P,Q, R study groups in common. Because you have difﬁculty in portuguese, maths and english. Because it is a most searched course

What recommender systems really do? 1. Predict how much you would like a certain product or service 2. Suggest a list of N items ranked based on your interests3. Suggest a list of N users ranked for a product/ service4. Explain to you why those recommendations were given5. Adjust the prediction and the recommendation based on your feedback and from others.

Mee,ng Recommenda,on Systems Content Based Filtering Similar Text Portuguese Maths Biology Items Interpretation recommend likes Marcel Users

Mee,ng Recommenda,on Systems Collabora,ve Based Filtering Portuguese Literature Maths Physics Items like like recommends Marcel Rafael Amanda Users Similar

Mee,ng Recommenda,on Systems Our Approach -‐ Hybrid Recommender System Meta recommender system architectureCustomized control over the generation ofa recommendation list !"#$%*+,-)% !"#$%&$()#%It adapts in accordance to the structure of the data ./0#$-+1/% Recommendation of Friends ? More weight to Collaborative Filtering % % % % !"#$%#$&()%*&+,-$%.,#/& !"--(0".(12%&()%*&+,-$%.,#/& % Recommendation of Courses ? 2$,#/3"%456575689% % !"#"$%&&%()*&+,-(.&/,-0&+,-(.& :+-1/;%%<#+,=#% %12%&303#2,&(",&2,"&34& More weight to Content Filtering *+>)-$">,?;%%@$-3A-0#3% % % Cold Start: Mitigate using Popular Recommendations B#0-%<#+CC#/3#$% %% %&-$-C#0#$"%% accepted from another users <#+CC#/3-1/"% Feedback and temporal slicing: Learn from users and select limited results by time Figure 2: Meta Recommender Components Interac- tion be highly beneﬁcial given that students do not meet phys- ically. It may result in their becoming more socially con-

Methodology and Current Results !"#$%*+,-)% ./0#$-+1/% this engine with the popular brazilian social network AtéPassar Integrated More than 70.000 students registered studying for the public examinations Recommend StudyGroups, Friends,Video Classes, Questions and Concursos More than 70.000 items available for recommend % % !"--(0".(12%&()%*&+,-$%.,#/& % Written in Python using a open-source framework Crab !"#"$%&&%()*&+,-(.&/,-0&+,-(.& %12%&303#2,&(",&2,"&34& % Framework available for building recommender systems (My contribution) It is running since January 2011 In March B#0-%<#+CC#/3#$% was performed. 2011 , questionnaire %% %&-$-C#0#$"%% Liked Not Liked -1/"% 23%mender Components Interac- 77% Figure 3: AtePassar Recommender Syste face

Expected Results Improve the Learning Process Analyze how the recommenders can increase the learning process in online educational social networks. Mining all types of source in social networks in forms of recommendations Explore the Hidden Knowledge There are several knowledge sources in a educational social network. All those sources as basis for discovery novel content and learning resources Better Recommendation Understanding The explanations can be quite helpful to better understanding of the given resource Better learning interaction and simpliﬁcation of the human/computer interaction

ConclusionsProposal of a personalized recommender systemthat incorporate different types of source of knowledgeIt can be applied in adaptive social networksBring machine learning to web educational systemsHelp the students to ﬁnd an optimal path to learning resourcesHelp the researchers and designers how to designthe social network to achieve those tasks efﬁciently

Contribu)ons

Crab A Python Framework for Building Recommendation EnginesMarcel Caraciolo Ricardo Caspirro Bruno Melo @marcelcaraciolo @ricardocaspirro @brunomelo

What is Crab ? A python framework for building recommendation enginesA Scikit module for collaborative, content and hybrid ﬁltering Mahout Alternative for Python Developers :D Open-Source under the BSD license https://github.com/muricoca/crab

The current CrabCollaborative Filtering algorithms User-Based, Item-Based and Slope OneEvaluation of the Recommender Algorithms Precision, Recall, F1-Score, RMSE Precision-Recall Charts

The current Crab

The current Crab>>>#load the dataset

The current Crab>>>#load the dataset>>> from crab.datasets import load_sample_movies

The current Crab>>>#load the dataset>>> from crab.datasets import load_sample_movies>>> data = load_sample_movies()

The current Crab>>>#load the dataset>>> from crab.datasets import load_sample_movies>>> data = load_sample_movies()>>> data

The current Crab>>>#load the dataset>>> from crab.datasets import load_sample_movies>>> data = load_sample_movies()>>> data{DESCR: sample_movies data set was collected by the book called nProgramming the Collective Intelligence by Toby Segaran nnNotesn----- nThis data set consists ofnt* n ratings with (1-5) from n users to n movies., data: {1: {1: 3.0, 2: 4.0, 3: 3.5, 4: 5.0, 5: 3.0}, 2: {1: 3.0, 2: 4.0, 3: 2.0, 4: 3.0, 5: 3.0, 6: 2.0}, 3: {2: 3.5, 3: 2.5, 4: 4.0, 5: 4.5, 6: 3.0}, 4: {1: 2.5, 2: 3.5, 3: 2.5, 4: 3.5, 5: 3.0, 6: 3.0}, 5: {2: 4.5, 3: 1.0, 4: 4.0}, 6: {1: 3.0, 2: 3.5, 3: 3.5, 4: 5.0, 5: 3.0, 6: 1.5}, 7: {1: 2.5, 2: 3.0, 4: 3.5, 5: 4.0}}, item_ids: {1: Lady in the Water, 2: Snakes on a Planet, 3: You, Me and Dupree, 4: Superman Returns, 5: The Night Listener, 6: Just My Luck}, user_ids: {1: Jack Matthews, 2: Mick LaSalle, 3: Claudia Puig, 4: Lisa Rose, 5: Toby, 6: Gene Seymour, 7: Michael Phillips}}

The current Crab

The current Crab>>> from crab.models import MatrixPreferenceDataModel

The current Crab>>> from crab.models import MatrixPreferenceDataModel>>> m = MatrixPreferenceDataModel(data.data)

The current Crab>>> from crab.models import MatrixPreferenceDataModel>>> m = MatrixPreferenceDataModel(data.data)>>> print mMatrixPreferenceDataModel (7 by 6) 1 2 3 4 5 ...1 3.000000 4.000000 3.500000 5.000000 3.0000002 3.000000 4.000000 2.000000 3.000000 3.0000003 --- 3.500000 2.500000 4.000000 4.5000004 2.500000 3.500000 2.500000 3.500000 3.0000005 --- 4.500000 1.000000 4.000000 ---6 3.000000 3.500000 3.500000 5.000000 3.0000007 2.500000 3.000000 --- 3.500000 4.000000

The current Crab

The current Crab>>> #import pairwise distance

The current Crab>>> #import pairwise distance>>> from crab.metrics.pairwise import euclidean_distances

The current Crab>>> #import pairwise distance>>> from crab.metrics.pairwise import euclidean_distances>>> #import similarity

The current Crab>>> #import pairwise distance>>> from crab.metrics.pairwise import euclidean_distances>>> #import similarity>>> from crab.similarities import UserSimilarity

The current Crab>>> #import pairwise distance>>> from crab.metrics.pairwise import euclidean_distances>>> #import similarity>>> from crab.similarities import UserSimilarity>>> similarity = UserSimilarity(m, euclidean_distances)

The current Crab>>> #import pairwise distance>>> from crab.metrics.pairwise import euclidean_distances>>> #import similarity>>> from crab.similarities import UserSimilarity>>> similarity = UserSimilarity(m, euclidean_distances)>>> similarity[1]

The current Crab>>> #import pairwise distance>>> from crab.metrics.pairwise import euclidean_distances >>> #import similarity >>> from crab.similarities import UserSimilarity >>> similarity = UserSimilarity(m, euclidean_distances) >>> similarity[1] [(1, 1.0),(6, 0.66666666666666663),(4, 0.34054242658316669),(3, 0.32037724101704074),(7, 0.32037724101704074),(2, 0.2857142857142857),(5, 0.2674788903885893)]

The current Crab>>> #import pairwise distance>>> from crab.metrics.pairwise import euclidean_distances >>> #import similarity >>> from crab.similarities import UserSimilarity >>> similarity = UserSimilarity(m, euclidean_distances) >>> similarity[1] [(1, 1.0),(6, 0.66666666666666663), MatrixPreferenceDataModel (7 by 6) 1 2 3 4 5(4, 0.34054242658316669), 1 3.000000 4.000000 3.500000 5.000000 3.000000(3, 0.32037724101704074), 2 3.000000 4.000000 2.000000 3.000000 3.000000 3 --- 3.500000 2.500000 4.000000 4.500000(7, 0.32037724101704074), 4 2.500000 3.500000 2.500000 3.500000 3.000000 5 --- 4.500000 1.000000 4.000000 ---(2, 0.2857142857142857), 6 3.000000 3.500000 3.500000 5.000000 3.000000(5, 0.2674788903885893)] 7 2.500000 3.000000 --- 3.500000 4.000000

The current Crab

The current Crab>>> from crab.recommenders.knn import UserBasedRecommender

The current Crab>>> from crab.recommenders.knn import UserBasedRecommender>>> recsys = UserBasedRecommender(model=m,similarity=similarity, capper=True,with_preference=True)

The current Crab>>> from crab.recommenders.knn import UserBasedRecommender>>> recsys = UserBasedRecommender(model=m,similarity=similarity, capper=True,with_preference=True)>>> recsys.recommend(5)array([[ 5. , 3.45712869], [ 1. , 2.78857832], [ 6. , 2.38193068]])

The current Crab>>> from crab.recommenders.knn import UserBasedRecommender>>> recsys = UserBasedRecommender(model=m,similarity=similarity, capper=True,with_preference=True)>>> recsys.recommend(5)array([[ 5. , 3.45712869], [ 1. , 2.78857832], [ 6. , 2.38193068]])>>> recsys.recommended_because(user_id=5,item_id=1)array([[ 2. , 3. ], [ 1. , 3. ], [ 6. , 3. ], [ 7. , 2.5], [ 4. , 2.5]])

The current Crab>>> from crab.recommenders.knn import UserBasedRecommender>>> recsys = UserBasedRecommender(model=m,similarity=similarity, capper=True,with_preference=True)>>> recsys.recommend(5)array([[ 5. , 3.45712869], [ 1. , 2.78857832], [ 6. , 2.38193068]])>>> recsys.recommended_because(user_id=5,item_id=1)array([[ 2. , 3. ], [ 1. , 3. ], MatrixPreferenceDataModel (7 by 6) 1 2 3 4 5 ... [ 6. , 3. ], 1 3.000000 4.000000 3.500000 5.000000 3.000000 2 3.000000 4.000000 2.000000 3.000000 3.000000 [ 7. , 2.5], 3 --- 3.500000 2.500000 4.000000 4.500000 [ 4. , 2.5]]) 4 2.500000 3.500000 2.500000 3.500000 3.000000 5 --- 4.500000 1.000000 4.000000 --- 6 3.000000 3.500000 3.500000 5.000000 3.000000 7 2.500000 3.000000 --- 3.500000 4.000000

Distributed Computing with mrJob https://github.com/Yelp/mrjob

Distributed Computing with mrJob https://github.com/Yelp/mrjobIt supports Amazon’s Elastic MapReduce(EMR) service, your own Hadoop cluster or local (for testing)

Distributed Computing with mrJob https://github.com/Yelp/mrjob """The classic MapReduce job: count the frequency of words. """ from mrjob.job import MRJob import re WORD_RE = re.compile(r"[w]+") class MRWordFreqCount(MRJob): def mapper(self, _, line): for word in WORD_RE.findall(line): yield (word.lower(), 1) def reducer(self, word, counts): yield (word, sum(counts)) if __name__ == __main__: MRWordFreqCount.run()It supports Amazon’s Elastic MapReduce(EMR) service, your own Hadoop cluster or local (for testing)

Studies with Sparse Matrices Real datasets come with lots of empty values http://aimotion.blogspot.com/2011/05/evaluating-recommender-systems.htmlSolutions: scipy.sparse package Sharding operations Matrix Factorization techniques (SVD) Apontador Reviews Dataset

Studies with Sparse Matrices Real datasets come with lots of empty values http://aimotion.blogspot.com/2011/05/evaluating-recommender-systems.html Solutions: scipy.sparse package Sharding operations Matrix Factorization techniques (SVD) Crab implements a MatrixFactorization with Expectation Maximization algorithm Apontador Reviews Dataset

Studies with Sparse Matrices Real datasets come with lots of empty values http://aimotion.blogspot.com/2011/05/evaluating-recommender-systems.html Solutions: scipy.sparse package Sharding operations Matrix Factorization techniques (SVD) Crab implements a MatrixFactorization with Expectation Maximization algorithm scikits.crab.svd package Apontador Reviews Dataset

Why migrate ?Old Crab running only using Pure Python Recommendations demand heavy maths calculations and lots of processingCompatible with Numpy and Scipy libraries High Standard and popular scientific libraries optimized for scientific calculations in PythonScikits projects are amazing! Active Communities, Scientific Conferences and updated projects (e.g. scikit-learn)Turn the Crab framework visible for the community Join the scientific researchers and machine learning developers around the Globe coding with Python to help us in this project Be Fast and Furious

How are we working ? Sprints, Online Discussions and Issueshttps://github.com/muricoca/crab/wiki/UpcomingEvents

How are we working ? Our Project’s Home Pagehttp://muricoca.github.com/crab

Future Releases Planned Release 0.1 Collaborative Filtering Algorithms working, sample datasets to load and test Planned Release 0.11 Evaluation of Recommendation Algorithms and Database Models support Planned Release 0.12 Recommendation as Services with REST APIs....

Join us!1. Read our Wiki Page https://github.com/muricoca/crab/wiki/Developer-Resources2. Check out our current sprints and open issues https://github.com/muricoca/crab/issues3. Forks, Pull Requests mandatory4. Join us at irc.freenode.net #muricoca or at our discussion list http://groups.google.com/group/scikit-crab

Recommended booksToby Segaran, Programming Collective SatnamAlag, Collective Intelligence inIntelligence, OReilly, 2007 Action, Manning Publications, 2009 Sites such as TechCrunch and ReadWriteWeb

Recommended Conferences- ACM RecSys.–ICWSM: Weblogand Social Media–WebKDD: Web Knowledge Discovery and Data Mining–WWW: The original WWW conference–SIGIR: Information Retrieval–ACM KDD: Knowledge Discovery and Data Mining–ICML: Machine Learning

plore more the learning resource materials, one of the goals of the e-learning system. Conference, Bratislava, 2005. 229-234. [7] Romero, C., Ventura, S.: Educational Data Mining: a References 3.5.3 Better Recommendation Understanding The explanations can be quite helpful for users to better understand if the given resource is (or is not ) suitable for Survey from 1995 to 2005. Expert Systems with Applications. Elsevier 1:33, 2007. 135-146. [8] Romero, C., Ventura, S.: Data mining in e-learning. th the 5. REFERENCES Wit Press, 2006. them. Those explanations can be beneficial in how to for- er en- mulate Brusilovsky, P., Peylo, C.: Adaptive and to simply and [1] a better learning interaction and hence Intelligent [9] Atepassar. Available at: http://atepassar.comed the shorten the human/computer Systems. International Journal Web-based Educational interaction. [10] J. B. Schafer, J. A. Konstan, and J. Riedl.:rought of Artificial Intelligence in Education. 13, 2003, 156 - E-commerce recommendation applications. Data Miningquality 169. and Knowledge Discovery, 2001.nalized 4. [2] Conole, G. & Culver, J.: The design of Cloudworks: CONCLUSIONS [11] Shen, L.P., Shen, R.M.: Learning Content In this Applying social networking practice to foster the sys- paper, we presents a personalized recommender Recommendation Service Based-on Simple Sequencing tem that incorporate different types of sourcesand knowledge exchange of learning and teaching ideas of designs In: Specification. In Proceedings of Advanced in Web-based in a adaptative educational social network by3using a meta- Computers & Education , Vol. 54 , Nr. Elsevier, 2010 Learning 2004 363- recommender approach. We believe that this technique will , S. 679-692 . bring more personalized andE., Grigoriadou, M., Samarakou, [12] M. Montaner, B. L’opez, and J. Rosa.: A taxonomy of orking [3] Gogoulou, A., Gouli, better recommendations for the recommender agents on the internet. In Artificial students. and Chinou, D.: A Web-based Educational Setting M.,mmen- Intelligence Review, volume 19, pages 285-330. Kluwer ing to Supporting Individualized Learning, Collaborative Academic, 2003. Although we have integrated the In ProceedingsAtePassar, it Learning and Assessment. system in the oftion of [13] J. Herlocker, J. Konstan, and J. Riedl.: Explaining e that can, in Educational Technology & Society. 2007, 242-256. principle, also be used in other web-based educational systems. MoreJ.B.: The application of data-mining the area [4] Schafer, research need to be conducted in to collaborative filtering recommendations. In Ineduca- proceedings of ACM 2000 Conference on Computermprove of the recommendation systemsJ. Wang (Ed.), Encyclopedia recommender systems. In in social learning and how they can data warehousing and mining. Hershey, PA. Idea of influence the student satisfaction, motivation and Supported Cooperative Work, 2000. learning process. 44 - 48. Group 2005 [14] B. Schafer, J. Konstan, and J. Riedl.: [5] Farzan, R., Brusilovsky, P.: Social Navigation Support Meta-recommendation systems: user-controlled in a Course Recommendation System. In proceedings of Bringing machine learning and artificial intelligence to edu- integration of diverse recommendations. In Proceedingsrces in of the 11th conference on Information and knowledge of the cational social networks Conference on main contributions of 4th International is one of the Adaptive Hypermedia this work, Adaptive resulting on a new generation of web- and which is Web-based Systems. Dublin, 2006. management, pages 43-51. ACM Press, 2002. of the 91-100. educational systems. The task of recommendation in web- [15] Crab. Available at: http://muricoca.github.com/crabve themation based education A.: Recommender System for Adaptive col- [6] Ksristofic, can be considered as special type of [16] AtePassar Video. Available at:nd ex- laborative learning, due to the fact that they of Informatics Hypermedia Applications. In Proceeding help student http://www.youtube.com/watch?v=gzn7I1EJmb4e goals and Information Technology Student Research with similar interests to find an optimal path through the learning material and facilitate the 229-234. Conference, Bratislava, 2005. social network design to [7] Romero, C., Ventura, S.: Educational Data Mining: a be effectively be used to achieve these targets. We provide a videoSurvey from 1995 to recommender Systemsin action at demonstrating the 2005. Expert system with Applications. Elsevier 1:33, 2007. 135-146. AtePassar, which can be accessible at œ[16].better [8] Romero, C., Ventura, S.: Data mining in e-learning.ble for Wit Press, 2006.to for-ly and [9] Atepassar. Available at: http://atepassar.com [10] J. B. Schafer, J. A. Konstan, and J. Riedl.: E-commerce recommendation applications. Data Mining and Knowledge Discovery, 2001. [11] Shen, L.P., Shen, R.M.: Learning Contenter sys- Recommendation Service Based-on Simple Sequencingwledge

AtéPassar -‐ hOp://atepassar.com Data available for researching -‐ marcel@atepassar.com

Where will you be ? Source: Hunch.com Thanks !!

Content Recommenda,on Based on Data Mining in Adap,ve Social Networks Marcel Pinheiro Caraciolo marcelcaraciolo@acm.org

http://www.mobideia.com	910
http://flavors.me	3
http://translate.googleusercontent.com	1

Recomendação de Conteúdo para Redes Sociais Educativas

by Marcel Caraciolo on Oct 21, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

3 Embeds 914

Statistics

Recomendação de Conteúdo para Redes Sociais Educativas — Presentation Transcript