Recked is an informal event for engineers interested in recommender systems

1. RECKED
Jim Shur
Amsterdam, The Netherlands
26 Jan 2009
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ
N

2. ^ÖÉåÇ~
! qçé=päáÅÉW=tÜ~í=áë=píê~åÇë
! qÜÉ=jÉ~í=Eçê=îÉÖÖáÉëFW=
====J=_ìáäÇáåÖ=~=oÉÅçããÉåÇÉê
! _çííçã=päáÅÉW=píê~åÇë=pçÅá~ä=oÉÅçããÉåÇÉê
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ
O

3. tÜ~í=áë=píê~åÇë
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ
P

4. tÜ~í=áë=píê~åÇë
píê~åÇë=ÇÉîÉäçéë=
êÉÅçããÉåÇ~íáçå=íÉÅÜåçäçÖáÉë=íç=
ÜÉäé=éÉçéäÉ=ÇáëÅçîÉê=åÉï=íÜáåÖë
aÉîÉäçéÉêëW= _ìëáåÉëëÉëW=
cêÉÉ=~åÇ=É~ëó=íç=~ÇÇ= båÇ=ìëÉêëW=
qÜêçìÖÜ=~=ÜçëíÉÇ=ëçäìíáçåI=
êÉÅçããÉåÇ~íáçå= píê~åÇëKÅçã
píê~åÇë=ÜÉäéë=ÄìëáåÉëëÉë=
ïáÇÖÉíë=ïáíÜ=çìê=^mfë= ãçåÉóKpíê~åÇëKÅçã=
ÇÉäáîÉê=íÜÉ=êáÖÜí=ÅçåíÉåí=íç=
íÜÉ=êáÖÜí=éÉçéäÉ=~í=íÜÉ=
cçê=ÇÉîÉäçéÉêë=~åÇ=
êáÖÜí=íáãÉ
ëã~ää=éêçàÉÅíë
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ
Q

5. mêÉëÉåíáåÖ
qÜÉ=låÉJäáåÉJçÑJ`çÇÉ=
oÉÅçããÉåÇÉê
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ
R

6. my $recdata =
`grep -h -e quot;$seedquot; $dataDir/alf_*.txt
| tr , 'n'
| grep -v -e quot;$seedquot; -e quot;^imdb_quot;
| sort
| uniq -c
| sort -k1,1nr
| head`;
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ
S

7. `çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ
T

8. ^ic
! ^ëëçÅá~íáçå=iáëí=cçêã~í
! iáëíë=çÑ=áíÉãë=Ñçê=ïÜáÅÜ=óçì=Ü~îÉ=ëçãÉ=êÉ~ëçå=
íç=ÄÉäáÉîÉ=íÜ~í=É~ÅÜ=áíÉã=ÅçìäÇ=ëÉêîÉ=~ë=~=
êÉÅçããÉåÇ~íáçå=Ñçê=íÜÉ=çíÜÉê
! _ÉÜ~îáçê~ä
! jìëáÅ=éä~óäáëíëI=éä~ó=ëíêÉ~ãëI=Ñ~îçêáíÉëI=KKK
! mìêÅÜ~ëÉ=Ä~ëâÉíëI=é~ÖÉ=îáëáíëI=KKK
! `çããçå=ãÉí~Ç~í~=EÅçäÇ=ëí~êíF
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ
U

9. labs:~/cgi-bin/data/movie$ wc -l alf_*.txt
214711 alf_movie_imdb_actors.txt
129865 alf_movie_imdb_actresses.txt
24970 alf_movie_imdb_cinematographers.txt
19018 alf_movie_imdb_composers.txt
7120 alf_movie_imdb_costume-designers.txt
47145 alf_movie_imdb_directors.txt
28048 alf_movie_imdb_distributors.txt
24544 alf_movie_imdb_editors.txt
24886 alf_movie_imdb_keywords.txt
13691 alf_movie_imdb_locations.txt
11639 alf_movie_imdb_miscellaneous-companies.txt
65584 alf_movie_imdb_miscellaneous.txt
57846 alf_movie_imdb_producers.txt
34519 alf_movie_imdb_production-companies.txt
7574 alf_movie_imdb_production-designers.txt
1333 alf_movie_imdb_special-effects-companies.txt
57309 alf_movie_imdb_writers.txt
769802 total
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ
V

10. ^ëëçÅá~íáçå=iáëíë
imdb_actors_Bogaert%2C Frans,Bleeke Bet (1923),Bulldog Drummond (1922),Fate's Plaything (1920)
imdb_actors_Bogaerts%2C Sam,Ellektra (2004),Sicilia%2C La (1997),Steve + Sky (2004),Witte%2C De (1980)
imdb_actors_Bogan%2C Hugh,Men Cry Bullets (1997),Zoo Radio (1990)
imdb_actors_Bogard%2C Jerrod,Gatorman (2006),LovecraCked! The Movie (2006)
imdb_actors_Bogart%2C Humphrey,Across the Pacific (1942),Action in the North Atlantic (1943),African Queen%2C The (1951),...
imdb_costume-designers_Adamcova%2C Ruzena,Frona (1954),Igelfreundschaft%2C Die (1962),Jestrab kontra Hrdlicka (1953)
imdb_costume-designers_Adams%2C Annalisa,Billy Schulz (2006),Coffee Date (2006),Confessions of a Late Bloomer (2005)...
imdb_costume-designers_Adams%2C Katherine,Safe Side%2C The (2005),Trip%2C The (2003)
imdb_costume-designers_Adams-Galligan%2C Kimberly,House Divided%2C A (1998),Minotaur (1994),Minus Man%2C The (1999)
imdb_costume-designers_Adams-Santos%2C Stephanie,Last Good Man%2C The (2006),Model Man (2006)
imdb_directors_Spielberg%2C Steven (I),,Amistad (1997),Artificial Intelligence: AI (2001),Close Encounters of the Third Kind ...
imdb_directors_Spierig%2C Peter,Big Picture%2C The (2000),Daybreakers (2008),Undead (2003)
imdb_directors_Spires%2C Tony,Tears of a Clown (2005),Two Degrees (2001)
imdb_directors_Spiridakis%2C Tony,Last Word%2C The (1995),Noise (2004),Tinseltown (1997)
imdb_directors_Spiridellis%2C Evan,Ahnuld for Governor (2003),Big Box Mart (2005),Second Term (2005),This Land (2004)
mdb_keywords_dog-sled,Alaska: Spirit of the Wild (1997),Alaska's Silver Millions (1936),Arizona Dream (1993)...
imdb_keywords_newlywed,10 (1979),Above Suspicion (1943),Another Day%2C Another Man (1966)
imdb_keywords_surrealism,2001: A Space Odyssey (1968),300 (2006),8? (1963),8 X 8: A Chess Sonata in 8 Movements (1957)...
imdb_keywords_whimsical,10 (1979),10 Things I Hate About You (1999),13 Going on 30 (2004),Addams Family%2C The ,,,
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ
NM

11. fíÉã=J=fíÉã=
! eÉêÉ=ïÉ=~êÉ=ÇáëÅìëëáÄäÉåÖ=?áíÉãJáíÉã?=
êÉÅçããÉåÇ~íáçåë
! mÉçéäÉ=ïÜç=äáâÉ=EÄçìÖÜíI=ìëÉI=KKKF=íÜáëI=ïÉ=íÜáåâ=
ïáää=äáâÉ=EÄìóI=ìëÉI=KKKF=íÜ~í
! iáâÉ=~=ÄáÖ=ã~íêáñ=ïáíÜ=~ää=íÜÉ=áíÉãë=~ë=ÄçíÜ=
êçïë=~åÇ=Åçäìãåë
! b~ÅÜ=Éåíêó=E~IÄF=áë=~=äáâÉäáÜççÇ=íÜ~í=Ä=áë=~=
ÖççÇ=êÉÅçããÉåÇ~íáçå=Ñçê=~
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ
NN

12. grep -h -e quot;$seedquot; $dataDir/alf_*.txt
The seed is that from which we are
generating the recommendations, in this
case “Casablanca (1942)”
Hence this first step returns all the lists from all
the ALF files that contain “Casablanca (1942)”
(imagine huge list of lists, here)
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ
NO

13. ... | tr , 'n' | grep -v -e quot;$seedquot; -e quot;^imdb_quot;
• Pipe all those lists into tr, converting the commas into newlines,
putting each movie on its own line, then remove the list
headers and the seed itself
...
13th Letter%2C The (1951)
633 Squadron (1964)
Mr. Brown Comes Down the Hill (1965)
Fox%2C The (1967)
imdb_writers_Epstein%2C Julius J.
Letter from an Unknown Woman (1948)
Happy Deathday (1968)
In Our Time (1944)
Casablanca (1942)
In This Our Life (1942)
Intimate Stranger%2C The (1956)
Journey Out of Darkness (1967)
Letter%2C The (1940)
Mission to Moscow (1943)
Greengage Summer%2C The (1961)
... `çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ
NP

14. ... | sort | uniq -c
• Sort the list and qive to uniq -c to replace duplicate lines
with a single line plus the count of occurrences
...
1 Angel's Flight (1965)
1 Angel's Holiday (1937)
1 Angela's Ashes (1999)
1 Angels One Five (1952)
2 Angels Over Broadway (1940)
1 Angels Wash Their Faces%2C The (1939)
1 Angels of Death Island%2C The (2003)
14 Angels with Dirty Faces (1938)
3 Anger Management (2003)
3 Angst essen Seele auf (1974)
1 Angus (1995)
2 Anima (1998)
...
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ
NQ

15. ... | sort -k1,1nr | head
• Now sort again, numerically in reverse order, by the
counts. Then take the top of the list, and those are your
recommendations!
14 Angels with Dirty Faces (1938)
14 To Have and Have Not (1944)
13 Conspirators%2C The (1944)
13 Now%2C Voyager (1942)
12 Maltese Falcon%2C The (1941)
12 Passage to Marseille (1944)
12 Special Agent K-7 (1936)
11 Big Sleep%2C The (1946)
11 Dark Victory (1939)
11 Key Largo (1948)
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ
NR

16. `çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ
NS

17. oÉîáÉï
! fíÉã=ñ=áíÉã=êÉÅçããÉåÇ~íáçåë=êÉíìêå=íÜÉ=ãçëí=
~ëëçÅá~íÉÇ=áíÉãë=íç=~=ÖáîÉå=ëÉÉÇ
! lêÖ~åáòÉ=óçìê=áíÉãë=áå=~ëëçÅá~íáçåë=äáëíë
! cáåÇ=~ää=íÜÉ=äáëíë=ïáíÜ=íÜÉ=ëÉÉÇ
! oÉíìêå=íÜÉ=áíÉãë=íÜ~í=~ééÉ~ê=çå=íÜÉ=ãçëí=äáëíë===
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ
NT

18. bñéä~å~íáçåë
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ
NU

19. `çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ
NV

20. `çåëáÇÉê~íáçåë
! qÜÉ=a~í~
! tÉáÖÜíáåÖ
! mçéìä~êáíó=L=råéçéìä~êáíó
! mÉêëçå~äáò~íáçå=C=mêÉëÉåí~íáçå
! j~âáåÖ=áí=c~ëí
! qÉëíáåÖ
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ
OM

21. qÜÉ=a~í~
! qÜÉ=a~í~=Ü~ë=íç=ÅçãÉ=Ñêçã=ëçãÉïÜÉêÉI=~åÇ=
óçì=éêçÄ~Ääó=åÉÉÇ=~=äçí=çÑ=áí
! qÜÉ=ÄÉëí=~äÖçêáíÜã=ïáíÜ=åç=Ç~í~=áë=~ää=ÇêÉëëÉÇ=
ìé=ïáíÜ=åç=éä~ÅÉ=íç=Öç=EÄìí=ÅçåîÉêëÉäóKKKF
! håçïáåÖ=ïÜÉå=íïç=áíÉãë=~êÉ=?íÜÉ=ë~ãÉ?
! j~íÅÜáåÖ=íç=~=Å~í~äçÖ
! _É=ÅêÉ~íáîÉ
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ
ON

22. tÉáÖÜíáåÖ
! pÅçêÉë=ÇçåÛí=Ü~îÉ=íç=ÄÉ=N=éçáåí=éÉê=çÅÅìêêÉåÅÉ
! iáëí=ïÉáÖÜíáåÖ
! fë=ëÜ~êáåÖ=~=ÇáêÉÅíçê=ãçêÉ=áãéçêí~åí=íÜÉå=
ëÜ~êáåÖ=~=ÅçëíìãÉ=ÇÉëáÖåÉê
! qÉêã=ïÉáÖÜíáåÖ
! fë=NM=éä~óäáëí=ÅçJçÅÅìêêÉåÅÉë=ïáíÜ=`çäÇéä~ó=
ïçêíÜ=íÜÉ=ë~ãÉ=~ë=NM=ÅçJçÅÅìêêÉåÅÉë=ïáíÜ=
gçå~íÜ~å=oáÅÜã~å
! j~óÄÉI=ã~óÄÉ=åçí=JJJ=ãçêÉ=ä~íÉê==
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ
OO

23. mçéìä~êáíó
! _ÉÜ~îáçê~ä=Ç~í~
! qÜÉ=?`çäÇéä~ó?=mêçÄäÉã
! ^ëëçÅá~íáçåë=îëK=`çêêÉä~íáçåë
! jçêÉ=~ëëçÅá~íÉÇ=Äìí=äÉëë=ÅçêêÉä~íÉÇ
25 C
B A
800
23
100
27
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ
OP

24. mçéìä~êáíó
! pçäìíáçåW=qÉêã=tÉáÖÜíáåÖ=Et^icF
! list1,A,B:0.2,C:0.3
! qÉêã=cêÉèìÉåÅó=J=fåîÉêëÉ=açÅìãÉåí=cêÉèìÉåÅó
! qcJfac
! ^äëç=ëçäîÉë=aáäìíÉÇ=iáëí=mêçÄäÉã
! ^ëëçÅá~íáçåë=ã~ó=ÄÉ=äÉëë=ãÉ~åáåÖÑìä=áå=îÉêó=
ä~êÖÉ=äáëíë
! pÉÉ=~äëçW=mÉêëçå~äáò~íáçå
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ
OQ

25. råéçéìä~êáíó
! oÉÅçããÉåÇ~íáçåë=Ñçê=~å=áíÉã=çå=çåÉ=çê=îÉêó=
ÑÉï=äáëíë
! ^â~=íÜÉ=?ïÜó=íÜÉ=ÜÉÅâ=ÇáÇ=óçì=êÉÅçããÉåÇ=
íÜáë?=éêçÄäÉã
! jççåäáÖÜí=pçå~í~=JJ[=qçñáÅ=E_êáíåÉó=péÉ~êëF
! pçäìíáçåW=qeobpelia
! oÉèìáêÉ=ëçãÉ=ãáåáãìã=ëÅçêÉ=L=ÉîáÇÉåÅÉ
! qÜêÉëÜçäÇ=J=`çîÉê~ÖÉ=íê~ÇÉçÑÑ
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ
OR

26. mÉêëçå~äáò~íáçå
! êÉÅEìëÉêJNI=ëÉÉÇF=!=êÉÅEìëÉêJOI=ëÉÉÇF
! rëÉ=Üáëíçêó=~åÇ=éêçÑáäÉ=Ñçê=áãéäáÅáí=ÑáäíÉêáåÖ=çê=
ïÉáÖÜíáåÖ=E~äêÉ~Çó=ÄçìÖÜíI=îáëáíÉÇ=êÉÅÉåíäóI=KKKF
! ^ääçï=ÉñéäáÅáí=ÑáäíÉêáåÖI=Åìëíçãáò~íáçå=EëÉÉW=éçéF
! cÉÉÇÄ~Åâ=J=Ñ~îçêáíÉëI=Ä~ååáåÖ
! pÉÉÇ=ÖÉåÉê~íáçå=L=~ìÖãÉåí~íáçå
! =x=ÑEìëÉêF=Z=êÉÅçããÉåÇ~íáçåë=z
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ
OS

27. mêÉëÉåí~íáçå
! dççÇ=îëK=rëÉÑìä
! lÄîáçìëåÉëë
! péêáåÖëíÉÉå=quot;_çêå=íç=oìåÒ=J[=péêáåÖëíÉÉå=
quot;qÜìåÇÉê=oç~ÇÒ=Eilqo=f=J[=ilqo=ffF
! aáîÉêëáíó
! kçí=~ää=ëÜçÉë=Äó=íÜÉ=ë~ãÉ=ÇÉëáÖåÉê
! cçê=ÑêÉèìÉåí=îáëáíÉêI=ÇçåÛí=ëÜçï=ë~ãÉ=
êÉÅçããÉåÇ~íáçåë=ÉîÉêó=íáãÉ
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ
OT

28. péÉÉÇ
! fÑ=åÉÅÉëë~êó> recommender
! mêÉJÅçãéìí~íáçå
offline
lists
association
matrix top n
correlations
! fåÅêÉãÉåí~ä=ìéÇ~íÉë per item
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ OU

29. qÉëíáåÖ
! pìÄàÉÅíáîÉ=~å~äóëáë
! eçäÇJçìíë
! _ÉÑçêÉ=L=^ÑíÉê
! ^J_=qÉëíáåÖ
! fåÇáêÉÅí=ãÉíêáÅë
! jÉ~ëìêáåÖ=ïÜ~íÉîÉê=?ëìÅÅÉëë?=ãÉ~åë=Ñçê=
óçìê=ëáíÉ
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ
OV

30. píê~åÇë=pçÅá~ä=oÉÅçããÉåÇÉê
! qÜÉ=ÅçêÉ=ÅçåÅÉéí=áë=ëíê~áÖÜíÑçêï~êÇ
! ^=ÑÉï=êÉ~äáíáÉë=ã~âÉ=áí=ãçêÉ=ÇáÑÑáÅìäí
! píê~åÇë=áë=áå=íÜÉ=ÄìëáåÉëë=çÑ=ã~âáåÖ=áí=~ë=É~ëó=
~ë=éçëëáÄäÉ
! áÜÉ~êíãçîáÉëKÅçã=~ÇÇÉÇ=êÉÅçããÉåÇ~íáçåë=
íç=íÜÉáê=ëáíÉ=áå=çåÉ=Üçìê
! q~äâ=íç=gÉë#ë=çê=Åçåí~Åí=ìë=Ñçê=ãçêÉ=áåÑç
! ÜííéWLLÄáòëçäìíáçåëKëíê~åÇëKÅçã
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ
PM

31. p~~p=oÉÅçããÉåÇÉê
dÉåÉêáÅ=ÉåÖáåÉ=Ñçê=ÉJÅçããÉêÅÉI=îáÇÉçI=~åÇ=ëçÅá~ä=ÅçåíÉåí=ëáíÉë
`lkcfabkqf^i=C=molmofbq^ov
PN
`çéóêáÖÜí=OMMPJOMMUI=píê~åÇëI=fåÅ

32. pçÅá~ä=oÉÅçããÉåÇÉê=^mf
Item visited
Recommended item visited
Item reviewed
Capturing behavior in a retail site Item added to shopping cart
Item added to favorites
Item added to wishlist
Item purchased
Content visited
Recommended content visited
Content reviewed
Capturing behavior in a content site Content searched
Content uploaded
Content downloaded
Get item recommendation
Get user recommendation
Getting recommendations Explain a recommendation
Session start
Session start/end Session end
Rate an item
Rating items Show item rating
Set user tags
Get user tags
Tagging users and items Set item tags
Get item tags
`lkcfabkqf^i=C=molmofbq^ov
PO
`çéóêáÖÜí=OMMPJOMMUI=píê~åÇëI=fåÅ

33. bñ~ãéäÉW=ÄÉÜ~îáçê=áå=êÉí~áä
! customerId
! itemId
Item visited Tracks an item that has been visited
! userId
by a user
! sessionId
/tracking/event_visited.sbs
! needResult
! customerId
! itemId
Recommended item ! userId
Tracks a recommended item that has
! reqId
visited been clicked
! sessionId
! needResult
/tracking/event_clickedrecommendation.sbs
! widgetId
! customerId
! itemId
Item reviewed Tracks an item reviewed or commented
! userId
by a user
! sessionId
/tracking/event_reviewed.sbs
! needResult
! customerId
Item added to ! itemId
Tracks an item that has been added to
! userId
shopping cart the shopping cart
! sessionId
! needResult
/tracking/event_addshoppingcart.sbs
`lkcfabkqf^i=C=molmofbq^ov
PP
`çéóêáÖÜí=OMMPJOMMUI=píê~åÇëI=fåÅ

34. aÉîÉäçéÉêë=páíÉ
ÜííéWLLÄáòëçäìíáçåëKëíê~åÇëKÅçãLíÉÅÜ
oÉÖáëíÉê=C dÉí=
`~éíìêÉ
`~í~äçÖ=ìéäç~Ç oÉÅçããÉåÇ~íáçåë
_ÉÜ~îáçê~ä=bîÉåíë
`lkcfabkqf^i=C=molmofbq^ov
PQ
`çéóêáÖÜí=OMMPJOMMUI=píê~åÇëI=fåÅ

35. bñ~ãéäÉ=rëÉêë
`lkcfabkqf^i=C=molmofbq^ov
PR
`çéóêáÖÜí=OMMPJOMMUI=píê~åÇëI=fåÅ

36. a~ëÜÄç~êÇ=^å~äóíáÅë
`lkcfabkqf^i=C=molmofbq^ov
PS
`çéóêáÖÜí=OMMPJOMMUI=píê~åÇëI=fåÅ

37. ^ÅâåçïäÉÇÖÉãÉåíë
! qÜ~åâë=íç=hÉå=_~äÇïáå=ïÜç=çêáÖáå~ääó=
ÅçåÅÉáîÉÇ=çÑ=íÜÉ=?çåÉJäáåÉ=êÉÅçããÉåÇÉê?=áå=
íÜáë=í~äâ
! qÜ~åâë=íç=t~âççé~=~åÇ=píê~åÇë=Ñçê=
ëéçåëçêáåÖ=ob`hba
! qÜ~åâë=íç=~ää=Ñçê=~ííÉåÇáåÖ>
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ
PT

38. `çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ
PU