Online learning platforms have grown tremendously in recent years, having an impact from K-12 to lifelong learning. Come learn about Recommender System theory applied into practice to the domain of Online Education. This talk presents the algorithms behind course recommendations with insights drawn from large scale A/B Testing experiments.

1. “Learning to be Relevant”
Course Recommendations for
Shivani Rao
Senior Software Engineer,
Learning Relevance, LinkedIn

2. Learning Relevance Team
Sachin Rajendra
Shivani Rao
Deepak Kumar
(Senior Manager)
Dhritiman Das
Mahesh JoshiKonstantin Salomatin
Ashish Bhutani
Vladislav Tcheprasov
Christopher Lloyd
(New york)
Vinayak Shukl
Fares Hedayati
India
Bay Area, CA
Gungor Polatkan
(Manager)

3. The Problem of Skills-gap is Real

4. Need for Lifelong Learning
• Learning does not stop after school.
• Students and professionals need to:
• Find a job by acquiring relevant skills
• Keep a job: Re-skilling to staying relevant in a
dynamic job market
• Grow in their career or transition to their dream
role/ job

5. Rise of Online Education
Research on Online
Education has also
increased in the past
few years

6. www.lynda.com
▪ LinkedIn acquired
Lynda.com in April
2015 to about $1.5B
▪ Subscription based
business- once you
subscribe
(monthly/yearly), all of
the content is
unlimited.
▪ LinkedIn’s mission:
Provide economic
opportunity for
everyone.

7. www.linkedin.com/learning
▪ In September 2016
we launched
LinkedIn Learning
for members
(consumer) and
enterprises
▪ Provide Economic
Opportunity for
everyone via
learning

8. Goal
• Learn about the algorithms that power
Course Recommendation Algorithms for an
Online Education platform
• Learn about new and upcoming problems in
the online education space

9. Cold-Start Models

10. Member to Course Recommendations
Used for the offline email flows
-LiL Desktop/Mobile
-Flagship Neptune/Voyager/Titan Feed
-Self/Non-Self/Public Profile
-Enterprise

11. Skill Affinity Based Course
Recommendations
LinkedIn Confidential ©2016 All Rights Reserved

12. Member to Skill Mapping
Only 30% of the members
have explicit skills in their
profile.
Use member’s inferred title and the
most distinct skills associated with it
based on member’s cohort
Coverage is doubled, +10% in Feed
Impressions, +6.2% improvement in
Feed-CTR

13. Course to Skill Mapping
• Only 2% of the skills
are covered by
manual tagging
• Solution: Learn a
supervised model
using the manual
tags as labels

14. Supervised Course to Skill Mapping
LinkedIn Confidential ©2016 All Rights Reserved

15. Offline Results
LinkedIn Confidential ©2016 All Rights Reserved
Human
Taxonomy
Supervised
Original
Denoised
Supervised
Recall of the high precision
human labeled positives by
the supervised models.
Improves both Feed
Impressions and CTR
by 5%.

16. Context
Annotations
• Trending in
your
title/industry/co
mpany/
globally
• Because of the
skill you
have/followed LinkedIn Confidential ©2016 All Rights Reserved

17. During Launch

18. Online Skill Follow Based
Recommendations
Reminder: Course Offline Recommendations

19. Warm-Start Models

20. Supervised Offline Model Training

21. Features
Demography
Job seeker
subscription
titles
skills
industry
Location
Author
category
CTR
Type
skills
Skill
Overlap
Impressions/
Engagement
Engagements
with similar
course

22. Scoring Pipeline: Generating Course
Recommendations
+10% CTR over skill-overlap based models

23. Collaborative Filtering
Feed Click Data
LinkedIn Learning
Engagements
Completed/ Watched
Courses
Tracking
Data
User-
Course
Matrix
• ALS
• Co-occurence Matrix
• 8% Improvement in Send to
Click Rate in Weekly Emails
• 8% Improvement in learner
engagement

24. Carousel Ranking
24
-70 -60 -50 -40 -30 -20 -10 0 10 20 30 40
BECAUSE YOU WATCHED
EDITORS PICKS
FEATURED LEARNINGPATHS
INFLUENCER
NEW COURSES
RECOMMENDATIONS ALL
SHORT RELEVANT
SIMILAR TO BOOKMARKED
SKILLS EXISTING
SKILLS NEW
TITLE RECOMMENDATIONS
TRENDING INDUSTRY
TRENDING TITLE
Not all carousels are created equal

25. Carousel Ranking Contd’
• Order the carousel based
on a member’s CTR
25
+0.6 % Improvement in
Engagement compared to
static ordering of carousels

26. Micro-content for Micro-learning

27. Why Micro-content
(or bite-sized content)?
• Serving videos to
learners led to 10.4%
increase in engagement
Just-in-time learning vs Just-in-
case learning

28. Ongoing Projects
• Identifying Micro-content: Stand-alone Video
classifier
• Identifying skills associated with Micro-content

29. Concluding Thoughts

30. Future of Online Education?
“I do not have any formal training for the position, but I have read
all the relevant Wikipedia articles”

31. Thank You
Questions?
Email: sirao@linkedin.com