Romer Rosales is the Director of Artificial Intelligence at LinkedIn. He discusses how AI is used throughout LinkedIn's products and services to optimize member experiences. Key applications of AI include the home page feed, notifications, connections, and search. The goal is to automatically deliver the right information to the right user through the right channel. Originally, AI focused on single objectives like engagement, but now takes a holistic approach considering multiple objectives and tradeoffs across the ecosystem. Examples provided demonstrate how AI is used to balance objectives like reducing notifications while maintaining engagement.

1. Artificial Intelligence @ LinkedIn
Building Member Experiences
April 2018
​Romer Rosales
​Director of Artificial Intelligence
AI NEXTCon Silicon Valley 18
SANTA CLARA | APRIL 10-13
#ainextcon

3. Artificial Intelligence at LinkedIn
Vision
Every professional interaction with LinkedIn is
personal, relevant and helps in providing opportunity.
Mission
Automatically and optimally deliver the right
information to the right user at the right time through
the right channel

4. * More than 200+ countries | 1.5K fields of study | 600+ degrees | 24K titles ...
546M 20M 14M 50K (multi-lang)
11B endorsements
29K *

5. Talk Overview
▪ AI is in practically everything you see at LinkedIn
▪ From transactional to holistic optimization
– Optimizing transaction-item (single objective)
– Optimizing the ecosystem (multiple objectives)
▪ Building member experiences - Examples:
– Notifications
– Feed
– Connections
▪ Building a platform for AI
– Example: Online Model Selection
▪ Building scalable infrastructure for AI
– Example: Graph storage/compute

6. AI is in practically everything you see at LinkedIn
Home Page Feed

7. AI is in practically everything you see at LinkedIn
Connections

8. AI is in practically everything you see at LinkedIn
Notifications

9. AI is in practically everything you see at LinkedIn
Learning

10. AI is in practically everything you see at LinkedIn
Search

11. AI is in practically everything you see at LinkedIn
Jobs

12. AI is in practically everything you see at LinkedIn
SlideShare

13. And also in things you do not see
Anomaly
Detection

14. What is AI at LinkedIn?
Responsible for the framework for turning messy/ill-posed business
problems into optimal, well-posed and operational solutions
Product
Metrics
Relevance
Metrics
ML optim
framework
A/B testing
framework
ML platform,
automation, monitoring,...
Prod & Business
understanding,
Analytics, Data science
Scientific rigor,
statistical methodology

15. Building Member Experiences with AI

16. How Does Machine Learning/Statistics Help
Example:
A user u visits the Home Page. What should be served?
Core question:
What is the probability that a user u will value an item i ?

17. Single Objective: Serve the Most Valuable Item Now
.
.
.
.
.
.
.
.
.
.
.
.
Given a user u, maximize the probability of engaging with the served
item from all the available candidate items

18. Smart Replies
Increase in
messages sent,
reply rate,
CTR

19. Conversat.
(messages)
so far
Interlocutor
Attributes
(industry,
seniority, etc.)
Actual Reply
(train mode)
Candidate R.
(test mode)
Feature
Transformation
Feature
Transformation
Feature
Transformation
Statistical
Model
Statistical
Model
Predicted Reply for member u (maxi
pui
)
Under the Hood

20. Beyond a Single Objective
▪ The previous approach is not realistic
– It is almost never about a single value (e.g.; engagement)
▪ There are often conflicting objectives
– Feed engagement | revenue
– Notifications sent | total sessions | negative feedback
▪ These objectives are not directly comparable
– In general, there is a tradeoff
hbook.com

21. From
Artificial Intelligence done at the transactional-item level
Eg. Send notifications that maximize click/sessions
To
Artificial Intelligence done for the larger ecosystem
Eg. Send notifications that improve overall member experience
and ecosystem health

22. Obvious in Retrospect

23. Obvious in Retrospect

24. Air Traffic Controller (ATC) - Portfolio Optimization
Reduce notification send volume while maintaining value to our members (engagement)
(page_views↨, sessions ↨, complaints↓)
BUT ALSO
Require a minimum volume for each type of notification and maximum
volume of notification for each user (sendsj
↨, sends_to_useri
↧)
page_views >= interaction_target [interaction ~ user response]
sessions >= sessions_target [set of related interactions]
negative_feedback <= negative_feedback_tolerance
Minimize sends such that
[Gupta et al. Email Volume Optimization at LinkedIn. KDD 2016]
notifications_sent j
>= notification_targetj
(for all notification types j)
notifications_sent_useri
<= notifications_target_useri
(for all users i)

25. ATC at work (email volume optimization use-case)
Months
-53%
-40%
SEND VOLUME
COMPLAINT VOLUME
Actuals Projections (Organic)
Page-View Loss < 2%

26. ATC: From the Member’s POV
% of members receiving more than X emails a week
EMAILS A WEEK
%OFMEMBERS(CUMULATIVE)
...

27. Proactive Notifications/Emails
min
{ y }
u,k,c
∑ ∑ ∑ y u,k,c
u k c
0 ≤ y ≤ 1 ∀ (u,k,c).u,k,c
s.t.
● Start a conversation via a notification/email and trigger downstream exchanges
● Start a conversation with the member about something of interest to him/her (L2M)
○ Aim to generate downstream exchanges (e.g.; M2M or other L2M conversations)
minimize notifications/emails
global succesful_sessions
global negative_feedback
succesful_sessions driven by notification type k
negative_feedback per notification type k
Send with probability z (u:user, k: notification type, c: channel )
∑ ∑ ∑ g ( x ) y ≤ G
u k c
uk,c globalu,k,c
f ( x ) y ≥ F
u,k,cuk,c global
∑ ∑ ∑
u k c
f ( x ) y ≥ F , ∀k
u,k,cuk,c k
∑ ∑
u c
∑ ∑ g ( x ) y ≤ G , ∀k
u c
uk,c ku,k,c

28. Great but… Could do Better - Improving the Objective
○ Avoid redundancy] Many members visit organically and continue interacting naturally
○ [Anticipate interest] Some members do rely on our ability to anticipate interest and proactively
remind them
minimize sends [same]
s.t. [same negative constraints as before]
active_usersk
> Ak
, ∀k
Some users engage in
conversations often (are up
to date), but some depend
on LinkedIn to remind them
[Gupta et al. Optimizing Email Volume For Sitewide Engagement. CIKM 2017]

29. Why?

30. Revisiting Emails
New model achieves better distribution of emails across user segments
Months
-11%
-40%
SEND VOLUME COMPLAINT VOLUME
User Segment Daily Weekly Monthly Dormant
Send rate absolute difference
*compared to previous model
-54%↓ -6%↓ 0%↓ 8%↑
Page-View Loss < 0.1%
Months

31. A Conversational Feed
FN
TIME
Notification from connection (eg.
connection shared insights)
Notification-triggered visit
(“Stay connected’)
Increase conversations by:
1. Making the feed more tuned to (recent) actions
2. Surfacing events through Communication channels
MEMBER
EXPERIENCE
F
Member makes a connection N
Contextualized Feed
Email Notification
C
C Connection
p ( f ( x )) ∣ x ) - p ( v ∣ x ) > T ?k u u u u
Anticipatory model:
then trigger notification

32. Recommending Valuable Connections
Accept a Recommendation
Invite a new member
Ignore and Decline
Recommending the right connections to our members (L2M Conversation)
If member sends the invitation → L2M conversation
continues with immediate response about new invite
recommendations (who to invite next)
GOAL: USE CONTEXTUAL INFORMATION TO UNDERSTAND USER’S IMMEDIATE INTERESTS

33. Fast, contextualized recommendations have a large impact
on the network
Improvements of 17% in M2M connections made site-wide
(this is a really large number!)

34. Tuning Connection Recommendations to Member
Interaction
1. Simply forming more connections is not optimal
a. Instead encourage connections that will foster the long term M2M conversation
2. Multi-Objective Optimization using the connection “value”
a. Recommend connections to optimize long term conversations
b. Optimize the connection “value” (CVS) in addition to existing invites
∑max y pCVS pymkScoreij ij ij
ij
Probability of recommending member i to member j
∑s.t. y pymkScore ≥ Threshold
ij ij Invitations
ij
Bring conversations in the objective function
Preserve invite quality & volume
pCVS: Probability that the connection is of “value” (e.g. a conversation builder)
pymkScore: Likelihood that the connection will form
{yi j
}
+13% increase in M2M interactions (from newly formed connections).

35. Building an AI Platform

36. Machine Learning Platform at LinkedIn
Make the end-to-end ML development
easy, fast, robust and automatic
Explore a large space
of different variations
of a model
Model Creation Deployment
Deploy different
components of the model
and features in different
places in production
Maintenance
Continuously monitor
model health, data
quality and detect
anomalies

37. What Kind of Models
Example: deep learning, trees and GLMix
+ +
Global
Model
Per-User
Model
Per-Item
Model

38. Example
Member
STD Data
Member
Profile Text
Job
Post Text
Job
STD Data
User NNet
T. Embed.
Job NNet
T. Embed.
Interaction
Embed.
Per-Job
Model
Per-User
ModelForest
Sum
< 1 hour

39. Auto-Deploy And Maintain
MonitoringValidation Investigation

40. Example: Online Model Selection - Multiple Objectives
▪ In many machine learning applications, different models are combined
linearly using some weights or tuning parameters to generate a final score.
▪ Pain Points: total objective is a complex function of parameters, manual
tuning, need to launch/track several experiments...
▪ The weight vector controls the balance of the business
metrics: viral actions, job applications, clicks, etc.
▪ We are interested in solving, where utilities U can be evaluated (models)

41. Tracking Data
▪ In order to solve the problem we need to carefully track some information.
▪ denote whether the the i-th member during the j-th session
which was served by parameter x, did action k or not.
▪ We model this data as
where ni
(x) is the total number of sessions of member i which was served by x
and fi
is a latent function for the particular metric.

42. Problem Reformulation
Based on the modeling, we can now reformulate the optimization problem. We
estimate the utilities as
And try to solve the following optimization problem.

43. Illustration of the Thompson Sampling Algorithm
x
f(x)
(x*,f(x*))
x
f(x)

44. Illustration of the Thompson Sampling Algorithm
x
f(x)
* ** ***** ** ** ** ** * * x
f(x)
******* ***
Iterate...

45. Online Convergence Results

46. Building Scalable Online
Storage/Compute Infrastructure
for AI

47. Real-time AI: Graph Traversal/Computation
• A platform for real-time searches on large
graphs; used to serve nearline People You May
Know results 5x faster
• Uses include triangle closing for online PYMK,
community detection, random walks
simulation
• Better recommendations, faster = more
relevant member experienceTriangle
closing
Community
detection
Personalized
PageRank
REAL TIME

49. Recommendation Plugin Support: Graph Compute Framework
Graph Node Set
Edge Iterator &
Lambda
Node Set
A compute framework for manipulating node set while gathering information
during process
● Specify a node set
● Specify an iterator (edge or node iterator)
● Specify a function to operate on the iterator
● Generate another node set

50. How We Get Things Done
Long-term objectives
(return visits, advertising
revenue, sign-ups, job
applic,..)
Formulate objectives,
proxies (CTR, revenue/visit,
multiple-objectives, …)
Large scale
optimization via ML, UI
changes,...
Engage,
Experiment,
Learn,
Deploy,
Innovate