Computational Advertising Workshop @ ICML 2012

1. Computational
Advertising
in
Social Networks
Anmol Bhasin
Sr. Manager
Analytics Engineering
www.linkedin.com

2. Core Message
We live in fascinating times. Two new nascent technological
disciplines are coming together to transform how the
marketers go about their business of reaching consumers,
be it businesses or end users.
It is time for the practitioners in these disciplines to push
the envelope by creating innovative products and
sophisticated algorithms to define what the future will hold
in this new digitally social era..
Anmol Bhasin

3. Source : www.140proof.com

5. Our Mission
Connect the world’s professionals to make them
more productive and successful.
Our Vision
Create economic opportunity for every
professional in the world.
Value proposition
To make professionals better in the
Jobs that they are already in.

6. World’s largest professional network
Over 60% of members are now international
161+ M
90
~2/sec
New Members joining
82%
*
Fortune 100 Companies
use LinkedIn to hire
55
>2M
*
32
Company Pages
17
2
4
8
~4.2B
Professional searches in 2011
2004
2005
2006
2007
2008
2009
2010
LinkedIn Members (Millions)
*as of March 31, 2012
6

9. Challenges in Social Network Advertising
§ Information Seeking vs Information Consumption
§ Dedicated marketing channels for brand awareness required
§ Hypertargeting
§ Blending organic and sponsored content
§ Mobile ?

10. Sponsored Search vs Social Advertising
§ No Search Queries
Q : Home Remodel San Francisco Bay Area
E( pCTR(clicka | qi , u j,C)) >> E( pCTR(clicka | u j, C))

11. Sponsored Search vs.
Social Network Advertising
Source : Marin Software
www.marinsoftware.com

12. Dedicated Marketing Channels

13. Hypertargeting
E(max[c1, c2, c3.....c n ]) > E(max[c1, c2, c3.....c n−1 ])

14. Blending Organic and Sponsored

16. The thing called Mobile..
§ Cannibalizing website
page views
§ Small form factor
§ ~10% views from
Mobile but only ~1%
monetizable
§ Blending organic and
sponsored essential
§ Impression &
conversion tracking
loop hard to close

18. The good news..
Hey user.. I know thee

19. The good news..
And your friends..
source: h]p://inmaps.linkedinlabs.com
https://inmaps.linkedinlabs.com
19

20. The good news..
We also know what you read ..
And how much you liked it..

22. Advertising @ LinkedIn
LinkedIn Marketing Solutions
Performance Brand
Organic
Marketing
Marketing
Marketing
(LinkedIn Ads)
(Display Ads)
(Company Pages)

23. Advertiser spectrum
Higher
Educa-on
Tech
B2B
MBA
Programs
|
Masters
&
Graduate
Programs
CRM
|
SoGware/Biz
Hardware
|
ERP
|
Sales
Tools
Online
Degrees
|
Execu-ve
Leadership
Marke-ng
Automa-on
|
SaaS
Internet
Services
Staffing
&
Hiring
Website
Hos-ng
|
Video
Conferencing
Staffing
Agencies
|
Recrui-ng
SoGware
Prin-ng
|
Phone
Systems
Corporate
Recrui-ng
|
Job
Boards

24. Campaign creation

25. The basics -­‐‑ Ad Ranking
§ Given
U j , {(c0, b0 ), (c1, b1 ), (c2, b2 ), (c3, b3 )..(cn, bn )}, H
§ Objective
argmax( pCTR i *bidi )
i∈C
§ Goal:
§ Increase revenue
§ Respect daily budgets of Advertisers
§ Good user experience

26. Virtual Profiling
Title :
Title : Eng Mgr
Sr. SE<1>, Eng Mgr<1>,
Company : LinkedIn
Eng Dir<1>
Location : CA,USA
Skills : ML, RecSys
Company :
LinkedIn<2>, Google<1>,
Title : Sr. SE
Company : Google
Location : PA, USA
Location :
Skills : ML, DM
CA,USA <2>, PA, USA<1>
Skills :
Title : Eng Dir
ML<2>, RecSys<1>,
Company : Linkedin
Stats<1>, DM<1>
Location : PA, USA
Skills : ML, Stats, DM

27. Virtual Profiling
Title : Eng Mgr
Company : LinkedIn
Location : CA,USA
Skills : ML, RecSys
Title : Vice President
Company : Twi]er
Location : CA,USA
Skills : DM, ML,
RecSys
……………….

28. Virtual Profiling
Information Gain
§ Pick Top K overrepresented features from the
clicker distribution vs the target segment
A representative projection of the item in the
member feature space

29. CTR Prediction – CF Similarity
Ranker
MEMBER FEATURES
AD CREATIVE VIRTUAL PROFILE
Creative Score to
features
pCTR
pCTRi correction
§ L2 regularized Logistic Regression (Liblinear, VW, Mahout, ADMM)
§ Frequency or conditional smoothed oCTR as feature values from
activated features in the Virtual Profile
§ For new ad creatives back-­‐‑off to the advertiser / ad category nodes till
they reach critical impression/click volume (explore/exploit)

30. What about Hypertargeting ?
Done via
§ Transitions probability
§ Profile collocation analysis
§ Co-­‐‑Targeted segments
§ Virtual Profile Similarity
§ A/B tested for most effective
solutions

32. Recommendations: What are they worth? Think 50%
§ > 50% of connections are from recommendations
(PYMK)
§ > 50% of job applications are from
recommendations (JYMBII)
§ > 50% of group joins are from recommendations
(GYML)
RecLS 32

33. Hiring Solutions – Self Serve Jobs Postings

34. Sponsored Recommendations

35.
Talent Match

36. Recommendation Algorithm
Job
Corpus Stats
Matching
Transition probabilities
Connectivity
title
geo
industry
description
… Binary
Exact matches:
yrs of experience to reach title
education needed for this title
…
company
functional area geo, industry,
…
Soft
User Base
Similarity
(candidate expertise, job description)
transition
Filtered 0.56
probabilities,
Candidate
similarity,
…
Similarity
(candidate specialties, job description)
0.2
Transition probability
0.43
Text
(candidate industry, job industry)
General
Current Position
expertise
title
0.8
specialties
summary Title Similarity
education
tenure length
headline
industry
geo
functional area
0.7
Similarity (headline, title)
experience
…
deriv
ed
.
.
.

37. Feature Engineering – Entity Resolution
§ Companies
‘IBM’ has 8000+ variations
- ibm – ireland
- ibm research
- T J Watson Labs
- International Bus. Machines K-Ambiguous
- Deep Blue
• Huge impact on the
business and UE
• Ad targeting
• TalentMatch
• Referrals
37

38. Feature Engineering – Entity Resolution
§ Binary classifier (LR), not ranker
§ P({position, company entity} is
a match)
§ Features:
§ Content – name similarity
features, industry match,
location match, email domain
match, company size
§ Social Graph -­‐‑ # connections
at company entity
§ Behavior -­‐‑ # of invitations
received from company entity
members
§ Company candidate set 97% Precision
leveraged from Social graph at 50% Coverage
and cosine similarity
Asonam’11, KDD’11
38

39. Feature Engineering – Sticky locations
§ Zip code mapped to Regions
§ How sticky are those locations?
§ Huge impact on the business and UE
• Job Seeker, Recruiter

40. Feature Engineering – Sticky locations
§ Open to relocation ?
§ Region similarity based on profiles or network
§ Region transition probability
§ predict individuals propensity to migrate and
most likely migration target
§ Impact on job recommendations
§ 20% lift in views/viewers/applications/applicants

41. The Network effect
41

42. What should you transition to & when ?
42

43. Multiple Objective Optimization
Applicable in multiple contexts
§ Online Dating
§ Click Shaping
§ Revenue vs CTR optimization tradeoff
§ Talent Match
Luiz Pizzato, Tomek Rej, Thomas Chung, Irena Koprinska, Kalina Yacef, and Judy Kay. 2010. Reciprocal recommender system for online dating. In
Proceedings of the fourth ACM conference on Recommender systems (RecSys 'ʹ10). ACM, New York, NY, USA, 353-­‐‑354.
Deepak Agarwal, Bee-­‐‑Chung Chen, Pradheep Elango, and Xuanhui Wang. 2011. Click shaping to optimize multiple objectives. In Proceedings of the 17th
ACM SIGKDD international conference on Knowledge discovery and data mining (KDD 'ʹ11). ACM, New York, NY, USA, 132-­‐‑140
Mario Rodriguez, Christian Posse, Ethan Zhang.2012. Multiple Objective Optimization in Recommender Systems. To appear in Proceedings of the Sixth
ACM conference on Recommender systems (RecSys 'ʹ12)

44. Multiple Objective Optimization
JobSeeker
Intent
TalentMatch
Member Z, 0.89,
job
Active
…
MOO
Member X, 0.98, 0.98,
+40% InMail Response NonSeeker
Rate
Member Y, 0.91, 0.91,
NonSeeker
…
Formalism

45. Multiple Objective Optimization
§ TalentMatch
§ Logistic Regression model
§ JobSeeker Intent
§ Ordered Logistic Regression model
§ Active/Passive/NonSeekers
§ Outputs propensity score
§ MOO (Multiple Objective Optimization)
§ Grid Search on Objective function
§ sMOOth for large parameter spaces

46. Multiple Objective Optimization
Formalism
1. Rank top K’ > K semantically rank results
2. Perturb the ranking with a parametric function parameterized by α , β
which leads to inclusion of the secondary objective
3. Measure the perturbation using a delta function wrt to the primary
objective
4. Create a framework to quantify the tradeoff between the two objectives

47. Multiple Objective Optimization

48. Multiple Objective Optimization

49. Social Referral

50. Social Referral
§ Order recommendations by
§ Connection Strength between two users =>
(ui , u j )
σ
§ Recommendation Strength for the target user =>
R(u j , gk )
§ Combination thereof
Linkedin Group: Text Analytics
From: Deepak Agarwal – Engineering Director, LinkedIn
I found this group interesting, and I think you will too
Deepak
2X conversion
Linkedin Group: Text Analytics
> 2X Conversion
Mohammad Amin, Baoshi Yan, Sripad Sriram, Anmol Bhasin, Christian Posse. 2012. Social
Referral : Using network connections to deliver recommendations. To appear in Proceedings of
the Sixth ACM conference on Recommender systems (RecSys 'ʹ12)

51. Follow Ecosystem

52. Recommended Followers Targeting
Task : To identify a set (usually Millions) of
users likely to follow the given company
Scorer
MEMBER PROFILE FEATURES
COMPANY FOLLOWER VIRTUAL
PROFILE
Global
Company p( follow | ci , u j )
popularity
Other rankings –
1. User’s login probability in next X days
2. User’s PVs in the next X days
3. User’s propensity to follow any company
Weighted Borda count to for Information Fusion & A/B Test combinations
h]p://www.colorado.edu/education/DMP/voting_b.html -­‐‑ loss of information in plurality votes

53. A/B Testing
Is Option A Be]er Than Option B? Let’s Test
Beware of
§ novelty effect
§ Cannibalization
A/B allows seemingly subjective questions of design—color, layout, image
§ potential biases (time, targeted population)
selection, text—to become incontrovertible ma]ers of data-­‐‑driven social science.
§ random sampling destroying the network
-­‐‑ Dan Siroker, Digital Advisor to Barack Obama’s election campaign -­‐‑2008
effect
h]p://www.wired.com/business/2012/04/ff_abtesting/
Don’t forget to A/A test first
Enjoy testing furiously!. Hundreds of tests live on LinkedIn at all times..
job$views$per$5%$bucket$range$5$6/5/11$ job$views$6/19/11$
9,000"
7,000"
8,000"
7,000"
6,000"
6,000" 5,000"
5,000" 4,000"
job"views"per"5%"bucket"range"?"
4,000" 6/5/11"
3,000" job"views"6/19/11"
3,000"
2,000"
2,000"
1,000"
1,000"
0" 0"
0" 5" 10" 15" 20" 25" 0" 5" 10" 15" 20" 25"
(“Seven Pitfalls to Avoid when Running Controlled Experiments on the Web”, KDD’09
“Framework and Algorithms for Network Bucket Testing” WWW’12 submission)
53 53

55. Demand exceeds supply
Supply
Demand
Newer advertise
r
acquisition
Page view
growth , highest
Social media
in mobile
frenzy
# of pages with Newer Ad
ads
products like
“fans” &
“follows”
55

56. Real Time Bidding
Fun problems
§ When not to bid ?
§ CTR prediction on the publisher
§ What auction does the exchange run ?
§ Onsite vs Offsite impression tradeoff
for impression capped campaigns

57. Onsite/Offsite tradeoff
LinkedIn Ads shown to LinkedIn Member – zillow.com

58. Other initiatives..
§ Audience Forecasting
§ Bid Landscaping
§ Lookalike Modeling
§ Publisher DNA
§ Auto ad creative generation from landing pages
§ Explore Exploit strategies
And more..

59. New Product!
§ New guaranteed display ad product
§ Impressions guaranteed = 1
§ eCPI > $[0-­‐‑9]{1}000

60. Picture yourself with this New Job:
You
Applied Researcher /
Research Engineer

61. Credits
Engineering : Abhishek Gupta, Adam Smyczek, Adil Aijaz,
Alan Li, Baoshi Yan, Bee-­‐‑Chung Chen, Deepak Agarwal,
Ethan Zhang, Haishan Liu, Igor Perisic, Jonathan
Traupman, Liang Zhang, Lokesh Bajaj, Mario Rodriguez,
Mohammad Amin, Parul Jain, Sanjay Dubey, Tarun
Kumar, Trevor Walker, Utku Irmak
Product : Christian posse, Gyanda Sachdeva, Mike
Grishaver, Parker Barrile, Sachit Kamat, Andrew Hill

62. Thank You!
Questions?
Contact:
abhasin@linkedin.com
h]p://engineering.linkedin.com/
62