Sonya Liberman leads the Personalization team @ Outbrain's Recommendations group, developing large-scale machine learning algorithms for Outbrain's content recommendations platform serving tens of billions real-time recommendations a day. She specializes in Information Retrieval, Machine Learning, and Computational Linguistics. Before joining Outbrain, she led the Research and Algorithms @ ConvertMedia (acquired by Taboola). She holds an MSc in Computer Science and a BSc in Computer Science and Computational Biology. This invited talk was given at the Recommender Systems Workshop 2017, University of Haifa.

1. Looking at
Content
Recommendation
through a
Search Lens

2. ?
What Is

3. Distribution Partners

8. The Lighthouse
Help people discover content they can trust
to be interesting, relevant, and timely for
them

9. 554 MONTHLY
UNIQUEUSERS
GLOBALLY
MILLION

10. BILLION PAGE VIEWS
PER MONTH

11. 200BILLION
RECOMMENDATIONS
SERVED PER MONTH

12. 12
Recommending Content

13. 13
Content Recommendation Engine
Relevance
Rec Engine
Content Inventory

14. The Lighthouse
Help people discover content they can trust to
be interesting, relevant, and timely for them
Main Players - Publishers, Marketeers,
Users
Use
r
Publishe
r
Marketeer

15. 15
Challenges
• Personalization
• A Jungle of Market Rules
Geo targeting, publisher blacklisting of sites, URLs,
titles
• Scale
35K req/sec, 50ms latency, millions of potential content
recs

16. 16
Search Engines
What can they do?

17. 17
1. Score documents by relevance to query
Relevance
Query
Donald
Trump Search
Engine

18. 18
2. Filter documents by certain attributes

19. 19
3. Work Efficiently and at Scale

20. 20
3. Work Efficiently and at Scale
what the
day
brings

21. 21
3. Work Efficiently and at Scale
what the
day
brings

22. 22
3. Work Efficiently and at Scale
what the
day
brings

23. 23
3. Work Efficiently and at Scale
Open
Source
Distributed
Scalable
RESTful
Real-time
search

24. 24
3. Work Efficiently and at Scale

25. 25
How Do we Reduce the Problem of
Recommending Content to Users
to a Search Problem?

26. 27
John, www.angelina.com
Television and
Celebrities
Blacklist Site:
www.brad.com
Translate user and context to a query of
interests and market rules

27. 28
Translate articles to searchable documents in
the same feature space of user interests and
market rules
Is about:
Celebrities
site:
www.brad.com
Breakup: What’s Next?
Brad's
acting
career
continues to flourish
while he films a new …

28. 29
Representing Users and Content in the
Same Feature Space
In classical search – Queries and Documents are
represented in the feature space of ‘terms’
In content recommender system - Users and Content
are represented in a Semantic Feature Space

29. 30
What is a Document About?
Semantic Features
Categories
Entertainment/Television
Topics
Story, Murder, Television
Entities
Dolores, Westworld, HBO
NLP

30. 31
Constructing a User Profile
Time
Semantic Features
Categories
Topics
Entities

31. 32
User Profile

32. User Profile

33. 39
Let’s Write An Elasticsearch Query

34. 40
{
"query": {
"bool": {
"should": [
{"terms":{
"categories": ["television", ”celebrities"]}
},
{"terms":{
"topics": ["business", "cinema", "murder"]}
},
{"terms":{
"entities": [”hbo", ”dolores", ”nyse"]}
}
]
}
}
}
Create Elasticsearch Query with User Interests

35. 41
{
"query": {
"bool": {
"should": [
{ "terms": {
"categories": {
"query": "television", "boost": 2.3 }}},
{ "terms": {
"categories": {
"query": "investments", "boost": 1.6 }}},
{ "terms": {
"entities": {
"query": ”dolores", "boost": 1.2 }}}
]
}}}
Using Weights to Improve Relevance

36. 45
Adding Market Rules to The Game

37. 46
Query with Blacklisted Sites
”www.angelina.com"
Blacklisted: ”www.brad.com”
From Market Rules to Elasticsearch Filters

38. 47
www.angelina.com
Television and
Celebrities
Blacklist Site:
www.brad.com
The Market Rules of the Publisher are
Represented in the Query

39. 49
Query with Blacklisted Sites
{ "must_not": [
{ "terms": {
"site": [
“www.brad.com”,
]}}
]}
www.angelina.com

40. { "must_not": [
{ "terms": {
"site": [
“www.brad.com”,
]}}
]}
50
{
"title" : ”Breakup: what’s next?",
”site" : ”www.brad.com”
}
Query with Blacklisted Sites

41. { "must_not": [
{ "terms": {
"site": [
“www.brad.com”,
]}}
]}
51
{
"title" : ”Breakup: what’s next?",
”site" : ”www.brad.com”
}
Document is
Filtered Out
Query with Blacklisted Sites

42. {
"title" : ”Breakup: what’s next?",
”site" : ”www.brad.com”
}
{ "must_not": [
{ "terms": {
"site": [
“www.brad.com”,
]}}
]}
52
{
"title" : ”Madonna adopted son David",
”site" : “www.madonna.com”
}
Document
Passes Filter
Query with Blacklisted Sites

43. 53
From Market Rules to Elasticsearch Filters
Geo Targeting
”Music World – everything on NY Music Scene "
Targeting "US" users only

44. 54
Index Geo Field in the Document
{
"title" :”Music World–everything on NY Music Scene“,
"categories" : [”music"],
"entities" : [”aerosmith", ”ny"],
"geo" : ["us"]
}

45. 55
Add a Geo Filter to the Query
{
"query": {
"filtered": {
"query": {
"terms": { … }
},
"filter": {
"terms" : { "geo" : ["us"]}
}
}}}

46. 56
Apply Filter on Documents
{
"query": {
"filtered": {
"query": {
"terms": { … }
},
"filter": {
"terms" : { "geo" : ["us"]}
}
{
"title" :”Music World–everything on NY Music Scene“,
"categories" : [”music"],
"entities" : [”aerosmith", ”ny"],
"geo" : ["us"]
}

47. 57
Apply Filter on Documents
{
"query": {
"filtered": {
"query": {
"terms": { … }
},
"filter": {
"terms" : { "geo" : ["us"]}
}
Document
Passes Filter
{
"title" :”Music World–everything on NY Music Scene“,
"categories" : [”music"],
"entities" : [”aerosmith", ”ny"],
"geo" : ["us"]
}

48. 58
Apply Filter on Documents
{
"query": {
"filtered": {
"query": {
"terms": { … }
},
"filter": {
"terms" : { "geo" : ["fr"]}
}
{
"title" :”Music World–everything on NY Music Scene“,
"categories" : [”music"],
"entities" : [”aerosmith", ”ny"],
"geo" : ["us"]
}

49. 59
Apply Filter on Documents
{
"query": {
"filtered": {
"query": {
"terms": { … }
},
"filter": {
"terms" : { "geo" : ["fr"]}
}
Document is
Filtered Out
{
"title" :”Music World–everything on NY Music Scene“,
"categories" : [”music"],
"entities" : [”aerosmith", ”ny"],
"geo" : ["us"]
}

50. 60
What about Documents Without a Specific
Targeting?
{
"query": {
"filtered": {
"query": {
"terms": { … }
},
"filter": {
"terms" : { "geo" : ["us"]}
}
{
"title" :”Music World–everything on NY Music Scene“,
"categories" : [”music"],
"entities" : [”aerosmith", ”ny"],
"geo" : [“"]
}

51. 61
What about Documents Without a Specific
Targeting?
{
"query": {
"filtered": {
"query": {
"terms": { … }
},
"filter": {
"terms" : { "geo" : ["us"]}
}
Document is
Filtered Out
{
"title" :”Music Around the World“,
"categories" : [”music"],
"entities" : [”colplay", ”muse"],
"geo" : [“"]
}

52. 62
Solution – Index & Query the Value "all"
{
"query": {
"filtered": {
"query": {
"terms": { … }
},
"filter": {
"terms" : { "geo" : ["us", "all"]}
}
{
"title" :”Music Around the World“,
"categories" : [”music"],
"entities" : [”colplay", ”muse"],
"geo" : [“all"]
}

53. {
"query": {
"filtered": {
"query": {
"terms": { … }
},
"filter": {
"terms" : { "geo" : ["us", "all"]}
}
63
Solution – Index & Query the Value "all"
Document
Passes Filter
{
"title" :”Music Around the World“,
"categories" : [”music"],
"entities" : [”colplay", ”muse"],
"geo" : [“all"]
}

54. 65
Beyond the Content Based
Recommendations Model

55. 66
Supervised Learning from Explorative Data
Recommend high-variety content to users

56. 67
Supervised Learning from Explorative Data
Capture patterns beyond matching semantics

57. 68
Supervised Learning from Explorative Data
Offline Model
Learning
Framework
Online Serving
Request Response
Models

58. 69
Dynamic Signals and Data Leakage

59. 70
Some signals are highly dynamic
Offline signals must match online signals
Example: User Profile is changing with every user’s
interaction
Supervised Learning in an Environment of
Dynamic Signals

60. 71
Time
Music
Musi
c
Tech
Musi
c
Tech
Trave
l
Supervised Learning in an Environment of
Dynamic Signals

61. 72
Time
Music
Musi
c
Tech
Musi
c
Tech
Trave
l
Supervised Learning in an Environment of
Dynamic Signals

62. 73
Time
Music
Musi
c
Tech
Musi
c
Tech
Trave
l
Learn
Model
Supervised Learning in an Environment of
Dynamic Signals

63. 74
Time
Music
Musi
c
Tech
Musi
c
Tech
Trave
l
Learn
Model
Supervised Learning in an Environment of
Dynamic Signals

64. 75
Time
Music
Musi
c
Tech
Musi
c
Tech
Trave
l
Learn
Model
Supervised Learning in an Environment of
Dynamic Signals

65. User’s interactions affect user’s profile
76
Preventing Data Leakage
Time
Music
Musi
c
Tech
Learn
Model
Photo-
graph
y

66. User’s interactions affect user’s profile
77
Time
Music
Musi
c
Tech
Learn
Model
Photo-
graph
y
Preventing Data Leakage

67. User’s interactions affect user’s profile
78
Time
Music
Musi
c
Tech
Learn
Model
Photo-
graph
y
Preventing Data Leakage

68. 79
Solution - Log And Learn Framework
Serving-time
Logged Data
Offline Model
Learning
Framework
Online Serving
Request Response
Models
Signals
Snapshot
s

69. 80
Supervised Learning from Explorative Data
Regression Models
Collaborative Filtering Models
Factorization Machines

70. 81
Elasticsearch Custom Scoring Functions
Writing our own scoring functions with native Java via
Elasticseach plugins mechanism
Passing parameters to Elasticsearch
Applying machine learned models in
serving time

71. Thank You
sliberman@outbrain.com