1.
Improving TripAdvisor
Photo Selection
With Deep Learning
Greg Amis
May 25, 2018

2.
About Me: Applied ML, new to MV
2
Machine vision
● Photo selection
Text processing
● Inappropriate reviews
Metadata processing
● Review fraud
Online learning
● Adaptive radar jamming
Text processing
● Topic time series
Agent-based modeling
● Personnel forecasting
● Brain models
● Brain-inspired architectures
● Semi-supervised learning
● Some classes on biological
and machine vision
Department of
Cognitive &
Neural Systems

3.
About TripAdvisor
3
(1) Includes 1.1M hotels, inns, and bed & breakfasts, as well as 800K vacation rental listings
(2) TripAdvisor internal log files, average monthly unique visitors during Q2 2017
+ 40M photos
from professionals
= 150M total
● Largest travel website
● ~400 engineers
● ~40 data scientists and ML engineers

4.
TripAdvisor Redesign: Photo-centric
4

5.
5
Sometimes we show great photos...

6.
6
Sometimes… not so much

7.
Amenity-specific shelves
don’t always show the amenity
7

8.
Photo ordering matters
8
#6 hotel in Atlantic City
4-bubble hotel in Cancun, 6.7k reviews

9.
1. Good primary photos
2. Relevant amenity photos
3. Good default sort order
Goal: Show attractive, useful photos
9

10.
Approach
15 interns, a mini-fridge of GPUs, great OSS
10

11.
11
Gather Training Data
● Interns/MTurk labeled photos
● Pairwise ranking
○ 200,000 photo pairs
○ “Which one motivates you to click?”
● Label photos containing humans
● Label photos by scene type
○ Pool, beach, room, etc.
○ Food, drink, inside, outside, etc.
● Simple infrastructure
Python
Pandas
HTML
JavaScript
Python
CherryPy

12.
● Start with 50-layer ResNet convolutional neural network
trained on 1,000-class ImageNet data
● Remove upper layers concerned with classification
● Remaining lower layers make an excellent feature
extractor for other machine vision problems
Feature Space
12
(He et al, 2015)
...
ResNet
50 ...
2,048
“bottleneck features”

13.
For subjective scoring: Siamese Network
For classification: Multi-layer feedforward
networks with dropout
Model architecture
13
def create_mlp(input_size=2048,output_size=1,
hidden_layer_sizes=(2048, 2048),
dropout_rates=(0.5, 0.5)) -> (Model, Model):
model = Sequential()
model.add(Dense(hidden_layer_sizes[0],
activation='relu', input_shape=(input_size,)))
model.add(Dropout(dropout_rates[0]))
for (h, d) in zip(hidden_layer_sizes[1:], dropout_rates[1:]):
model.add(Dense(h, activation='relu'))
model.add(Dropout(d))
model.add(Dense(1, activation='sigmoid'))
model.compile(optimizer="adadelta",
loss="binary_crossentropy",
metrics=["binary_crossentropy", "accuracy"])
return model
Dense 2048
σ
Dense 2048
Dropout 0.5
Dense 1
Bottleneck
Features For
Better Image
Dropout 0.5
Dense 2048
Dense 2048
Dropout 0.5
Dense 1
Bottleneck
Features For
Worse Image
Dropout 0.5
+ −
Maximize
Siamese Network
Inspired by
Microsoft’s RankNet
(Burges et al, 2005)
and by Michael Alcorn’s Keras implementation.

14.
● R&D tech stack
○ Two consumer-grade GPUs
○ Keras + Tensorflow
○ Pandas
● Random hyperparameter search
○ Hidden layer width
○ Dropout rate
○ Mini-batch size
○ Epoch count
● Evaluation
○ Cross validation
○ A/B Testing (50% of users see photos selected by machine vision)
Training and Evaluation
14
Kenmore - our “mini-fridge” of GPUs

15.
Deployment
1515
Kubernetes Cluster
For Computation
Spark+YARN Cluster
For Storage
t_photo
t_photo
_bottlenecks
t_photo
_vision_models
1. Get URLs as
PySpark DataFrame
2. Split into partitions
3. Feed partitions to process pool:
a. Get image bytes from CDN
(thread pool)
b. Calc stats
c. Calc bottlenecks
(if necessary)
d. Calc model outputs
4. Write back partitions
asynchronously
SELECT …
FROM t_photo
LEFT ANTI JOIN
t_photo_vision_models
...
FROM tmp_new_partition
INSERT INTO
t_photo_bottlenecks
PARTITION(...)
SELECT ...
WHERE bottlenecks NOT NULL
INSERT INTO
t_photo_vision_models
PARTITION(...)
SELECT ...
NVIDIA
GeForce
GTX 1080 Ti

16.
Results
Pretty food, fewer bathrooms
16

17.
Better Restaurant Hero Photos
17
Initial
Hero
Photos
Photos
Selected
Using
Machine
Vision
Restaurant A Restaurant B Restaurant C

18.
Better Hotel Hero Photos
18
Initial
Hero
Photos
Photos
Selected
Using
Machine
Vision
Hotel A Hotel B Hotel C

19.
Hotels With A Pool
19
Original shelf, just showing hero photos
Same hotels, pool photos selected using machine vision

20.
Beachfront Hotels
20
Original shelf, just showing hero photos
Same hotels, beach photos selected using machine vision

21.
Better Sort Order
21
Original sort order
Same hotel,
photos sorted using
machine vision

22.
Better Sort Order
22
Original sort order
Same hotel,
photos sorted using
machine vision

23.
| tripadvisor is hiring!
○ Software Engineer - Machine Learning
○ Data Scientist - Attractions and Rentals
○ Data Scientist - Search Engine Marketing
○ Data Analyst - Attractions and Rentals
○ Software Engineer - Full Stack Web

24.
● At TripAdvisor
○ Jeff Palmucci
○ Aaron Gonzales
○ Anyi Wang
○ Tyler O’Brien
● Outside TripAdvisor
○ Google: Keras, Tensorflow
○ NVIDIA: CUDA
○ Microsoft: ResNet-50
○ Original research at Google, Microsoft, Facebook,
University of Toronto, NYU
Acknowledgements
24

25.
25

26.
Primary Green #00AF87
Primary Black #000A12
Accent Red #EF6945
Accent Blue #1C99CE
Accent Yellow #F8C40F
Secondary Dark Gray #666666
Secondary Gray #B7B7B7
Secondary Light Gray #E5E5E5
Colors
26